No Content, No Fuss: Category hacking

Parsing EXIF/IPTC photo tags using pyexiv2 on Leopard

anil@recoil.org (Anil Madhavapeddy) — Thu, 17 Apr 2008 11:58:00 +0100

I've been looking around at refreshing my current photo gallery to make it easier to update, since I'm around 1000 photos behind over the last year. Part of making it easier to use is to adopt an Aperture-based system of marking the metadata directly in the image itself, and figuring out how to render so many pictures more effectively in my gallery.

The best library I could find to do the job is Exiv2 and its associated Python bindings pyexiv2. They use the rather complex Boost.Python bindings, which are a total pain to compile on MacOS X Leopard.

In order to get it to work:

Use the latest Fink to install:
- python25
- boost1.34.python25
- libexiv2
- scons
Download the latest pyexiv2 distribution, extract it, and drop in my exiv2 Leopard Makefile into the extracted directory.
Type in:
- make
- sudo make install.
Test it out by running python2.5 and experimenting with importing pyexiv2 (see the pyexiv2 developer guide for more information.

I'll tidy this up into a fink package sometime, but for now I want to press on with finishing my Django experiments.

Deens, welcome to the Internet!

anil@recoil.org (Anil Madhavapeddy) — Sat, 30 Dec 2006 01:11:00 +0000

Inspired by finishing my PhD corrections (!) today, I decided to hook up the DNS server from our Melange project up to the Internet. The authoritative server is called deens (since the co-author is one Tim Deegan, geddit?), and is written in pure OCaml.

This is all rather experimental, to put it mildly, but I stuck in the zone file below, hooked it up as a delegate to our main name-servers, checked it against the DNS Report, and it all seems to be working!

$ORIGIN deens.recoil.org. ;
$TTL    240
deens.recoil.org. 604800 IN SOA  (
    deens.recoil.org. anil.recoil.org.
    2006122401 3600 1800 3024000 1800
)
        IN  NS     ns1.deens.recoil.org.
        IN  NS     deensns.recoil.org.
ns1     IN  A      194.70.3.132
dynamic IN  CNAME  dynamic.recoil.org.
static  IN  CNAME  static.recoil.org.
anil    IN  CNAME  dynamic
stats   IN  CNAME  dynamic

I also modified stats.recoil.org to be an alias to stats.deens.recoil.org, so all the requests for that domain will go via the deens setup. You actually need a user/pass to access the site, but that doesn't matter; if it gets that far, the DNS bit has worked.

There's still an awful lot of tedious work to get the server into a production-ready state, such as proper logging, more error handling and recovery, etc., but I really hope to find the time in 2007 to polish this up somewhat. Performance is excellent already; faster than BIND by quite a lot, and it can optionally use more memory to cache responses to shoot up to crazy levels.

Incidentally, the dig replacement utility also seems to be working fairly well, and David Scott has been messing around with a Bonjour implementation that will get finished sometime in 2007 as well (honest!).

Mercurial FastCGI module

anil@recoil.org (Anil Madhavapeddy) — Wed, 27 Dec 2006 20:59:00 +0000

Our lighttpd setup has been very unstable in recent months, probably brought on by the load of the large Mercurial repositories hosted on Recoil since the Google Summer of Code mentoring.

The source of the instability was really hard to track down, but it seems to be the automatic spawning of FastCGI processes by the web-server, and lighttpd failing to handle a SIGCHLD somewhere when a child process crashes. To sort this out, I just converted all the Ruby on Rails setups (this blog and Nick's) to use an external spawn.

This only leaves our Mercurial vhost hg.recoil.org to switch to using FastCGI, and I couldn't find a module for this anywhere and so lashed up some Python glue to do the job.

You can download the small distribution for Mercurial 0.9 (hg-fcgi-0.9.tar.gz). It has a FastCGI library written by someone else, the Python files to glue the Mercurial and FastCGI libraries together, and a simple rc script to launch the external web process.

Instructions are for lighttpd... install the Python files somewhere, modify them to point to the Mercurial directory, run the rc script to start the daemon, and then add something similar to the following to your lighttpd config file:

fastcgi.server = (
  ".fcgi" => ( "localhost" =>
    ( "socket" => "/var/cache/fcgi/sites/hg.recoil.org/dirsock" )),
  ".hg" => ( "localhost" =>
    ( "socket" => "/var/cache/fcgi/sites/hg.recoil.org/sock" )),
)

Also add "index.fcgi" to index-file.names in the config file, and touch it in the vhost directory to create an empty file (this is to avoid getting a 404 error and instead pass it through to the FastCGI process). Similarly, touch a .hg file for every repository you want to serve. You could do this differently by passing through a URL prefix and modifying the Python appropriately, but I prefer finer control over what we're serving.

Hope this is useful; I won't bother submitting it back to the Mercurial list as it looks like the official hg repo has a different code layout; I'll check it out later on when I have a bit more time and integrate properly.

I have no idea whether or not this will actually improve our stability, but it's at least easier to move onto a different web-server now that everything is FastCGI. All I need now is an OpenBSD/php5-fastcgi port, which doesn't seem to exist (yet).

Practical OCaml

anil@recoil.org (Anil Madhavapeddy) — Mon, 28 Aug 2006 23:12:00 +0100

I noticed on the OCaml mailing list that Practical OCaml is due to be released in the US quite soon. Could this be the first real competition that Jon Harrop will face for his excellent but pricey Objective Caml for Scientists ?!

Things are heating up on the OCaml book front at last, and Jon has been getting some rave reviews of his book! We have a long-running joke about how much I hate the colour syntax highlighting he used in his book, and it amused me greatly to note that someone else in the comments for the rave review of his book also shared my heretical opinion, ha ha!

Incidentally, I just pushed the beginnings of an OCaml config file parsing library to the Melange source tree. The code in there is almost beginning to look useable...

My first Cocoa steps

anil@recoil.org (Anil Madhavapeddy) — Sat, 15 Jul 2006 03:45:00 +0100

Ever since I got my first Powerbook back in 2000, I've been meaning to learn Cocoa and Carbon to hack on MacOS X GUIs. On the flight over to San Francisco, I finally found the uninterrupted time to knock up a simple GUI interface to my gallery meta-data files. Overall, the experience was pretty positive. It's definitely more satisfying than my experiences with Windows and UNIX GUI programming, mainly because the result is so pretty!

Objective-C has a really nice dynamic message dispatch mechanism, used to good effect by Cocoa. The autorelease mechanism plays well with C memory management to provide a reference-counted interface which is fairly well abstracted. My application still leaks memory like a sieve though; I miss OCaml!
Cocoa provides a huge number of ways to do the same thing. Bindings are the fancy new thing, MVC is the older "paradigm", and of course Carbon predates all of this. The documentation is a bit of a mess for a beginner to the framework, as crucial facts on how to get simple stuff done are scattered across a myriad of documents. Thanks Google.
Tiger has some funky new Cocoa controls. Check out the use of NSTokenField on the screenshot which lets the tag entry field auto-complete just like Mail does!
The APIs can be unfortunately verbose at times, as the following code snippet which does a little bit of simple string manipulation shows.

 NSArray *a = [tag componentsSeparatedByString:@" "];
    iter2 = [a objectEnumerator];
    while (s = [iter2 nextObject]) {
        NSString *e = [s stringByTrimmingCharactersInSet:
            [NSCharacterSet whitespaceCharacterSet]];
         if (![e isEqual:@""]) 
            [tags addObject:[e retain]];
    }

I'm quite looking forward to doing more work on this simple application now that the basics are mastered. Hmm... preferences panes next, then a bit of Cocoa bindings, and perhaps I feel OCamlCocoa coming on...

Cambridge quickies

avsm — Fri, 23 Sep 2005 15:38:10 +0100

Well I've been back in Cambridge for a week now, catching up on stuff that's been languishing in the last month of travelling. So here are some blog quickies!

Shouts out to the newest blogger, the least German-like German person I know! Whooooop yeah yeah...
Released a new version of OpenFX, with some really cool new features from Dr. Stuart Ferguson!
Sorry about the backlog of OpenBSD-related things; I think I've caught up on all my mails, and imported and updated lots of bits and bobs and ports. Please re-mail me if I've ignored anything you have sent me recently.
Released a test version of review2atom, the result of my ocamlduce hackery (and many thanks to Alan Frisch for answering my incessant questions!).
Submitted DarwinPorts for calendar (dports/devel/caml-calendar) and ocamlduce (dports/lang/ocamlduce).
I'm going to buy a juicer; turnip smoothie is en route!

Python on Symbian Series 60

avsm — Fri, 24 Jun 2005 14:14:37 +0100

I bought a Nokia 7610 some time ago, and have taken hundreds of pictures using it. When creating the gallery section of my site, I ran into an annoying limitation: the pictures contained on the phone did not embed the date the picture was taken as part of the image. Manually copying the date and time for 350 images was a bit much as well...

The problem with sending images via Bluetooth to my laptop is that the OBEX transfer stamps the date of the transfer instead of the date the picture was taken. There turned out to be a conveniently easy answer to this problem though, as Nokia released a version of Python for Symbian Series 60. This suddenly elevates the phone from being a complete pain to program (due to the Symbian C++ libraries being amazingly badly documented), to something that's actually quite usable.

It took about an hour to come up with the first cut of imgcollect.py, which assembles a list of mtimes for each image on my phone, performs Bluetooth Device Discovery, and transmits the mtimes and all the pictures to my laptop. After that, a little shell script on the laptop is enough to stamp the pictures with the correct mtime:

#!/bin/sh
while read FNAME MTIME; do
        DATE=`date -r $MTIME "+%G%m%d%H%M.%S"`
        touch -t $DATE $FNAME
done

I'm actually quite impressed with Nokia at last! Next step is to work out a two-way synchronization so that I don't keep re-transmitting images which I already have on the phone (unfortunately, iSync is pretty useless with the Nokia 7610, even in Tiger).

Updated Subethaedit OCaml mode

avsm — Tue, 21 Jun 2005 22:01:38 +0100

The Subethaedit syntax highlighting for OCaml is working nicely. I fixed up few minor bugs and packaged up version 1.1, available here. Have fun!

C2K5 thoughts

avsm — Sat, 04 Jun 2005 16:14:18 +0100

Finally had some time to get back from the OpenBSD hackathon and take stock of what I worked on. It was pretty interesting one this year, as I went without having much idea of what to work on (unlike last year, when I had a mad backlog to catch up on). Some stuff I did during the week included:

Clean up the atomicio interface used in OpenSSH and nc(1) to provide simpler semantics. Error checking from read/write functions are a real headache in C, as the functions return -1 on error, which means a signed ssize_t is returned. However, they accept an unsigned value as the size of the buffer to process, which means they could potentially return a value outside the range of the return value. This means you have to check if the return is -1, which indicates an error, and otherwise cast to a size_t to correctly get the buffer size back. With the new atomicio, it always returns a size_t, and returns 0 to signal an error (with errno containing the error, and EPIPE being set for an EOF condition).
Start looking at the Bluetooth stack to get L2CAP and RFCOMM support. We are half-way through un-netgraphing the FreeBSD stack and having a more traditional netbt socket interface (much like netinet or netinet6) to Bluetooth.
Use CIL to implement a few fun kernel source->source transforms. kerneltrace just accepts a regular expression and inserts a printf in the function prologue which outputs the function name and any arguments passed into it. Had this idea when chatting with Marius, and it turned out to be very useful when trying to figure out dataflow in the Bluetooth stack (just compile with make CC="/usr/local/bin/cilly --dokerneltrace --trace-regexp='ubt|ng_blue'"). The second one was even simpler; randomvars assigns a non-zero value to every local variable in a function call to help track down uninitialized-local-variable bugs. Heres one Chad Loder found in mg(1).
Other random signed/unsigned cleanups in OpenSSH. Boring but important I guess...

All in all, the hackathon re-motivated me to continue work on the OCaml-based daemons that Dave and I have been hacking on. I don't want to be fixing random buffer or integer overflows in an OpenBSD hackathon 5 years from now; we need to move on to more high-level issues.

Arise, MLBlog

avsm — Mon, 23 May 2005 21:00:04 +0100

I got tired of trying to combine the various blogging and gallery tools into something that did what I wanted: take a simple directory of images, blog entries, links, papers and output a nice HTML/RSS version of the directory. So I hacked up a quick blog tool in OCaml that does the trick, and put it live.

Its got quite a few rough edges at the moment, especially to do with the lack of date archives and the large number of images on the front page of the gallery. Note that the location of the RSS links have changed as well, as I've switched to using ocaml-rss for outputting it instead of the homebrew format used before in blosxom. One thing that has come out really well is the use of a flat tag namespace instead of the previous directory structure; it allows me to share stories and images among multiple categories without needing symlink hacks.