No Content, No Fuss

Parsing EXIF/IPTC photo tags using pyexiv2 on Leopard

anil@recoil.org (Anil Madhavapeddy) — Thu, 17 Apr 2008 11:58:00 +0100

I've been looking around at refreshing my current photo gallery to make it easier to update, since I'm around 1000 photos behind over the last year. Part of making it easier to use is to adopt an Aperture-based system of marking the metadata directly in the image itself, and figuring out how to render so many pictures more effectively in my gallery.

The best library I could find to do the job is Exiv2 and its associated Python bindings pyexiv2. They use the rather complex Boost.Python bindings, which are a total pain to compile on MacOS X Leopard.

In order to get it to work:

Use the latest Fink to install:
- python25
- boost1.34.python25
- libexiv2
- scons
Download the latest pyexiv2 distribution, extract it, and drop in my exiv2 Leopard Makefile into the extracted directory.
Type in:
- make
- sudo make install.
Test it out by running python2.5 and experimenting with importing pyexiv2 (see the pyexiv2 developer guide for more information.

I'll tidy this up into a fink package sometime, but for now I want to press on with finishing my Django experiments.

Sushi, Seattle and Seahawks

anil@recoil.org (Anil Madhavapeddy) — Wed, 11 Apr 2007 18:21:00 +0100

So I'm hanging at the XenSource Redmond office for a few days, and munching on delicious sushi at Flo, and it turns out that the dude behind us was none other than Matt Hasselbeck! It seems like only two years ago that I was cheering on the Pittsburgh Steelers and the Bus as they owned the Seahawks.

Unfortunately, I only realised my brief brush with celebrity on our way out of the restaurant, which also probably saved me from saying something stupid to him. As if! But anyway, the, err, moral of this post is that you must check out Flo Sushi if you're in Seattle and eat the spider roll, it's awesome.

Core stability exercising

anil@recoil.org (Anil Madhavapeddy) — Sun, 08 Apr 2007 21:57:00 +0100

It's been a wierd exercise year; I went from hard-core powerlifting (go 220lb bench!), to playing a lot of badminton and tennis, and then ending off the PhD era with breaking my knee (details not suitable for blog publication).

As anyone it's happened to knows, a knee injury is really annoying to recover from as it feels really "wobbly" for ages and can't quite be trusted. I've been doing physio on it myself for about half a year now, and can finally run around for 5km or so without too much discomfort, although yoga and Capoeria are perhaps a few more months away.

But then, the Forrest Gump-class runner and ex-officemate Alex Ho pointed out these amazing exercises by the captain of the CUH&H running team in Cambridge. They all emphasise smooth movement and core strength without the need for heavy weights, so can be done while travelling as well. So highly recommended, that even little Ajay (right) is trying them out!

Karnatik Jazz, Live Painting, and such things...

anil@recoil.org (Anil Madhavapeddy) — Sun, 08 Apr 2007 21:28:00 +0100

Had a great evening out in San Francisco on Friday, when we visited the Red Poppy Art House on Folsom Street to check out a really unique Jazz experience.

The boys from VidyA recently took up an artist-in-residence position there, and performed their fusion of traditional Indian Karnatik music with modern Jazz pieces. They've been making some waves in the SF Jazz scene already it seems, and to top it all of, the owner of the art house did a live painting session during the performance. The audience passed out random bits of stuff, and he incorporated it into a piece done over the course of an evening. Talk about artistic overload!

Love the area as well; there's a nice friendly coffee-house down the road (which is pretty much my only criteria, apart from not getting shot and such).

My school was never this cool.

The area was a mix of Latino and African, it seemed

And in the Red Poppy Art House, eclectic art abounds

Trac spamming is taking down Melange

anil@recoil.org (Anil Madhavapeddy) — Mon, 02 Apr 2007 11:15:00 +0100

After loads of reports that the Melange site keeps going 505 and crashing, I took at look at why. Turns out several spam crawlers were going mental and repeatedly adding tickets with spam links. Around 600 tickets and thousands of comments later, the process decided it had enough and terminated.

I've deleted all tickets (even the valid ones were spammed into oblivion) and turned off comment creation and modification for anonymous users. A look on the Trac Wiki shows that there are some SpamFilter extensions being developed which I'll investigate at some point.

Hans Rosling at TED last year on global development

anil@recoil.org (Anil Madhavapeddy) — Sun, 01 Apr 2007 23:36:00 +0100

Having coffee with AliB and discussing lecturing reminded me of one of the best presentations I've ever seen: Hans Rosling at TED and debunking myths about the developing world.

Even if you're not interested in the subject, it's worth checking out just to see how cool his presentation of otherwise rather boring statistics is. His company, Gapminder also got bought by Google a few weeks ago, so chances are Analytics will get more interesting soon.

Melange hits the euro-spotlight

anil@recoil.org (Anil Madhavapeddy) — Sun, 01 Apr 2007 23:12:00 +0100

It's been a busy old March (release management of the new XenEnterprise sucked up most of it). I did take a break and go over to EuroSys 2007 in Portugal to present the language and compiler I implemented as part of my PhD work (read it here).

Although the talk I gave was was a bit underwhelming (more preparation and time-practise next time!), I met a whole bunch of really interesting people. My argument about rewriting whole applications also didn't get laughed out the room as I thought it might, as people recognise that retro-fitting safety enhancements on existing languages is a bit of a dead-end road to go down. It has definitely inspired me to make more time to spend on polishing up the Melange applications for a proper release in 2007.

In a pleasant surprise, it also won the Best Student Paper award of the conference as well!

Deens, welcome to the Internet!

anil@recoil.org (Anil Madhavapeddy) — Sat, 30 Dec 2006 01:11:00 +0000

Inspired by finishing my PhD corrections (!) today, I decided to hook up the DNS server from our Melange project up to the Internet. The authoritative server is called deens (since the co-author is one Tim Deegan, geddit?), and is written in pure OCaml.

This is all rather experimental, to put it mildly, but I stuck in the zone file below, hooked it up as a delegate to our main name-servers, checked it against the DNS Report, and it all seems to be working!

$ORIGIN deens.recoil.org. ;
$TTL    240
deens.recoil.org. 604800 IN SOA  (
    deens.recoil.org. anil.recoil.org.
    2006122401 3600 1800 3024000 1800
)
        IN  NS     ns1.deens.recoil.org.
        IN  NS     deensns.recoil.org.
ns1     IN  A      194.70.3.132
dynamic IN  CNAME  dynamic.recoil.org.
static  IN  CNAME  static.recoil.org.
anil    IN  CNAME  dynamic
stats   IN  CNAME  dynamic

I also modified stats.recoil.org to be an alias to stats.deens.recoil.org, so all the requests for that domain will go via the deens setup. You actually need a user/pass to access the site, but that doesn't matter; if it gets that far, the DNS bit has worked.

There's still an awful lot of tedious work to get the server into a production-ready state, such as proper logging, more error handling and recovery, etc., but I really hope to find the time in 2007 to polish this up somewhat. Performance is excellent already; faster than BIND by quite a lot, and it can optionally use more memory to cache responses to shoot up to crazy levels.

Incidentally, the dig replacement utility also seems to be working fairly well, and David Scott has been messing around with a Bonjour implementation that will get finished sometime in 2007 as well (honest!).

Google Webmaster tools

anil@recoil.org (Anil Madhavapeddy) — Thu, 28 Dec 2006 15:04:00 +0000

The conversion of the Recoil web services to external FastCGI pinned our Trac installation at Melange as the source of the CPU hogging. It turned out the Google crawler was indexing the entire source tree via Trac, causing it to go ballistic.

I then stumbled on the latest cool Googlism: the Google Webmaster Tool, which lets you register your sites and displays options, diagnostics and statistics about how the Google crawler views your website. I turned down the frequency at which Google hits the Trac installation (as well as installing a suitable robots.txt file). This solved the immediate problem, but some of the search statistics were fun to check out as well.

It turns out the gallery is pretty highly ranked for image searches. My trips to Japan seems to have made it big, with popular searches including "Shibuya", "tokyo at night", and "japanese roof". My random pictures of indian buffaloes, smoggy skylines and fried ice-cream seem especially popular as well. It's a wierd old Internet eh?

The gallery has fallen a bit by the wayside in recent months. I'll update it when I get back to Cambridge!

Mercurial FastCGI module

anil@recoil.org (Anil Madhavapeddy) — Wed, 27 Dec 2006 20:59:00 +0000

Our lighttpd setup has been very unstable in recent months, probably brought on by the load of the large Mercurial repositories hosted on Recoil since the Google Summer of Code mentoring.

The source of the instability was really hard to track down, but it seems to be the automatic spawning of FastCGI processes by the web-server, and lighttpd failing to handle a SIGCHLD somewhere when a child process crashes. To sort this out, I just converted all the Ruby on Rails setups (this blog and Nick's) to use an external spawn.

This only leaves our Mercurial vhost hg.recoil.org to switch to using FastCGI, and I couldn't find a module for this anywhere and so lashed up some Python glue to do the job.

You can download the small distribution for Mercurial 0.9 (hg-fcgi-0.9.tar.gz). It has a FastCGI library written by someone else, the Python files to glue the Mercurial and FastCGI libraries together, and a simple rc script to launch the external web process.

Instructions are for lighttpd... install the Python files somewhere, modify them to point to the Mercurial directory, run the rc script to start the daemon, and then add something similar to the following to your lighttpd config file:

fastcgi.server = (
  ".fcgi" => ( "localhost" =>
    ( "socket" => "/var/cache/fcgi/sites/hg.recoil.org/dirsock" )),
  ".hg" => ( "localhost" =>
    ( "socket" => "/var/cache/fcgi/sites/hg.recoil.org/sock" )),
)

Also add "index.fcgi" to index-file.names in the config file, and touch it in the vhost directory to create an empty file (this is to avoid getting a 404 error and instead pass it through to the FastCGI process). Similarly, touch a .hg file for every repository you want to serve. You could do this differently by passing through a URL prefix and modifying the Python appropriately, but I prefer finer control over what we're serving.

Hope this is useful; I won't bother submitting it back to the Mercurial list as it looks like the official hg repo has a different code layout; I'll check it out later on when I have a bit more time and integrate properly.

I have no idea whether or not this will actually improve our stability, but it's at least easier to move onto a different web-server now that everything is FastCGI. All I need now is an OpenBSD/php5-fastcgi port, which doesn't seem to exist (yet).