Posted by Anil Madhavapeddy
Sat, 30 Dec 2006 01:11:00 GMT
Inspired by finishing my PhD corrections (!) today, I decided to hook up the DNS server from our Melange project up to the Internet. The authoritative server is called deens (since the co-author is one Tim Deegan, geddit?), and is written in pure OCaml.
This is all rather experimental, to put it mildly, but I stuck in the zone file below, hooked it up as a delegate to our main name-servers, checked it against the DNS Report, and it all seems to be working!
$ORIGIN deens.recoil.org. ;
$TTL 240
deens.recoil.org. 604800 IN SOA (
deens.recoil.org. anil.recoil.org.
2006122401 3600 1800 3024000 1800
)
IN NS ns1.deens.recoil.org.
IN NS deensns.recoil.org.
ns1 IN A 194.70.3.132
dynamic IN CNAME dynamic.recoil.org.
static IN CNAME static.recoil.org.
anil IN CNAME dynamic
stats IN CNAME dynamic
I also modified stats.recoil.org to be an alias to stats.deens.recoil.org, so all the requests for that domain will go via the deens setup. You actually need a user/pass to access the site, but that doesn't matter; if it gets that far, the DNS bit has worked.
There's still an awful lot of tedious work to get the server into a production-ready state, such as proper logging, more error handling and recovery, etc., but I really hope to find the time in 2007 to polish this up somewhat. Performance is excellent already; faster than BIND by quite a lot, and it can optionally use more memory to cache responses to shoot up to crazy levels.
Incidentally, the dig replacement utility also seems to be working fairly well, and David Scott has been messing around with a Bonjour implementation that will get finished sometime in 2007 as well (honest!).
Posted in research, hacking, net | no comments
Posted by Anil Madhavapeddy
Thu, 28 Dec 2006 15:04:00 GMT
The conversion of the Recoil web services to external FastCGI pinned our Trac installation at Melange as the source of the CPU hogging. It turned out the Google crawler was indexing the entire source tree via Trac, causing it to go ballistic.
I then stumbled on the latest cool Googlism: the Google Webmaster Tool, which lets you register your sites and displays options, diagnostics and statistics about how the Google crawler views your website.
I turned down the frequency at which Google hits the Trac installation (as well as installing a suitable robots.txt file). This solved the immediate problem, but some of the search statistics were fun to check out as well.
It turns out the gallery is pretty highly ranked for image searches. My trips to Japan seems to have made it big, with popular searches including "Shibuya", "tokyo at night", and "japanese roof". My random pictures of indian buffaloes, smoggy skylines and fried ice-cream seem especially popular as well. It's a wierd old Internet eh?
The gallery has fallen a bit by the wayside in recent months. I'll update it when I get back to Cambridge!
Posted in travel, recoil, net | 2 comments
Posted by Anil Madhavapeddy
Wed, 27 Dec 2006 20:59:00 GMT
Our lighttpd setup has been very unstable in recent months, probably brought on by the load of the large Mercurial repositories hosted on Recoil since the Google Summer of Code mentoring.
The source of the instability was really hard to track down, but it seems to be the automatic spawning of FastCGI processes by the web-server, and lighttpd failing to handle a SIGCHLD somewhere when a child process crashes. To sort this out, I just converted all the Ruby on Rails setups (this blog and Nick's) to use an external spawn.
This only leaves our Mercurial vhost hg.recoil.org to switch to using FastCGI, and I couldn't find a module for this anywhere and so lashed up some Python glue to do the job.
You can download the small distribution for Mercurial 0.9 (hg-fcgi-0.9.tar.gz). It has a FastCGI library written by someone else, the Python files to glue the Mercurial and FastCGI libraries together, and a simple rc script to launch the external web process.
Read more...
Posted in hacking, net | no comments
Posted by Anil Madhavapeddy
Wed, 27 Dec 2006 00:01:00 GMT
The switch to qpsmtpd does seem to have reduced my spam intake somewhat, so out of curiousity I looked at the statistics from 2 years of procmail logs to see what's been happening in terms of filtering effectiveness.
A quick import and bug-fix of Log::Procmail into OpenBSD, and some lashed up Perl and gnuplot later, the graph on the right showed up. The red and green are ham and spam respectively, as classified by SpamAssassin.
The large amount of ham in 2004 was not actually real mail, but mostly postmaster bounces from forged spam; I am currently forced to destroy all domain bounces without even reading them due to the sheer volume. This is something that Sender Permitted From promises to help solve once we determine if any our users send @recoil.org mail from sources other than our mail server.
Since the turn of this year the amount of spam has jumped, but more concerningly, SpamAssassin has been missing increasing amounts, and it's been flowing through straight to my Inbox (despite sa-update running daily). I'm going to do these graphs again in a few months and see just how much the switch to the new paranoid SMTP has helped.
Posted in recoil, net | no comments
Posted by Anil Madhavapeddy
Mon, 25 Dec 2006 23:35:00 GMT
It's Christmas Day, I've eaten far too much, and am lounging around doing the now-traditional Annual Recoil Cleanup as the year's todo list has grown ever larger. I've been meaning to switch from our venerable qmail-smtpd for some years now, and finally made the move over to qpsmtpd.
qpsmtpd is a drop-in replacement for the SMTP portion of qmail, and is written in Perl with a number of plug-ins which lets us increase our paranoia levels considerably. It's a pity we have to do this, but the policy of 'accept anything' has been under increasing stress for the last few years, and when I looked at my e-mail stats last night, I realised over 99.99% of my incoming e-mail was some kind of virus or spam. Even a 1% miss rate on SpamAssassin is enough to chuck 100s of mails into my inbox!
So now the new e-mail setup at Recoil includes virus scanning via the wonderful clamav, reverse DNS RBL looksup via rfc-ignorant.org, and even early-chatter detection of viruses which blindly blast messages before the initial SMTP greeting has completed. I'm hoping to enable global SpamAssassin checking soon if all else is stable and I don't get bleating about missing mail from our users.
I played with Greylisting as well to see if it had improved from my earlier experiments a couple of years ago. Unfortunately, it still looks as if there are many broken MTAs out there which don't cope well with rejection, and manual whitelists are required, which sounds a bit unreliable for setups like ours which sometimes don't get looked at for years on end (ahem).
So it's with a tear in my eye that I wave goodbye to qmail-smtpd, the first ever network-facing service deployed on Recoil back in 1998, and incredibly, the only one I've never had to upgrade in the 8 years since.
Posted in recoil | no comments
| 
