Posted by Anil Madhavapeddy
Sat, 30 Dec 2006 01:11:00 GMT
Inspired by finishing my PhD corrections (!) today, I decided to hook up the DNS server from our Melange project up to the Internet. The authoritative server is called deens (since the co-author is one Tim Deegan, geddit?), and is written in pure OCaml.
This is all rather experimental, to put it mildly, but I stuck in the zone file below, hooked it up as a delegate to our main name-servers, checked it against the DNS Report, and it all seems to be working!
$ORIGIN deens.recoil.org. ;
$TTL 240
deens.recoil.org. 604800 IN SOA (
deens.recoil.org. anil.recoil.org.
2006122401 3600 1800 3024000 1800
)
IN NS ns1.deens.recoil.org.
IN NS deensns.recoil.org.
ns1 IN A 194.70.3.132
dynamic IN CNAME dynamic.recoil.org.
static IN CNAME static.recoil.org.
anil IN CNAME dynamic
stats IN CNAME dynamic
I also modified stats.recoil.org to be an alias to stats.deens.recoil.org, so all the requests for that domain will go via the deens setup. You actually need a user/pass to access the site, but that doesn't matter; if it gets that far, the DNS bit has worked.
There's still an awful lot of tedious work to get the server into a production-ready state, such as proper logging, more error handling and recovery, etc., but I really hope to find the time in 2007 to polish this up somewhat. Performance is excellent already; faster than BIND by quite a lot, and it can optionally use more memory to cache responses to shoot up to crazy levels.
Incidentally, the dig replacement utility also seems to be working fairly well, and David Scott has been messing around with a Bonjour implementation that will get finished sometime in 2007 as well (honest!).
Posted in research, hacking, net | no comments
Posted by Anil Madhavapeddy
Thu, 28 Dec 2006 15:04:00 GMT
The conversion of the Recoil web services to external FastCGI pinned our Trac installation at Melange as the source of the CPU hogging. It turned out the Google crawler was indexing the entire source tree via Trac, causing it to go ballistic.
I then stumbled on the latest cool Googlism: the Google Webmaster Tool, which lets you register your sites and displays options, diagnostics and statistics about how the Google crawler views your website.
I turned down the frequency at which Google hits the Trac installation (as well as installing a suitable robots.txt file). This solved the immediate problem, but some of the search statistics were fun to check out as well.
It turns out the gallery is pretty highly ranked for image searches. My trips to Japan seems to have made it big, with popular searches including "Shibuya", "tokyo at night", and "japanese roof". My random pictures of indian buffaloes, smoggy skylines and fried ice-cream seem especially popular as well. It's a wierd old Internet eh?
The gallery has fallen a bit by the wayside in recent months. I'll update it when I get back to Cambridge!
Posted in travel, recoil, net | 2 comments
Posted by Anil Madhavapeddy
Wed, 27 Dec 2006 20:59:00 GMT
Our lighttpd setup has been very unstable in recent months, probably brought on by the load of the large Mercurial repositories hosted on Recoil since the Google Summer of Code mentoring.
The source of the instability was really hard to track down, but it seems to be the automatic spawning of FastCGI processes by the web-server, and lighttpd failing to handle a SIGCHLD somewhere when a child process crashes. To sort this out, I just converted all the Ruby on Rails setups (this blog and Nick's) to use an external spawn.
This only leaves our Mercurial vhost hg.recoil.org to switch to using FastCGI, and I couldn't find a module for this anywhere and so lashed up some Python glue to do the job.
You can download the small distribution for Mercurial 0.9 (hg-fcgi-0.9.tar.gz). It has a FastCGI library written by someone else, the Python files to glue the Mercurial and FastCGI libraries together, and a simple rc script to launch the external web process.
Read more...
Posted in hacking, net | no comments
Posted by Anil Madhavapeddy
Wed, 27 Dec 2006 00:01:00 GMT
The switch to qpsmtpd does seem to have reduced my spam intake somewhat, so out of curiousity I looked at the statistics from 2 years of procmail logs to see what's been happening in terms of filtering effectiveness.
A quick import and bug-fix of Log::Procmail into OpenBSD, and some lashed up Perl and gnuplot later, the graph on the right showed up. The red and green are ham and spam respectively, as classified by SpamAssassin.
The large amount of ham in 2004 was not actually real mail, but mostly postmaster bounces from forged spam; I am currently forced to destroy all domain bounces without even reading them due to the sheer volume. This is something that Sender Permitted From promises to help solve once we determine if any our users send @recoil.org mail from sources other than our mail server.
Since the turn of this year the amount of spam has jumped, but more concerningly, SpamAssassin has been missing increasing amounts, and it's been flowing through straight to my Inbox (despite sa-update running daily). I'm going to do these graphs again in a few months and see just how much the switch to the new paranoid SMTP has helped.
Posted in recoil, net | no comments
Posted by avsm
Fri, 08 Oct 2004 12:49:45 GMT
While assisting various graduating PhD students (lucky for some) with their resumes, I ran across the sublime urban legend that is Hugh Graham's "College Essay". I'm now inspired for future job applications! Here is a taster of the first paragraph:
I am a dynamic figure, often seen scaling walls and crushing ice. I have been known to remodel train stations on my lunch breaks, making them more efficient in the area of heat retention. I translate ethnic slurs for Cuban refugees, I write award-winning operas, I manage time efficiently. Occasionally, I tread water for three days in a row.
Read on...
|
Posted in humour, net | no comments
Posted by avsm
Thu, 07 Oct 2004 17:14:05 GMT
Just stumbled across a beta of Clusty, which is a pretty good search engine in its own right (not as minimal as google, but still usable).
The novel thing about Clusty is that it automatically clusters searches into groups to help narrow down the search. So searching for my name brings up a bar on the left with categories such as "OpenBSD", "High Energy Magic", etc. Not bad!
It's a pity that I'm kind of locked into google now, just by virtue of the Safari toolbar not having an easy option to remap the search engine to use. It does appear people have started hacking Safari though, so perhaps a Clusty bar isn't too far off!
Posted in net | no comments
Posted by avsm
Thu, 29 Jul 2004 08:53:48 GMT
The amount of spam sent to Recoil accounts has dramatically sprung
up over the last few years, sending the machine loads skyrocketing
accordingly. Luckily, we're running
OpenBSD, which added a fun
tool called
spamd(8)
a couple of releases ago.
It's activated by tracking IP addresses of known
spammers from blacklists like Spamhaus,
and redirecting them to the spam daemon via pf rules. Once the mail reaches spamd, it "tarpits"
it by dropping its TCP send and receive buffers to a very small value,
encouraging the spammers and virii to (slowly) send their malware on. If they
ever do reach the end of their data, it then rejects it with a temporary
failure - costing the spammers more resourcs if they decide to retransmit it.
The load has dropped quite a bit since I activated this filtering; it seems
to help against some of the latest worms quite a lot, which just
connect to port 25, spew off a buffer-overflow attempt, and repeat this
once every few seconds. Since spamd, things take a bit longer though!
quick spamd: 221.2.232.138: connected (9/9), lists: spamhaus
quick spamd: 221.2.232.138: disconnected after 431 seconds. lists: spamhaus
Very satisfying. I did play with the greylisting mode of spamd as well, but it wasn't quite as successful as some valid mail sites such as EDAS (bless its underwhelming soul) take five days to send conference paper rejections into a greylisted system. Public whitelists do exist, but I think I'll wait a while and see if things mature a little more first.
Posted in hacking, net | no comments
Posted by avsm
Thu, 07 Aug 2003 23:22:00 GMT
In what could be a silly move, I stumbled across the Friendster web-site, and joined up. It works on the famous six degrees of separation principle, which means that it takes very few hops to know someone else in the world.
Imagine my amusement when it turns out that Nick is already a member ... I should have guessed :-) If you join up, do invite me - I'm really curious to see how this network works out. They seem to be "good guys" and promise not to use the information they collect for nefarious purposes.
Posted in net | no comments
| 
