Blimey, I got promoted to a full professor at Cambridge this week (glamorously known as Grade 12 in Cambridge parlance), and had it announced in the Cambridge Reporter! My first thought when I got the news was how much fun this job has been over the years! As a kid, I always wanted to be a bus conductor (a job that increasingly no longer exists)...
I have nothing but gratitude to my community that has made this all so much fun. A quick count across my papers shows 295 coauthors across 142 papers, but the vast majority of this diversity has been since I pivoted with Srinivasan Keshav to environmental science back in 2021. Getting promoted wasn't a sure thing at all after I did that, since it's a bit odd from a CV perspective to flit between subjects mid-career.
Luckily (aside from never actually caring about having a conventional 'career'), I've had incredible senior academic mentors support me. Jon Crowcroft takes the lead with a remarkable instinct that usually makes him eventually correct but never at the expense of joy, my officemate Simon Peyton Jones has endless enthusiasm for encouraging me to follow my instincts and not just keep doing the same thing; and several formative chats with Phillipa Gardner where she connected the dots between computer/conservation science just enough to make me feel like I wasn't diving entirely into the unknown.
My top accolade, of course, is to all the wonderful students that make the academic job a perpetually dynamic and engaging one. Pembroke College has been a wonderful home from which to explore academia alongside my department. I can't imagine being in a University that doesn't have the College/department split like we do here, as it ensures we're exposed to a barrage of diverse ideas from different fields all the time.
But anyway, onto more pragmatic matters: Tessera v1.1 is out and I've been refreshing io_uring and Eio to move all its data around; I got cut off from Fable and went local with a DeepSeek library; got self-hosted email feedback; and a never-ending review pile and usual fun links.
1 Tessera v1.1 released
We released TESSERA v1.1 this week as a retrained, drop-in successor to v1.0. Downstream code just points at new embedding tiles and should see its accuracy tick up for free, and the headline improvements are fewer artefacts in low-observation areas (v1.0 used to show grid-like seams where Sentinel-1 ascending/descending coverage met and one side had three times the valid observations of the other; this still exists in some regions of the world but is much better in v1.1); better year-on-year temporal stability which makes change detection and cross-year classifier transfer work better; and an expanded 20km coastal buffer that finally brings small islands and coastlines into coverage.
Read the details elsewhere; we're now generating embeddings for two model variants and so I've updated the live coverage map to track v1.0 and v1.1 side-by-side so you can see exactly which tiles exist for each year:
Going global with multiple models has surfaced its own crop of artefacts as we can now compare generations, but they're markedly less severe and we're continuing to grind them out. I released geotessera 0.9 and also uploaded the model weights to Hugging Face, all under CC0.
1.1 More io-uring and Eio work
Moving all those embeddings around at scale is a lot of bulk I/O, which gave me the perfect excuse to keep hacking on the ocaml-uring I started last week. Thomas Leonard is hacking away on Eio alongside me and Patrick Ferris so this is a good time to get the feature queue into shape. Most is unglamorous plumbing unless you're into surfing Linux syscall tables, but it's the good kind:
- PR #151 lets us set
CLOEXECwhen uring creates a socket, so eio#843 can move that path fully onto uring rather than dropping back to a plain syscall. Close-on-exec is a weird default in the modern world, so always setting it is a good idea. - PR #154 threads per-operation
preadv2/pwritev2flags through the submission path. I was rooting around in liburing and noticed thatRWF_DSYNClets us fold anfsyncinto the preceding write, andRWF_NOCACHEinEio.Flow.copycan stop bulk copies from trashing the page cache. Both of these will in turn help add better batched copy support directly into Eio for things like shifting Zarr files around. - PR #155 sorts out vectored
readv/writevfixed-buffer stubs which were misnamed before. I need to knock up a benchmark suite to figure out why fixed buffers are underperforming so much, but it would help to actually do vectored IO into them! - A bigger change is PR #153 which switches the underlying buffer type from bigarray/cstruct over to plain
bytes. It's a surprisingly small diff since Sadiq Jaffer pointed out that bytes in the OCaml 5 runtime never actually relocate unless they're small (<2KB). Patrick Ferris had also done a similar experiment. I think this will need to rejig who actually allocates the buffers, since IO memory needs to be caller allocated where possible.
On the Eio side, the work is mostly about getting these new capabilities up into a portable form where applications can actually reach them:
- PR #850 adds
ftruncate/fdatasyncand routesfsync,shutdownandsocketthrough uring instead of the blocking fallbacks, so the whole file path stays on the shared ring. It's just generally useful to avoid syscalls, although it would be unusual to have any of these be a bottleneck (vs read/write). - PR #854 propagates the real
errnoback out of the "fork actions" C binding intoEio.Process, so a failedexecraises a preciseUnix.Unix_error. This in turn makes it much easier to add pseudoterminal support, which I started back in 2023 (!) for Ryan Gibb and might actually finish this year. - Still in flight is another longstanding PR in the form of
Eio_unix.Sockoptmodule for getting and setting socket options, so things like capnp-rpc can control TCP behaviour better.
Thomas Leonard has meanwhile been simplifying the scheduler around by moving the fixed-buffer read/write retries out of the scheduler and coping gracefully when there isn't enough lockable memory for the ring itself. None of this is glamorous, but it's the bedrock on which a lot of my OCaml code depends, so I'm very happy to put the effort into getting my piece of it right.
2 Going hyper local with in-process LLMs
I was midway through an agent-assisted sweep of the OCaml runtime with Claude Fable when it abruptly cut out on me. The suddenness was enough to get me serious about trying some open-weights models I can run myself.
I bound Antirez' Dwarfstar DeepSeek-v4 engine into OCaml 5 and Eio as an ordinary library, and built a small agent up from primitives on top of it. I wrote it up in its own post, but the TL;DR is that it works surprisingly well for local customisation. The model is just a plain OCaml function call, its tools are ordinary OCaml functions, and there are no serialisation woes with REST endpoints, MCP servers or external services anywhere.
Since that post, I have been surprised to see how many weird things I can do with it: for example, I've currently got a process that's using OCaml's runtime events to introspect itself and use the builtin LLM to modify its own GC parameters. The LLM can understand some pretty low-level system concepts...
3 Self-hosted email feedback and DKIM2
My self-hosting email post from a couple of weeks ago is drawing surprisingly useful feedback. David Verdin pointed out that the DKIM signing (which I find really complicated) is going through a revision.
DKIM2 is in IETF draft form, with the headline change being that every hop in the email delivery chain now signs the message rather than just the originating server. That fixes two things that make DKIM(1) brittle: the signature breakage you get when a forwarder or mailing list touches a message, and replay attacks where a legitimately-signed message is re-spammed to millions.
The latest spec draft arrived last month and ensures that existing keys stay compatible. It's still (like JMAP) not widely deployed yet, with big mailbox providers promising the end of the year. I'm pretty glad to see email continues to maintain a story for independent hosting!
I also got pinged by the founder of CoMail, which is a cooperative ATProto based email service that uses the social database to verify identity instead. This seems related to some of the thoughts on Tangled vouching from a few weeks ago, as that also uses the ATProto.
4 Fun and not fun links
I wrapped up my Royal Society International Newton panel reviews and my share of Cambridge exam marking. But then I had to roll straight into reviewing for EuroSys and their resubmission round (since I reviewed earlier versions of the paper in a previous iteration of the conference) and also get new papers dropping in from editors in remote-sensing journals. I don't begrudge any of it individually, of course, but something has to change about peer review to make it more sustainable in the coming years. Or maybe we'll just learn not to sleep.
On more fun topics:
- Josh Bressers argues that we have to change the rules of security (even with Mythos offline) since security teams are drowning in CVEs and alerts. The only lever left is choosing what not to respond to as time-gating and not updating too fast came up last week from the ExploitGym benchmark as well. Maybe my local LLM agents will mutate their way into protecting themselves...
- A lovely history of how photosynthesis evolved in Quanta last week, told through a newly-discovered Panamanian bacterium whose paddle-shaped light-harvesting machinery hasn't changed in billions of years. The article is also beautifully laid out.
- I continue to greatly enjoy Amol Rajan's podcasts and the latest one on "The End of Endless Growth: Should We Put the Brakes on Economic Expansion?". This doesn't promote degrowth (which will never happen and is probably more harmful than good to individuals) but rather has an excellent discussion with Kate Raworth on Doughnut economics and planetary boundaries.
- Welcome Lachlan Kermode to the addiction of weeknotes.