I'm almost set for the start of teaching term now and meeting our newest intake and catching up with everyone after a year away! My 1A Foundations of CS starts next week, and I've decided to lock in the use of Forester for this. For the first time, we won't be giving out the traditional printed notes. I'm a little worried about not giving away since I've seen students scribbling on them well into their second years in the past! I'm hoping that the cross-referenced Forester will be more friendly to the tablets and notebooks that students are all using these days.
1 GeoTessera 0.11.0 now has wall-to-wall embeddings!
As its start-of-term here, David Coomes Sadiq Jaffer and I will give an intro to Tessera talk at the CRI next week, so pop along if you're in town!
I also shipped geotessera v0.11.0, which now provides wall-to-wall embeddings for anywhere in the world from 2017-2025, a major first for the project! Yay! See the release notes above for details.
There are now two separate full copies of the v1.1 embeddings available, in either Icechunk or Zarr format, with the choice depending on your particular usecase. This has been a (validly) confusing point for early users, so here's an explanation of the innards.
1.1 Icechunk for bulk streaming
First, the dClimate team built a custom inference stack that uses Icechunk, and publishes this in the s3://tessera-embeddings bucket on AWS.
This has been optimised for 'streaming access' with relatively large shard and chunk sizes to minimise round-trip traffic.
However, it can only be accessed using Icechunk-compatible bindings since although Icechunk has a Zarr-compatible store interface, its on-disk (via HTTP) format completely differs from standard Zarr. Therefore, a client in something like OCaml that speaks the Zarr standard will not be able to interoperate with the Icechunk store.
1.2 Zarr v3 for smaller mobile-friendly chunks
This motivated us to transcode the Icechunk store to a separate pure Zarr v3 format. While we lost some cool management features available with Icechunk (like versioning and snapshots), we gained compatibility with clients that implement pure Zarr (like my own OCaml code for example). While we were here, I also changed the size of the chunking to be lower to make it more friendly for mobile streaming. This is what allows mobile JavaScript frontends like TZE to work.
Therefore as a user, with GeoTessera 0.11 you now have a choice between the two:
- The Zarr store on Source Cooperative is the default for streaming reads.
- Pass an
.icechunkURL toGeoTesseraZarr()instead if you want the Icechunk repository directly. The production store is ats3://tessera-embeddings/v1.1/dclimate.icechunk. I'll expose this as a builtin option in a future geotessera.
1.3 Bypassing the Source Coop proxy for direct S3
One reason that many users saw bad performance initially is that the Source Coop has an additional caching layer over Amazon via a Rust proxy (data.source.coop), and this appears to be throttling rapid connections in a way that directly going to S3 isn't. GeoTessera 0.11 now reads straight from the S3 bucket by default to avoid putting any undue pressure on the poor sole proxy server that Source Coop is kindly running for us, with the gateway still available as an opt-in in the geotessera CLI via --via-gateway if you want its Cloudflare edge caching.
Phew! There's a lot going on here with where the embeddings are stored, but I hope we're hitting the right notes as to how to make them as maximally accessible as possible. My next step here is to publish the HTTP perma proxy I wrote a few months ago for caching Zarr tiles, in order to provide a gateway proxy in Cambridge that stashes commonly used years locally to avoid lots of network traffic.
2 Going to the Other Place to talk at the Intelligent Earth CDT
I was invited by the enthusiastic crew to give a talk on on "Technology for Living Evidence" (my overall name for the work we're doing on Tessera, habitat and species mapping and evidence ) to the new student cohort at the Intelligent Earth CDT in Oxford.
The venue was the incredibly impressive new Centre for Life and Mind, which opened about a year ago and still had that 'new building' smell. I was told it cost a cool ~£200m and is Oxford's largest-ever building project, except for certain other hilariously bad bridges. I'm told it supports 1,400 researchers, >1,000 undergraduates a year, and has shielded EEG rooms, eye-tracking labs, and even a sleep laboratory tucked away somewhere!
The Intelligent Earth CDT is a UKRI-funded initiative that'll train 100 PhD students years across climate, biodiversity, natural hazards, and environmental solutions. This is temporally a follow-on to our own AI4ER program here that's drawing to a close now, so it was also excellent to have Yihang She along from our group to act as a bridge between generations of students!
The director, Philip Stier, was a brilliant host and also tutored me on his expert topic of the physics of clouds. He recently built a flow-matching model that downscales high-resolution (~1km) cloud structure from the coarse (~25km) fields that climate models calculate.
2.1 The Oxford-Cambridge rail certainly isnt moving very fast
Getting to Oxford was a nightmare as usual, as the only useful options were trains via London (~3 hours but the line was down), a bus via Bedford (~2.5h), or motorbiking down the single-carriageway A421 (~2h "on a good day"). As others have noted the East-West railway can't come soon enough:
Today, travelling between Oxford and Cambridge by public transport is a joke. [...] With East West Rail, that journey time drops to ninety minutes. [...] This means a researcher in Cambridge can collaborate with a lab in Oxford without losing a day to the journey
Despite the Varsity line dating back well into the 19th century, the campaign to reinstate has been going on for 30 years (!), and under East West Railway Company's revised April 2026 proposals the full end-to-end Oxford–Cambridge service isn't scheduled until the mid-to-late 2030s. Absolute snail's pace progress here...
3 Scrutineer fixes keep rolling
InfoQ picked up my 'rumour is the exploit' post, on how AI agents can basically find working exploit faster than traditional embargoed disclosure can keep up. In the meanwhile, Scrutineer continues to be extremely successful at finding issues, and Hannes Mehnert and Edwin Torok continue to fix many of them (thank you!!). I'm going to catch up on my Patch the Planet planning backlog later as I've been extremely busy preparing for the start of term.
In terms of where activity has been focussed, mirage/awa-ssh joins the list of repos being fixed, with a nonce fix, a CPU/memory fix and a required-authenticator fix. Elsewhere I spotted ocaml-tls's signature-algorithm fix, happy-eyeballs's empty-address fix, more Solo5 bounds checks (#682/#684 closed, #683 carried over, plus new dev-type and note-size checks).
4 The LIFE tour, and offsetting ourselves
I built an interactive LIFE tour using an ensemble of agents narrating and animating the LIFE metric over the live Zarr store. This was worryingly quick and good, and with several reactions on the socials like LinkedIn.
This ranged from Alison Eyres being very impressed, to Chris Sandbrook asked how the process actually worked and worrying about the implications. It's fair to say that I am worried as well; I'm feeling awe at the one-shot quality, intrigue at the educational potential for personalisation, and a little shock at the implications for jobs down the line. Will we end up needing to 'offset' our use of AI to support human endeavour, the way we do for carbon credits?
5 Released a new OCaml mdx to unblock Eio dev
I released mdx 2.7.0 to add support by Thomas Leonard for a new mdx_skip directive and shonfeder's dune 3.24 support fix. Thanks to a Windows VM that Mark Elvers setup for me I no longer need a physical Windows desktop, which has made me much less grumpy. I've resumed working on Eio's Windows process spawning and nt_path normalisation, as well as dusting off my changes to the IOCP bindings. This is the 'relaxing' hacking of the week, but may need to be put on ice as term starts!