Am back in Ireland again this week, after a quick stint back in Cambridge to catch the 90% eclipse of the heart. The major preprint that went out this week was on weather downscaling with TESSERA embeddings. I also did a chunk of teaching prep (October is coming up fast!) and started porting Foundations of CS to Forester.
The week itself then went on getting TESSERA 1.0 onto Source Cooperative in Zarr with global RGB previews to go with it, putting up a public website for Evidence TAP, teaching the paper downloader some manners, and pointing a VLM at the PDFs. There's some OCaml hacking and fun links at the end.
1 Tessera dons the mask of Zarro
We finally have TESSERA 1.0 fully on source.coop in Zarr format, with 1.1 well on its way! Since we finally have enough disk space to stage everything thanks to source.coop, I also generated global RGB preview pyramids. You can now use TZE to zoom in and out of a dimensionality-reduced view of the 128 embedding dimensions in glorious technicolour:
And of course you can keep zooming in the browser and inspect individual shards:
The RGB previews make some of the inference artefacts obvious; just look for the checkerboarding and the banding across the Amazon. This doesn't actually affect downstream inference accuracy too much, but it's very ugly. Luckily this is all fixed in TESSERA v1.1 and even more so in v2.0, and I can't wait to see those previews to compare next week!
On the client side, geotessera#357 switches client downloads over to source.coop for the npy layout as well as Zarr, and picks up our new dataset variants like v2-2B-L~beta1 so that users can choose between v1/1.1/2.0~beta easily. The Zarr portion of the PR follows the layout I settled on earlier in the year. I'll release all this in a new version of geotessera early next week.
The giant transcode of the embeddings finished after days (and hundreds of terabytes) of back and forth between our compute cluster in Cambridge and the AWS S3 store. At the nth moment, Mark Elvers and I decided to rename some directories and rediscovered that S3 doesn't do renames, which led to another day of delay while six million individual server-side copies happened. Mark found that a plain aws s3 cp stays entirely server-side, whereas specifying a CRC64NVME checksum would have dragged 400TB back through Cambridge. Ho hum...
2 Papers and teaching
Pedro Sousa put his first preprint online on how "Earth observation embeddings are effective sub-grid descriptors for probabilistic weather downscaling". This is joint work with Sadiq Jaffer, Robin Young, Will Tebbutt and Rich Turner, and since its my first foray into weather prediction I wrote a longer splainer on how weather downscaling works and why a frozen annual TESSERA embedding beats handcrafted terrain descriptors.
I also ported 1A Foundations of CS over to Forester and the lecture notes are now a forest of transcluded trees with stable URLs, and there's also an OCaml toplevel compiled into the browser with js_of_ocaml so every code transcript in the course is runnable (and editable) in place.
Also a reminder that ICFP 2026 kicks off on the 24th of August in Indianapolis, where Ryan Gibb will be presenting his package management calculus paper. Do say hi to him if you're there!
3 Evidence TAP goes public
There's a website up now at evidencetap.org! It's a very barebones site, but it does list what's going on and who is involved, and we'll populate it more over the coming months (especially by Mélanie Gréaux, who wrote up a fantastic AI4Good trip report from her recent travels).
One important reason to get the site up is to have our text and data mining policy online, which we need in order to keep publishers informed about our intended use for the large corpus of papers we're downloading.
3.1 TAP TAP TAP, is this downloader on?
So why are we are hoovering up quite so many papers in the first place? Our vision for living evidence databases has to screen, appraise and extract from the full text of a study rather than just its abstract. We're pinning down inclusion criteria for systematic reviews, which (e.g. in conservation) might be what the intervention was, where, on what, etc. Doing that continuously across conservation and now education requires holding the full corpus here so that local-only models can read it without any of it leaking to a third party and also be reproducible.
Those ten million papers we've obtained are a mixture of closed- and open-access publisher XML and PDFs that still need converting to TEI XML. This involves a pool of Pub2TEI instances (each of which tops out at around 10k documents an hour) which put the backlog at about a fortnight. I added another 128-core box and it's now at a peak of 53,000 documents an hour. The backlog's now about two days, after which we can get on with metadata analysis.
The paper downloader itself is surprisingly complex because of all the per-publisher logic. Publishers throttle and block us regularly, with a variety of error codes. An HTTP 500 is now treated as a general "slow down": we wait five minutes by default and block all other requests to that origin before asking again. A 403's body is also now read and interpreted per-publisher.
Some sites simply refuse our machines until we're on an IP allowlist, so all fetch traffic now leaves through a SOCKS5 proxy on one stable, allowlistable IP address. The table below shows some of the reasons we need a custom fetcher at all. Each domain has its own request rate, authentication mechanism, and IP source, all easily customisable. We'll eventually need over 200+ publishers in here!
| Strategy | Gets | Authentication | Quirks |
|---|---|---|---|
| Elsevier | XML (full text) | X-ELS-APIKey/X-ELS-Insttoken headers |
Bad key = 401, unavailable = 403 |
| Springer (PDF) | Network IP | One request per 30s | |
| Springer (XML) | XML (JATS) | api_key query param |
Books answer 403 (access rights). Occasionally 404s instead of 500 |
| Wiley | PDF (TDM API) | Wiley-TDM-Client-Token header |
403 if not entitled to the content (books, old papers). Throttling is 500 rather than a 429 |
| TandF | Network IP | 403 if the IP allowlist expires | |
| PLOS | XML | None | Also available in bulk via allofplos. Super easy to fetch. |
| CUP | Network IP | 30s pacing per request, often times out | |
| CORE | Bearer token |
Search IDs ≠ output IDs, so the returned DOI needs verifying |
The winner by far is PLOS, which provides a simple Python allofplos library to fetch fulltexts.
3.2 Working on metadata quality
Our Taposaur pipeline converts as much of the fulltext to TEI XML as it can, but it does miss things in complex paper layouts. On a suggestion from Sadiq Jaffer, I've been experimenting with Nemotron Parse 2.0 alongside a Qwen 3.8 27B VLM to get high quality metadata and references out of even the older PDFs in our database.
My fledgling 'parsosaur' tool started with pure Nemotron to parse out bounding boxes. You can browse the output for our 2024 conservation horizon scan paper, which shows the PDF and the parsed TEI side by side. This works surprisingly well as even the magazine-style layout of Trends in Ecology & Evolution is matched reasonably accurately to the full text in the TEI, right down to the author affiliations. Mouse over either side and it'll match the scroll position of the other.
Nemotron does less well at interpreting complex diagrams. I chucked it at our Nature Climate Change paper on impermanence, whose figures are dense and multi-panel and it degenerated into a loop of near-identical cells, which at least it marked as unreliable:
Sadiq set me up a local Qwen 3.8 27B endpoint, and my initial experiments using the VLM to interpret diagrams when supplied with the full text of the paper are going really well. I'll try to cut a release of Parsosaur next week, as Shane Weisz also expressed interest in learning more about VLM usage for his Dash for Life dashboards.
4 OCaml hacking
Most of my OCaml time went into the infrastructure above, but a few things escaped into the wild.
4.1 Making Eio traces free again
While profiling Taposaur I've been adding custom events in various places. Thomas Leonard and Thomas Gazagnaire discussed on eio#914 that our OCaml runtime events allocate 23 words per event as soon as the runtime events shared memory ring is active. Since a lot of runtime events usage is about finding performance bugs, it's not ideal that the tracing calls themselves allocate memory.
My fix upstream in ocaml/ocaml#14984 is hopefully a simplification. There's a per-domain write-buffer cache for custom events that was formerly a list accessed through closures, and I changed it to a single atomic bytes slot held in domain-local storage. Now, the steady state doesn't allocate at all and the buffer is only checked out for the duration of the serializer call. Thanks to Sadiq Jaffer for for the review; I'm still not entirely sure I understand the interplay between the C and OCaml runtimes here, but the new version is at least simpler. The fix will be in OCaml 5.6 later this year.
4.2 Escaping the docroot in cohttp
More seriously, I've been triaging bug reports from Jane Street and with their help fixed a path traversal in cohttp's resolve_local_file, which could be escaped using urlencoded path components. cohttp#1145 urldecodes before resolving and also exposes a Cohttp.Path.normalise function so that anyone gating requests on a path prefix can apply the the same normalisation logic, rather than authorising a decoded path and then serving an undecoded one:
let callback _conn req _body =
let path = Cohttp.Path.normalise (Cohttp.Request.uri req) in
match String.split_on_char '/' path with
| "admin" :: _ when not (authorised req) -> Server.respond_not_found ()
| _ -> Server.respond_file ~fname:path ()
This affects cohttp-lwt, cohttp-mirage and cohttp-async, but not cohttp-eio, which has its own path resolution and uses sandboxed fd operations anyway to prevent traversal. I also experimented with stripping control characters as well, but after discussing with the reporter decided to back that out for compatibility but left the tests in place for a future major bump.
(I would ordinarily not talk about security issues until the fix is merged, but in the new agentic world just opening the PR is enough to reveal the existence of the issue, and so I figured getting more eyes on this early is now better while waiting for reviewers. If you use cohttp, please do inspect the patch.)
4.3 Prometheus and the opam-repository treadmill
Following on from the prometheus-1.4 release last week, Thomas Leonard merged prometheus#65 to move Lwt-specific logic out of the core package entirely.
Defining and recording a metric no longer pulls in any concurrency library, since it just involves updating an in-memory scoreboard. Mark Elvers has already pushed the migration through opam-health-check and ocluster, which is a good sign that the deprecation path works as advertised. Next up is the Eio backend as an independent package and the next major version should be good to go.
As many of the other maintainers are holiday, I also did a bunch of opam-repository merging, I also fixed oxcaml/opam-repository#59 to let upstream dune 3.24.2, eio 1.4 and mdx 2.6.0 through the guards so all three now build with OxCaml. My website monorepo continues to build on ox-minus39 so I can update this website, yay!
5 Fun links
- There's a brilliant mobile game on being a prime minister simulator, No 10: Full Confidence, which briefly ranked above Minecraft on the UK App Store chart (!). I managed two and a half years before being ousted by my own cabinet.
- Frank Feng sent on a report from China that they've been using in-house Moffett S40 Computing Card to accelerate Tessera inference significantly. I'd heard about these when I went with the Royal Society delegation to China in 2023, but it's the first time I'm seeing sparse-weight AI accelerators 'in the wild'.
- Fascinating article on "Why Are Rivers So Mathematical?" in Quanta. I've been thinking about how to model roads and rivers generatively using Tessera to help with flood predictions, so this article is well timed!
- It's been an insanely fast week of progress on local LLMs, with Qwen 3.8 performing incredibly, and I've been trialling Deepseek v4 Pro 813 as well. More on that next week!