.plan-26-31: Sorting out Tessera and Evidence TAP infrastructure

I had a bunch of infrastructure work I had to get done this week, and so I went headsdown hacking to sort out both TESSERA and the Evidence TAP this week!

1 Moving a petabyte of TESSERA embeddings

Mark Elvers and I have been getting the full range of TESSERA embeddings up on Source Cooperative for the past few weeks. We split this latest work into two steps: first move the existing npy-based embeddings as-is, and then transcode to Zarr v3. This week the npy ones fully synched (almost a petabyte!) covering TESSERA v1 alongside selectively generated v1.1 and v2-beta ones.

The first half of my geotessera#337 PR switches us over to the data.source.coop endpoint for HTTP downloads, which lets us drop our S3-specific logic we had in geotessera 0.9. That code was weirdly problematic and complex; EC2 seems to drop connections quite quickly on bulk downloads, which is what most of our users do. My tests on the Source Coop proxy show Cloudflare R2 doing a great job of edge caching, so this looks like a solid long-term solution that's still using S3 under the hood, but not exposing that to our library users.

I'm also in the Source Coop Slack now, with extremely responsive developers on the other end. I'll cut a geotessera release with the npy support this week, with the Zarr conversion to follow shortly I hope!

1.1 A detour to investigate Icechunk

The Zarr conversion requires transcoding the npy sources over to a fresh Zarr store, so I spent some time exploring how best to do this.

The only alternative that's credible is Icechunk, Earthmover's transactional storage engine for Zarr that adds git-like version control over object storage. That's obviously very attractive for TESSERA, since we dynamically generate embeddings for a growing set of model variants and could really use version control. I knocked up a quick version of geotessera using the Icechunk Python library instead of Zarr's, which was a trivial drop-in since Icechunk speaks the Zarr Python interface.

Behind the scenes though, Icechunk writes to S3 in a custom format. That format seems rock solid in my testing, but is a blocker for us since we need client access from languages other than Python. There's an Icechunk Rust crate that isn't intended for external binding, a JavaScript version for browser use, but nothing I could bind (e.g.) OCaml against myself. There's a vibe-coded PR for a C API and Julia bindings, but it's not high quality enough so was rejected (rightly I think) by the Earthmover team. So I'll reluctantly come back to Icechunk when it grows a more stable FFI story, but won't use it just yet over a 'raw' Zarr v3 publish.

1.2 Transcoding to Zarr on AWS Fargate Spot

I went back to my Zarr v3 geoembeddings layout, with a static store on source.coop spanning 2017-2025 (extensible to future years, though not easily to earlier ones without using the aforementioned Icechunk). Finding the CPU to transcode ~10 million tiles without burning a fortune in egress bandwidth was surprisingly tricky! Running it in Cambridge via our current server (which we're trying to deprecate in favour of cloud hosting) would take around six months (!). After a number of failed experiments, I came across Amazon Fargate Spot, an order of magnitude cheaper way to run a Docker container in the same region as the embeddings (us-west-2) and serverless so we didn't have to manage all the VMs manually.

The rest of my geotessera#337 PR makes the zarr-fill cli stateless/incremental. It scans each zone first via Zarr to determine what's needed for a particular UTM zone, and also stores sampled metadata so the RGB pyramids are fast to calculate at the end. Without the samples we'd have to read the full petabyte back over the network just to run a PCA for the preview you see in tze.geotessera.org!

Mark did a quick trial run, and our spot instance goes about 100x faster than from Cambridge, so we're going to run every UTM zone in parallel for a rough cost of 1500 quid for the whole world (I'll find out in a few days if our back-of-the-envelope math is right or not). This compares well to the 50 grand bill from my original naive plan! Once TESSERA v1 is converted, v1.1 will be quick as it has fewer tiles as we've not done a full global run yet. In the future I'm aiming for all our embeddings generation to be Zarr-first, so getting this giant transcode out of the way will be a big boost.

2 Arise Taposaur and obtain the literature for Evidence TAP!

I've also been making inroads on the processing pipeline for Evidence TAP, now that we have more users. This is codenamed 'Taposaur', and has two major pieces: a custom downloader to deal with the (often ridiculous) hoops publishers make us jump through, and a metadata index over the millions of resulting PDFs that we've snarfed. My goal is to get this piece integrated with Sam Reynolds work on the Conservation Copilot this summer so it can have secure (and local model only) access to our fulltext paper database.

2.1 Getting PDF structured metadata with GROBID

Since we already have millions of downloaded papers from the last year's work, the first thing I cleaned up was the metadata processing. I'm building on three brilliant upstream projects biblio-glutton, GROBID and Pub2TEI, which all fit together to mirror public indices like Crossref and provide HTTP APIs that scan a PDF into structured metadata:

The 'TEI' here refers to an XML format for scholarly publishing that acts as an interchange format across all the publishers' slightly different schemas. I'll write more on this later, but here's a preview of the web interface:

The overall database runs from an OCaml scheduler that dispatches jobs over to a Docker Swarm running Grobid and the other services. It'll take about a month to churn through the full set, and outputs the TEI form into a nice local filesystem that I can then stash on our Ceph storage cluster.

The best thing about TEI is that is has a structured representation of much paper metadata. For example, I grabbed a random paper by Bill Sutherland about replacing bar charts with histograms and it has ORCIDs and even the corresponding author for the paper.

Then there's also the usual Crossref data, and because we have a local database, I can introduce overrides or evidence synthesis specific metadata quite easily for our own use. I'm very grateful to the Grobid team for open-sourcing their infrastructure; they are also working on some really cool open mining and science literature datasets that I'm excited to read more about later in the year.

2.2 A capability-based OCaml paper downloader

The PDF paper downloader itself is surprisingly complex, since it needs quite low-level HTTP control that depends on the quirks of each publisher's API. If we just do a simple curl based download it often goes wrong (i.e. HTML from a Cloudflare bot trap), despite us supplying the right API keys and having full legal permission to fetch the content. So a while back I defined per-publisher strategies for how the downloader should behave, which was subsequently improved by Robin Message and Sadiq Jaffer. I've been building a new version in Eio, porting the excellent Python work Robin did, which was itself based on OCaml code I hacked together last year when first getting started!

The new design was driven by my need to use custom downloaders in some cases, for example the built-in Windows HTTP downloader (necessary for some very obscure literature sources that require a special Windows client). Taposaur therefore uses an Eio frontend providing the high-level download logic, with pluggable backends performing the actual transfers. These backends include libcurl, which works with a vast number of real-world quirks; there's also a macOS NSURLSession backend, a browser one using fetch(), and a cohttp-eio one that isn't quite functional yet due to some missing protocol pieces.

The frontend extends the Eio capability model. Each publisher module can define a more narrow HTTP capability that carries its own API keys and rate limits. For example, Wiley wants a client token in a custom header, drawn here from a rotating pool, and requests paced a few seconds apart. Article requests then redirect to a separate content host, which the capability permits without opening up the rest of the web:

let wiley =
  http
  |> Fetch.with_limits ~clock ~min_interval:3.0
  |> Wiley.client ~tokens
in Wiley.pdf ~sw wiley doi

Springer instead binds an API key as a query parameter (so a URL a caller builds never carries it), batches up to 25 DOIs per query, and expects you to keep one request in flight and sit out a half-hour pause after every 429. The same combinators compose in a different order, with the service's own retry schedule wrapped in before the key is bound:

let springer =
  http
  |> Fetch.with_limits ~clock ~max_concurrent:1
  |> Fetch.with_retry ~clock ~random ~config:Springer.retry
  |> Springer.client ~api_key
in
Springer.articles ~sw springer dois

Note that both of these capabilities are minted from the very same Eio http client; each service module just wraps it with its own polymorphic variant tag. The tags mean the two end up with different types, so passing the Springer client to Wiley.pdf is a compile-time error rather than a confusing burst of 403s at 3am:

# let wiley = Wiley.client ~tokens http;;
val wiley : Wiley.t
# let springer = Springer.client ~api_key http;;
val springer : Springer.t
# Wiley.pdf ~sw springer doi;;
Error: The value springer has type Springer.t = Springer.tag ty t
       but an expression was expected of type Wiley.t = Wiley.tag ty t
       Type Springer.tag = [ `Generic | `Springer ]
       is not compatible with type Wiley.tag = [ `Generic | `Wiley ]
       The second variant type does not allow tag(s) `Springer

However, both can still be driven by the same Eio http calls. A service capability remains an ordinary Fetch client underneath despite the service-specific tag. Generic operations like Fetch.get are polymorphic in the tag, but the service-specific credentials and rate limits still attached to every request:

let wiley_pdf = Fetch.get ~sw wiley (Wiley.url ~doi) in
let springer_batch = Fetch.get ~sw springer (Springer.url ~dois) in
(* ... *)

This means a download strategy can be written once against the plain client interface and handed whichever publisher capability is appropriate. So far this is a pretty cool way to write HTTP interfaces in OCaml where site-specific HTTP overrides are needed. The work-in-progress code is in my taposaur repository on Tangled, and I'll publish the fetch library separately once I've iterated on the design within Taposaur some more and stabilise it all.

3 Eio pathfinding on Windows and beyond

Following on from last week's Windows promise, I've started on Eio work with a draft PR implementing Windows native paths as a Pi backend, following the suggestion from Thomas Leonard. There's enough spec divergence between POSIX and Windows logic that we dispatches via the Eio Pi layer and treat them separately. I'm still fuzzing the Windows parsing logic to find simplications (esp. Cygwin and WSL), as it's remarkably fiddly.

For example, Windows can have drive-local relative paths that have separate cwds per processs:

C:\src> D:
D:\> cd scratch
D:\scratch> C:
C:\src> type D:notes.txt
  -->  reads D:\scratch\notes.txt (D: remembers its own directory)
C:\src> type C:notes.txt
  -->  reads C:\src\notes.txt
C:\src> type C:\notes.txt
  -->  reads \notes.txt at the root of C:

However, after much spelunking it turns out that these drive-relative paths are only emulated in cmd.exe these days via magic environment variables, so I can disregard them entirely for the purposes of Eio support. Argh!

In other minor improvements, Taposaur's millions of PDFs running slowly on Ceph prompted me to submit a improvement to getdents in Eio, making traversal of directories with ~1m files about 30% faster by cutting syscalls by an order of magnitude.

I've also been extremely enjoying myself playing with the work Patrick Ferris has done with his new merry shell. As part of that, I put up a Eio mknod PR for creating device nodes directly in Eio. This removes one of the C bindings in a prototype wasm shell he's experimenting with.

4 The Dash for Life

Another place where the literature database will eventually be useful is in Shane Weisz work on Dash for Life, the new name for the Enki codename! He has been brilliantly pouring in more and more data sources while preserving the ergonomics of the UI.

As part of my infrastructure push this week, I also created a mirror of the iNaturalist and OpenStreetMap databases internally here as well, with a view to running an API server so that we don't put an undue load on public services for Shane's work.

5 Job opportunities on deep learning and SDMs

If you're looking for a really fun research opportunity and want to get into deep learning, I'm helping my buddy Andrea Manica over at Zoology with a project on deep learning for population genomics and biodiversity forecasting. See advert 1 on the genomics side:

The successful candidates will design and implement deep learning models capable of integrating heterogeneous data sources, including genomic variation, species occurrence records, climate reconstructions, environmental layers, and remotely sensed observations. The methods will be applied to three case studies focussing on African megafauna, European butterflies and moths, and UK pollinators for which we have extensive genomic resources, including time series based on museum specimens. The researchers will contribute directly to the development of a new generation of predictive biodiversity models that combine mechanistic understanding with state-of-the-art artificial intelligence.

And advert 2 on the LLM side (we hope to use the Evidence TAP infra here of course):

The principal aim of this post is the development of agentic LLM-based systems that can extract, organise, and validate biodiversity information from the published scientific literature at unprecedented scale. The successful candidate will design and implement AI workflows capable of processing more than one million scientific papers to identify and extract georeferenced information on species distributions, ecological interactions, demographic processes, environmental associations, and other biodiversity-relevant data. These data will form a key component of the CISGeM framework, complementing genomic, climatic, and environmental datasets and enabling a richer representation of biodiversity dynamics through space and time. The researcher will contribute directly to the development of a new generation of biodiversity forecasting models that combine mechanistic understanding with state-of-the-art artificial intelligence.

Andrea spent an hour with me at the Mill explaining the intricacies of ensemble-based deep learning approaches for genomic mapping, so I can't wait to dig into this more later in the year! Also very closely tied to Michael Dales work on contemporary habitat mapping and Aneesh Naik on plant SDMs, just displaced a few hundred thousand years backwards!

6 Fun links

Chris Smith, the Chancellor of Cambridge and former Master of my own Pembroke College, wrote a nice column in the Telegraph on the new Whittle Laboratory, which the King opened last month.

Simon Peyton Jones walked into the office and dropped a physical copy of the Cambridge Evening News, featuring yours truely! I'm not entirely sure where they got the interview from; it must have been from the AMD/Vultr press coverage recently. Either way, I've saved the precious paper copy for posterity!

While updating OCaml Thicket I noticed a cool PR adding a baremetal OxCaml runtime variant, and so I messed around over the weekend and got Eio booting on UEFI under KVM. Nothing production worthy in the slightest, but also didn't take much extra code.

BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)
efi_main   : heap ready, starting oxcaml runtime
[   0.00ms] === hello-uefi-ocaml: oxcaml running as a UEFI boot service ===
[   1.03ms] list       : 2000 elements, sum = 2664667000
[   1.83ms] gc         : 7134 minor words, 3 major collections
[   2.64ms] exceptions : caught Failure "nth"
[   3.25ms] floats     : sqrt 2 = 1.414214, cos pi = -1.000000, exp 1 = 2.718282
[   4.25ms] strings    : UEFI-BOOT-SERVICE
[   4.82ms] gop        : 1280x800, stride 1280 px, 4000 KiB framebuffer, 30 modes
[  12.42ms] gop        : filled 1024000 px from oxcaml in 6564 us
[  13.24ms] gop        : centre pixel reads back 0x007f7f80
[  13.97ms] effects    : yielded 1 4 9 16 25, 5 resumes
[  14.65ms] effects    : caught: Failure("from inside a fiber")
[  15.44ms] effects    : fiber recursion depth 100 ok
[  16.26ms] effects    : fiber recursion depth 10000 ok
[  18.46ms] effects    : fiber recursion depth 100000 ok
[  34.50ms] effects    : fiber recursion depth 1000000 ok
[  35.27ms] revents    : before start: ring=0 words=0 nonzero=0
[  36.21ms] revents    : Runtime_events.start () returned
[  37.05ms] revents    : after start:  ring=1 words=196626 nonzero=45
[  38.00ms] revents    : wrote 500 events, nonzero words 45 -> 1545 (+1500)
[  38.93ms] eio        : starting scheduler
[  39.48ms] eio        : fiber A waiting on promise
[  40.15ms] eio        : fiber B sleeping 1ms then resolving
[  41.90ms] eio        : fiber A got 42 from promise
[  42.85ms] eio        : interleaved a1 b1 a2 b2 a3 b3
[  48.52ms] eio        : Fiber.first winner = fast
[  49.16ms] eio        : scheduler drained cleanly
[  49.80ms] === oxcaml finished, returning to efi_main ===
efi_main   : oxcaml returned; powering off