# .plan-26-31: Sorting out Tessera and Evidence TAP infrastructure
*2026-08-02 — note*
I had a bunch of infrastructure work I had to get done this week, and so I went headsdown hacking to sort out both [TESSERA](https://anil.recoil.org/projects/tessera) and the [Evidence TAP](https://anil.recoil.org/projects/ce) this week\!
Yours truely in the Cambridge Evening News
I spent it [moving a petabyte of TESSERA embeddings](#moving-a-petabyte-of-tessera-embeddings) onto Source Cooperative, taking [a detour to investigate Icechunk](#a-detour-to-investigate-icechunk) before settling on [transcoding to Zarr on AWS Fargate Spot](#transcoding-to-zarr-on-aws-fargate-spot). Then my custom PDF downloader [Taposaur arose](#arise-taposaur-and-obtain-the-literature-for-evidence-tap) to obtain the literature for Evidence TAP, by [extracting structured metadata with GROBID](#getting-pdf-structured-metadata-with-grobid) and downloading papers with [a capability-based OCaml downloader](#a-capability-based-ocaml-paper-downloader). I continued [Eio work Windows](#eio-pathfinding-on-windows-and-beyond) now that 1.4 is out, mirrored some databases for [the Dash for Life](#the-dash-for-life), and wanted to advertise some [super fun jobs on deep learning SDMs](#job-opportunities-on-deep-learning-and-sdms). The [fun links](#fun-links) at the end feature an unexpected newspaper appearance by yours truely and my first new unikernel boot trace in some years\!
## Moving a petabyte of TESSERA embeddings
[Mark Elvers](https://www.tunbury.org/) and I have been getting the full range of [TESSERA](https://anil.recoil.org/projects/tessera) embeddings up on [Source Cooperative](https://source.coop) for the [past few weeks](https://anil.recoil.org/notes/2026w29). We split this latest work into two steps: first move the existing npy-based embeddings as-is, and then transcode to Zarr v3. This week the npy ones fully synched (almost a petabyte!) covering TESSERA v1 alongside selectively generated v1.1 and v2-beta ones.
The first half of my [geotessera\#337](https://github.com/ucam-eo/geotessera/pull/337) PR switches us over to the `data.source.coop` endpoint for HTTP downloads, which lets us drop our S3-specific logic we had in geotessera 0.9. That code was weirdly problematic and complex; EC2 seems to drop connections quite quickly on bulk downloads, which is what most of our users do. My tests on the Source Coop proxy show Cloudflare R2 doing a great job of edge caching, so this looks like a solid long-term solution that's still using S3 under the hood, but not exposing that to our library users.
I'm also in the Source Coop Slack now, with extremely responsive developers on the other end. I'll cut a geotessera release with the npy support this week, with the Zarr conversion to follow shortly I hope\!
### A detour to investigate Icechunk
The Zarr conversion requires transcoding the npy sources over to a fresh Zarr store, so I spent some time exploring how best to do this.
The only alternative that's credible is [Icechunk](https://icechunk.io), Earthmover's transactional storage engine for Zarr that adds git-like version control over object storage. That's obviously very attractive for TESSERA, since we dynamically generate embeddings for a [growing set](https://anil.recoil.org/papers/2026-tessera-v2) of model variants and could really use version control. I knocked up a quick version of geotessera using the Icechunk Python library instead of Zarr's, which was a trivial drop-in since [Icechunk speaks the Zarr Python interface](https://icechunk.io/en/latest/getting-started/quickstart/).
Behind the scenes though, Icechunk writes to S3 in a [custom format](https://icechunk.io/en/stable/reference/spec-v2-1/). That format seems rock solid in my testing, but is a blocker for us since we need client access from languages other than Python. There's an [Icechunk Rust crate](https://icechunk.io/en/stable/reference/icechunk-rust/) that isn't intended for external binding, a [JavaScript version](https://icechunk.io/en/stable/reference/icechunk-js/) for browser use, but nothing I could bind (e.g.) OCaml against myself. There's a [vibe-coded PR for a C API and Julia bindings](https://github.com/earth-mover/icechunk/pull/1722), but it's not high quality enough so was rejected (rightly I think) by the Earthmover team. So I'll reluctantly come back to Icechunk when it grows a more stable FFI story, but won't use it just yet over a 'raw' Zarr v3 publish.
### Transcoding to Zarr on AWS Fargate Spot
I went back to my [Zarr v3 geoembeddings layout](https://anil.recoil.org/notes/tessera-zarr-v3-layout), with a static store on source.coop spanning 2017-2025 (extensible to future years, though not easily to earlier ones without using the aforementioned Icechunk). Finding the CPU to transcode ~10 million tiles without burning a fortune in egress bandwidth was surprisingly tricky! Running it in Cambridge via our current server (which we're trying to deprecate in favour of cloud hosting) would take around six months (!).
After a number of failed experiments, I came across [Amazon Fargate Spot](https://aws.amazon.com/fargate/pricing/), an order of magnitude cheaper way to run a Docker container in the same region as the embeddings (us-west-2) and serverless so we didn't have to manage all the VMs manually.
The rest of my [geotessera\#337](https://github.com/ucam-eo/geotessera/pull/337) PR makes the `zarr-fill` cli stateless/incremental. It scans each zone first via Zarr to determine what's needed for a particular UTM zone, and also stores sampled metadata so the RGB pyramids are fast to calculate at the end. Without the samples we'd have to read the full petabyte back over the network just to run a PCA for the preview you see in [tze.geotessera.org](https://tze.geotessera.org)\!
Mark did a quick trial run, and our spot instance goes about 100x faster than from Cambridge, so we're going to run every UTM zone in parallel for a rough cost of 1500 quid for the whole world (I'll find out in a few days if our back-of-the-envelope math is right or not). This compares well to the 50 grand bill from my original naive plan\!
Once TESSERA v1 is converted, v1.1 will be quick as it has fewer tiles as we've not done a full global run yet. In the future I'm aiming for all our embeddings generation to be Zarr-first, so getting this giant transcode out of the way will be a big boost.
## Arise Taposaur and obtain the literature for Evidence TAP\!
I've also been making inroads on the processing pipeline for [Evidence TAP](https://anil.recoil.org/notes/2026w30), now that we have more users. This is codenamed 'Taposaur', and has two major pieces: a custom downloader to deal with the (often ridiculous) hoops publishers make us jump through, and a metadata index over the millions of resulting PDFs that we've snarfed. My goal is to get this piece integrated with [Sam Reynolds](https://samreynolds.org) work on [the Conservation Copilot](https://www.samreynolds.org/#copilot) this summer so it can have secure (and local model only) access to our fulltext paper database.
### Getting PDF structured metadata with GROBID
Since we already have millions of downloaded papers from the last year's work, the first thing I cleaned up was the metadata processing. I'm building on three brilliant upstream projects [biblio-glutton](https://github.com/kermitt2/biblio-glutton), [GROBID](https://github.com/kermitt2/grobid) and [Pub2TEI](https://github.com/kermitt2/Pub2TEI), which all fit together to mirror public indices like [Crossref](https://crossref.org) and provide HTTP APIs that scan a PDF into structured metadata:
|service|port|role|
|---|---|---|
|glutton|8080|bibliographic lookup over 175M+ Crossref records|
|grobid|8070|convert PDF to TEI via deep-learning models|
|pub2tei|8060|convert publisher XML to TEI|
|elasticsearch|9200|glutton's search index|
The 'TEI' here refers to an [XML format for scholarly publishing](https://tei-c.org/release/doc/tei-p5-doc/en/html/SG.html) that acts as an interchange format across all the publishers' slightly different schemas. I'll write more on this later, but here's a preview of the web interface:
The Taposaur index with a subset of our PDFs, and the corpus-wide sharded TEI conversion
The overall database runs from an OCaml scheduler that dispatches jobs over to a Docker Swarm running Grobid and the other services. It'll take about a month to churn through the full set, and outputs the TEI form into a nice local filesystem that I can then stash on our Ceph storage cluster.
The author metadata is parsed out of the PDF into structured form. Here's a Bill Sutherland paper for example
The best thing about TEI is that is has a structured representation of much paper metadata. For example, I grabbed a random paper by [Bill Sutherland](https://www.zoo.cam.ac.uk/directory/bill-sutherland) about [replacing bar charts with histograms](https://doi.org/10.1002/ece3.11237) and it has ORCIDs and even the corresponding author for the paper.
Journal, licence, keyword and funder records for the same paper, matched against Crossref via biblio-glutton
Then there's also the usual Crossref data, and because we have a local database, I can introduce overrides or evidence synthesis specific metadata quite easily for our own use. I'm very grateful to the Grobid team for open-sourcing their infrastructure; they are also working on some really cool [open mining](https://arxiv.org/abs/2210.15600) and [science literature datasets](https://arxiv.org/abs/2512.11192) that I'm excited to read more about later in the year.
### A capability-based OCaml paper downloader
The PDF paper downloader itself is surprisingly complex, since it needs quite low-level HTTP control that depends on the quirks of each publisher's API. If we just do a simple `curl` based download it often goes wrong (i.e. HTML from a Cloudflare bot trap), despite us supplying the right API keys and having full legal permission to fetch the content.
So a while back I defined per-publisher strategies for how the downloader should behave, which was subsequently improved by [Robin Message](https://www.lambdacambridge.com/robin-message) and [Sadiq Jaffer](https://toao.com). I've been building a new version in [Eio](https://github.com/ocaml-multicore/eio), porting the excellent Python work Robin did, which was itself based on OCaml code I hacked together last year when first getting started\!
The new design was driven by my need to use custom downloaders in some cases, for example the built-in Windows HTTP downloader (necessary for some very obscure literature sources that require a special Windows client). Taposaur therefore uses an Eio frontend providing the high-level download logic, with pluggable backends performing the actual transfers. These backends include [libcurl](https://curl.se/libcurl/), which works with a vast number of real-world quirks; there's also a [macOS NSURLSession](https://developer.apple.com/documentation/foundation/urlsession) backend, a browser one using [fetch()](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API), and a cohttp-eio one that isn't quite functional yet due to some missing protocol pieces.
The frontend extends the Eio [capability model](https://roscidus.com/blog/blog/2023/04/26/lambda-capabilities/). Each publisher module can define a more narrow HTTP capability that carries its own API keys and rate limits. For example, Wiley wants a client token in a custom header, drawn here from a rotating pool, and requests paced a few seconds apart. Article requests then redirect to a separate content host, which the capability permits without opening up the rest of the web:
```ocaml
let wiley =
http
|> Fetch.with_limits ~clock ~min_interval:3.0
|> Wiley.client ~tokens
in Wiley.pdf ~sw wiley doi
```
Springer instead binds an API key as a query parameter (so a URL a caller builds never carries it), batches up to 25 DOIs per query, and expects you to keep one request in flight and sit out a half-hour pause after every 429. The same combinators compose in a different order, with the service's own retry schedule wrapped in before the key is bound:
```ocaml
let springer =
http
|> Fetch.with_limits ~clock ~max_concurrent:1
|> Fetch.with_retry ~clock ~random ~config:Springer.retry
|> Springer.client ~api_key
in
Springer.articles ~sw springer dois
```
Note that both of these capabilities are minted from the very same Eio `http` client; each service module just wraps it with its own polymorphic variant tag. The tags mean the two end up with different types, so passing the Springer client to `Wiley.pdf` is a compile-time error rather than a confusing burst of 403s at 3am:
```ocaml
# let wiley = Wiley.client ~tokens http;;
val wiley : Wiley.t
# let springer = Springer.client ~api_key http;;
val springer : Springer.t
# Wiley.pdf ~sw springer doi;;
Error: The value springer has type Springer.t = Springer.tag ty t
but an expression was expected of type Wiley.t = Wiley.tag ty t
Type Springer.tag = [ `Generic | `Springer ]
is not compatible with type Wiley.tag = [ `Generic | `Wiley ]
The second variant type does not allow tag(s) `Springer
```
However, both can still be driven by the same Eio http calls. A service capability remains an ordinary `Fetch` client underneath despite the service-specific tag. Generic operations like `Fetch.get` are polymorphic in the tag, but the service-specific credentials and rate limits still attached to every request:
```ocaml
let wiley_pdf = Fetch.get ~sw wiley (Wiley.url ~doi) in
let springer_batch = Fetch.get ~sw springer (Springer.url ~dois) in
(* ... *)
```
This means a download strategy can be written once against the plain client interface and handed whichever publisher capability is appropriate. So far this is a pretty cool way to write HTTP interfaces in OCaml where site-specific HTTP overrides are needed. The work-in-progress code is in my [taposaur repository](https://tangled.org/anil.recoil.org/taposaur) on Tangled, and I'll publish the fetch library separately once I've iterated on the design within Taposaur some more and stabilise it all.
## Eio pathfinding on Windows and beyond
Following on from [last week's Windows promise](https://anil.recoil.org/notes/2026w30), I've started on Eio work with [a draft PR implementing Windows native paths](https://github.com/ocaml-multicore/eio/pull/908) as a Pi backend, following the suggestion from [Thomas Leonard](https://github.com/https://roscidus.com). There's enough spec divergence between POSIX and Windows logic that we dispatches via the Eio Pi layer and treat them separately. I'm still fuzzing the Windows parsing logic to find simplications (esp. Cygwin and WSL), as it's remarkably fiddly.
For example, Windows can have drive-local relative paths that have separate cwds per processs:
```
C:\src> D:
D:\> cd scratch
D:\scratch> C:
C:\src> type D:notes.txt
--> reads D:\scratch\notes.txt (D: remembers its own directory)
C:\src> type C:notes.txt
--> reads C:\src\notes.txt
C:\src> type C:\notes.txt
--> reads \notes.txt at the root of C:
```
However, after much spelunking it turns out that these drive-relative paths are
only _emulated_ in `cmd.exe` these days via magic environment variables, so I
can disregard them entirely for the purposes of Eio support. Argh\!
In other minor improvements, Taposaur's millions of PDFs running slowly on Ceph
prompted me to submit [a improvement to getdents in
Eio](https://github.com/ocaml-multicore/eio/pull/910), making traversal of
directories with ~1m files about 30% faster by cutting syscalls by an order of
magnitude.
I've also been extremely enjoying myself playing with the work [Patrick Ferris](https://patrick.sirref.org) has done with his new [merry shell](https://git.sirref.org/merry). As part of that, I put up [a Eio `mknod` PR](https://github.com/ocaml-multicore/eio/pull/911) for creating device nodes directly in Eio. This removes one of the C bindings in a prototype wasm shell he's experimenting with.
## The Dash for Life
Another place where the literature database will eventually be useful is in [Shane Weisz](https://shaneweisz.com) work on [Dash for Life](https://dashforlife.org), the new name for the [Enki](https://anil.recoil.org/projects/enki) codename! He has been brilliantly pouring in more and more data sources while preserving the ergonomics of the UI.
As part of my infrastructure push this week, I also created a mirror of the iNaturalist and OpenStreetMap databases internally here as well, with a view to running an API server so that we don't put an undue load on public services for Shane's work.
Obligatory pika dashboard
## Job opportunities on deep learning and SDMs
If you're looking for a _really_ fun research opportunity and want to get into
deep learning, I'm helping my buddy [Andrea Manica](https://www.zoo.cam.ac.uk/people/andrea-manica) over at Zoology with a
project on deep learning for population genomics and biodiversity forecasting.
See [advert 1](https://www.jobs.cam.ac.uk/job/56448/) on the genomics side:
> The successful candidates will design and implement deep learning models
> capable of integrating heterogeneous data sources, including genomic
> variation, species occurrence records, climate reconstructions, environmental
> layers, and remotely sensed observations. The methods will be applied to
> three case studies focussing on African megafauna, European butterflies and
> moths, and UK pollinators for which we have extensive genomic resources,
> including time series based on museum specimens. The researchers will
> contribute directly to the development of a new generation of predictive
> biodiversity models that combine mechanistic understanding with
> state-of-the-art artificial intelligence.
And [advert 2](https://www.jobs.cam.ac.uk/job/56450/) on the LLM side (we hope to use the Evidence TAP infra here of course):
> The principal aim of this post is the development of agentic LLM-based
> systems that can extract, organise, and validate biodiversity information
> from the published scientific literature at unprecedented scale. The
> successful candidate will design and implement AI workflows capable of
> processing more than one million scientific papers to identify and extract
> georeferenced information on species distributions, ecological interactions,
> demographic processes, environmental associations, and other
> biodiversity-relevant data. These data will form a key component of the
> CISGeM framework, complementing genomic, climatic, and environmental datasets
> and enabling a richer representation of biodiversity dynamics through space
> and time. The researcher will contribute directly to the development of a new
> generation of biodiversity forecasting models that combine mechanistic
> understanding with state-of-the-art artificial intelligence.
Andrea spent an hour with me at the Mill explaining the intricacies of ensemble-based
deep learning approaches for genomic mapping, so I can't wait to dig into this more
later in the year! Also very closely tied to [Michael Dales](https://mynameismwd.org) work on [contemporary habitat mapping](https://digitalflapjack.com/weeknotes/2026-08-03/) and [Aneesh Naik](https://aneeshnaik.github.io/) on [plant SDMs](https://www.aneeshnaik.com/blogposts/20260720_weeknotes_2026_29.html), just displaced a few hundred thousand years backwards\!
## Fun links
[Chris Smith](https://en.wikipedia.org/wiki/Chris_Smith,_Baron_Smith_of_Finsbury), the Chancellor of Cambridge and former Master of my own Pembroke College, wrote a nice column [in the Telegraph on the new Whittle Laboratory](https://www.telegraph.co.uk/news/2026/07/24/frank-whittle-engineering-lab-jet-engine-tech-economy/), which [the King opened last month](https://www.cam.ac.uk/news/the-king-officially-opens-cambridges-new-whittle-laboratory).
[Simon Peyton Jones](https://simon.peytonjones.org/) walked into the office and dropped a physical copy of the Cambridge Evening News, featuring yours truely! I'm not entirely sure where they got the interview from; it must have been from the AMD/Vultr press coverage recently. Either way, I've saved the precious paper copy for posterity\!
Note that I neither died nor allowed instant forest loss, just unfortunate cropping
While updating [OCaml Thicket](https://thicket.dev) I noticed a cool PR adding a [baremetal OxCaml](https://github.com/oxcaml/oxcaml/pull/6549) runtime variant, and so I messed around over the weekend and got Eio booting on UEFI under KVM. Nothing production worthy in the slightest, but also didn't take [much extra code](https://github.com/avsm/flambda-backend/commit/b997c2d9ea795d398fcb16e6141865f1be74056c).
```
BdsDxe: loading Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)
BdsDxe: starting Boot0002 "UEFI QEMU HARDDISK QM00001 " from PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0xFFFF,0x0)
efi_main : heap ready, starting oxcaml runtime
[ 0.00ms] === hello-uefi-ocaml: oxcaml running as a UEFI boot service ===
[ 1.03ms] list : 2000 elements, sum = 2664667000
[ 1.83ms] gc : 7134 minor words, 3 major collections
[ 2.64ms] exceptions : caught Failure "nth"
[ 3.25ms] floats : sqrt 2 = 1.414214, cos pi = -1.000000, exp 1 = 2.718282
[ 4.25ms] strings : UEFI-BOOT-SERVICE
[ 4.82ms] gop : 1280x800, stride 1280 px, 4000 KiB framebuffer, 30 modes
[ 12.42ms] gop : filled 1024000 px from oxcaml in 6564 us
[ 13.24ms] gop : centre pixel reads back 0x007f7f80
[ 13.97ms] effects : yielded 1 4 9 16 25, 5 resumes
[ 14.65ms] effects : caught: Failure("from inside a fiber")
[ 15.44ms] effects : fiber recursion depth 100 ok
[ 16.26ms] effects : fiber recursion depth 10000 ok
[ 18.46ms] effects : fiber recursion depth 100000 ok
[ 34.50ms] effects : fiber recursion depth 1000000 ok
[ 35.27ms] revents : before start: ring=0 words=0 nonzero=0
[ 36.21ms] revents : Runtime_events.start () returned
[ 37.05ms] revents : after start: ring=1 words=196626 nonzero=45
[ 38.00ms] revents : wrote 500 events, nonzero words 45 -> 1545 (+1500)
[ 38.93ms] eio : starting scheduler
[ 39.48ms] eio : fiber A waiting on promise
[ 40.15ms] eio : fiber B sleeping 1ms then resolving
[ 41.90ms] eio : fiber A got 42 from promise
[ 42.85ms] eio : interleaved a1 b1 a2 b2 a3 b3
[ 48.52ms] eio : Fiber.first winner = fast
[ 49.16ms] eio : scheduler drained cleanly
[ 49.80ms] === oxcaml finished, returning to efi_main ===
efi_main : oxcaml returned; powering off
```
Synopsis: A petabyte of TESSERA embeddings moves to Source Cooperative, and Taposaur's GROBID metadata index and capability-based downloader take shape for Evidence TAP, while Eio gets some native Windows support.
Words: 3151
DOI: 10.59350/30e5y-n8p97
Discussion:
- LinkedIn:
- Mastodon:
- Twitter:
## Related
- [.plan-26-30: Bananas ripen, models open, and OxCaml releases](https://anil.recoil.org/notes/2026w30) (note, 2026-07-26)
- [.plan-26-29: Perfect weather, imperfectly measured, precisely predicted](https://anil.recoil.org/notes/2026w29) (note, 2026-07-19)
- [TESSERA v2: Scaling Pixel-wise Earth Foundation Models](https://anil.recoil.org/papers/2026-tessera-v2) (paper, 2026-07-01)
- [Streaming millions of TESSERA tiles over HTTP with Zarr v3](https://anil.recoil.org/notes/tessera-zarr-v3-layout) (note, 2026-03-14)
- [Enki, a Dashboard of Life on Earth](https://anil.recoil.org/projects/enki) (project, 2026-01-01)
- [TESSERA, a pixelwise geospatial foundation model](https://anil.recoil.org/projects/tessera) (project, 2025-01-01)
- [Conservation Evidence Copilots](https://anil.recoil.org/projects/ce) (project, 2024-01-01)
---
Canonical: https://anil.recoil.org/notes/2026w31
Type: note
Tags: tessera, ocaml, eio, ai, conservation, academia
