# .plan-26-33: Zarro rides out and evidence papers pour in *2026-08-16 — note* Am back in Ireland again this week, after a quick stint back in Cambridge to catch the [90% eclipse of the heart](https://www.bbc.co.uk/news/articles/c75g9g50y9eo). The major preprint that went out this week was on [weather downscaling with TESSERA embeddings](https://anil.recoil.org/notes/weather-downscaling-tessera). I also did a chunk of teaching prep (October is coming up fast!) and started [porting Foundations of CS to Forester](https://anil.recoil.org/notes/forester-teaching-notes). The week itself then went on getting [TESSERA 1.0 onto Source Cooperative in Zarr](#tessera-dons-the-mask-of-zarro) with global RGB previews to go with it, putting up [a public website for Evidence TAP](#evidence-tap-goes-public), teaching [the paper downloader some manners](#tap-tap-tap-is-this-downloader-on), and [pointing a VLM at the PDFs](#working-on-metadata-quality). There's some [OCaml hacking](#ocaml-hacking) and [fun links](#fun-links) at the end. ## Tessera dons the mask of Zarro We finally have [TESSERA](https://anil.recoil.org/projects/tessera) 1.0 fully on [source.coop](https://source.coop) in Zarr format, with 1.1 well on its way\! Since we finally have enough disk space to stage everything thanks to source.coop, I also generated global RGB preview pyramids. You can now use [TZE](https://tze.geotessera.org) to zoom in and out of a dimensionality-reduced view of the 128 embedding dimensions in glorious technicolour:
We have a global RGB preview for the first time
We have a global RGB preview for the first time
 And of course you can keep zooming in the browser and inspect individual shards:
One shard near Le Mans in UTM zone 31, with its per-year embedding norms
One shard near Le Mans in UTM zone 31, with its per-year embedding norms
The RGB previews make some of the inference artefacts obvious; just look for the checkerboarding and the banding across the Amazon. This doesn't actually affect downstream inference accuracy too much, but it's very ugly. Luckily this is all [fixed in TESSERA v1.1](https://anil.recoil.org/notes/tessera-v11-out) and even more so [in v2.0](https://anil.recoil.org/papers/2026-tessera-v2), and I can't wait to see those previews to compare next week\!
On the client side, [geotessera\#357](https://github.com/ucam-eo/geotessera/pull/357) switches client downloads over to source.coop for the npy layout as well as Zarr, and picks up our new dataset variants like `v2-2B-L~beta1` so that users can choose between v1/1.1/2.0~beta easily. The Zarr portion of the PR follows the [layout](https://anil.recoil.org/notes/tessera-zarr-v3-layout) I settled on earlier in the year. I'll release all this in a new version of geotessera early next week. The giant transcode of the embeddings finished after days (and hundreds of terabytes) of back and forth between our compute cluster in Cambridge and the AWS S3 store. At the nth moment, [Mark Elvers](https://www.tunbury.org/) and I decided to rename some directories and rediscovered that [S3 doesn't do renames](https://www.tunbury.org/2026/08/17/week-33-2025/#re-versioning-sourcecoop), which led to another day of delay while six million individual server-side copies happened. Mark found that a plain `aws s3 cp` stays entirely server-side, whereas specifying a `CRC64NVME` checksum would have dragged 400TB back through Cambridge. Ho hum... ## Papers and teaching [Pedro Sousa](https://www.linkedin.com/in/pedro-marques-sousa) put his first preprint online on how "[Earth observation embeddings are effective sub-grid descriptors for probabilistic weather downscaling](https://anil.recoil.org/papers/2026-weather-downscaling)". This is joint work with [Sadiq Jaffer](https://toao.com), [Robin Young](https://www.cst.cam.ac.uk/people/ray25), [Will Tebbutt](https://willtebbutt.github.io/) and [Rich Turner](https://rich-turner-group.github.io/), and since its my first foray into weather prediction I wrote [a longer splainer](https://anil.recoil.org/notes/weather-downscaling-tessera) on how weather downscaling works and why a frozen annual [TESSERA](https://anil.recoil.org/projects/tessera) embedding beats handcrafted terrain descriptors. I also [ported 1A Foundations of CS over to Forester](https://anil.recoil.org/notes/forester-teaching-notes) and the lecture notes are now a forest of transcluded trees with stable URLs, and there's also an OCaml toplevel compiled into the browser with `js_of_ocaml` so every code transcript in the course is runnable (and editable) in place. Also a reminder that [ICFP 2026](https://icfp26.sigplan.org/) kicks off on the 24th of August in Indianapolis, where [Ryan Gibb](https://ryan.freumh.org) will be presenting his [package management calculus paper](https://anil.recoil.org/papers/2026-package-calculus). Do say hi to him if you're there\! ## Evidence TAP goes public There's a website up now at [evidencetap.org](https://evidencetap.org)! It's a very barebones site, but it does list what's going on and who is involved, and we'll populate it more over the coming months (especially by Mélanie Gréaux, who wrote up a fantastic AI4Good trip report from her recent travels). One important reason to get the site up is to have our [text and data mining policy](https://evidencetap.org/text-and-data-mining) online, which we need in order to keep publishers informed about our intended use for the large corpus of papers we're downloading. ### TAP TAP TAP, is this downloader on? So why are we are hoovering up quite so many papers in the first place? Our vision for [living evidence databases](https://anil.recoil.org/papers/2025-evidence-tap) has to screen, appraise and extract from the full text of a study rather than just its abstract. We're pinning down inclusion criteria for systematic reviews, which (e.g. in conservation) might be what the intervention was, where, on what, etc. Doing that continuously across conservation and now education requires holding the full corpus here so that local-only models can read it without any of it leaking to a third party and also be reproducible.
The ingestion is now at over 9m of the papers, with around 4m left to go from the backlog
The ingestion is now at over 9m of the papers, with around 4m left to go from the backlog
Those ten million papers we've obtained are a mixture of closed- and open-access publisher XML and PDFs that still need converting to TEI XML. This involves a pool of Pub2TEI instances (each of which tops out at around 10k documents an hour) which put the backlog at about a fortnight. I added another 128-core box and it's now at a peak of 53,000 documents an hour. The backlog's now about two days, after which we can get on with metadata analysis. The paper downloader itself is surprisingly complex because of all the per-publisher logic. Publishers throttle and block us regularly, with a variety of error codes. An HTTP 500 is now treated as a general "slow down": we wait five minutes by default and block all other requests to that origin before asking again. A 403's body is also now read and interpreted per-publisher. Some sites simply refuse our machines until we're on an IP allowlist, so all fetch traffic now leaves through a SOCKS5 proxy on one stable, allowlistable IP address. The table below shows some of the reasons we need a custom fetcher at all. Each domain has its own request rate, authentication mechanism, and IP source, all easily customisable. We'll eventually need over 200+ publishers in here\! |Strategy|Gets|Authentication|Quirks| |---|---|---|---| |Elsevier|XML (full text)|`X-ELS-APIKey`/`X-ELS-Insttoken` headers|Bad key = 401, unavailable = 403| |Springer (PDF)|PDF|Network IP|One request per 30s| |Springer (XML)|XML (JATS)|`api_key` query param|Books answer 403 (access rights). Occasionally 404s instead of 500| |Wiley|PDF (TDM API)|`Wiley-TDM-Client-Token` header|403 if not entitled to the content (books, old papers). Throttling is 500 rather than a 429| |TandF|PDF|Network IP|403 if the IP allowlist expires| |PLOS|XML|None|Also available in bulk via `allofplos`. Super easy to fetch.| |CUP|PDF|Network IP|30s pacing per request, often times out| |CORE|PDF|`Bearer` token|Search IDs ≠ output IDs, so the returned DOI needs verifying| The winner by far is PLOS, which provides a simple Python [allofplos](https://github.com/plos/allofplos) library to fetch fulltexts. ### Working on metadata quality Our Taposaur pipeline converts as much of the fulltext to TEI XML as it can, but it does miss things in complex paper layouts. On a suggestion from [Sadiq Jaffer](https://toao.com), I've been experimenting with [Nemotron Parse 2.0](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0) alongside a [Qwen 3.8 27B](https://huggingface.co/Qwen/Qwen3.8-27B-FP8) VLM to get high quality metadata and references out of even the older PDFs in our database. My fledgling 'parsosaur' tool started with pure Nemotron to parse out bounding boxes. You can [browse the output for our 2024 conservation horizon scan](https://www.cl.cam.ac.uk/~avsm2/2024-ai-conhorizon/) paper, which shows the PDF and the parsed TEI side by side. This works surprisingly well as even the magazine-style layout of [Trends in Ecology & Evolution](https://anil.recoil.org/papers/2024-ai-conhorizon) is matched reasonably accurately to the full text in the TEI, right down to the author affiliations. Mouse over either side and it'll match the scroll position of the other.
The PDF and its extracted TEI, with the abstract bounding box highlighted
The PDF and its extracted TEI, with the abstract bounding box highlighted
 Nemotron does less well at interpreting complex diagrams. I [chucked it](https://www.cl.cam.ac.uk/~avsm2/2023-ncc-permanence/) at our [Nature Climate Change paper on impermanence](https://anil.recoil.org/papers/2023-ncc-permanence), whose figures are dense and multi-panel and it degenerated into a loop of near-identical cells, which at least it marked as unreliable:
Six panels of forecast release schedules in, seven columns of repetition artefacts out
Six panels of forecast release schedules in, seven columns of repetition artefacts out
 Sadiq set me up a local Qwen 3.8 27B endpoint, and my initial experiments using the VLM to interpret diagrams when supplied with the full text of the paper are going really well. I'll try to cut a release of Parsosaur next week, as [Shane Weisz](https://shaneweisz.com) also expressed interest in learning more about VLM usage for his [Dash for Life](https://anil.recoil.org/projects/enki) dashboards. ## OCaml hacking Most of my OCaml time went into the infrastructure above, but a few things escaped into the wild. ### Making Eio traces free again While profiling Taposaur I've been adding custom events in various places. [Thomas Leonard](https://roscidus.com) and [Thomas Gazagnaire](https://github.com/samoht) discussed on [eio\#914](https://github.com/ocaml-multicore/eio/issues/914) that our OCaml runtime events allocate 23 words per event as soon as the runtime events shared memory ring is active. Since a lot of runtime events usage is about finding performance bugs, it's not ideal that the tracing calls themselves allocate memory. My fix upstream in [ocaml/ocaml\#14984](https://github.com/ocaml/ocaml/pull/14984) is hopefully a simplification. There's a per-domain write-buffer cache for custom events that was formerly a list accessed through closures, and I changed it to a single atomic bytes slot held in domain-local storage. Now, the steady state doesn't allocate at all and the buffer is only checked out for the duration of the serializer call. Thanks to [Sadiq Jaffer](https://toao.com) for for the review; I'm still not entirely sure I understand the interplay between the C and OCaml runtimes here, but the new version is at least simpler. The fix will be in OCaml 5.6 later this year. ### Escaping the docroot in cohttp More seriously, I've been triaging bug reports from Jane Street and with their help fixed a path traversal in cohttp's `resolve_local_file`, which could be escaped using urlencoded path components. [cohttp\#1145](https://github.com/mirage/ocaml-cohttp/pull/1145) urldecodes before resolving and also exposes a `Cohttp.Path.normalise` function so that anyone gating requests on a path prefix can apply the the same normalisation logic, rather than authorising a decoded path and then serving an undecoded one: ```ocaml let callback _conn req _body = let path = Cohttp.Path.normalise (Cohttp.Request.uri req) in match String.split_on_char '/' path with | "admin" :: _ when not (authorised req) -> Server.respond_not_found () | _ -> Server.respond_file ~fname:path () ``` This affects cohttp-lwt, cohttp-mirage and cohttp-async, but not cohttp-eio, which has its own path resolution and uses sandboxed fd operations anyway to prevent traversal. I also experimented with stripping control characters as well, but after discussing with the reporter decided to back that out for compatibility but left the tests in place for a future major bump. *(I would ordinarily not talk about security issues until the fix is merged, but in the new agentic world just opening the PR is enough to reveal the existence of the issue, and so I figured getting more eyes on this early is now better while waiting for reviewers. If you use cohttp, please do inspect the patch.)* ### Prometheus and the opam-repository treadmill Following on from the [prometheus-1.4 release](https://anil.recoil.org/notes/2026w32) last week, [Thomas Leonard](https://roscidus.com) merged [prometheus\#65](https://github.com/mirage/prometheus/pull/65) to move Lwt-specific logic out of the core package entirely. Defining and recording a metric no longer pulls in any concurrency library, since it just involves updating an in-memory scoreboard. [Mark Elvers](https://www.tunbury.org/) has [already](https://www.tunbury.org/2026/08/17/week-33-2025/#prometheus-14) pushed the migration through [opam-health-check](https://github.com/ocurrent/opam-health-check/pull/112) and [ocluster](https://github.com/ocurrent/ocluster/pull/265), which is a good sign that the deprecation path works as advertised. Next up is the [Eio backend](https://github.com/mirage/prometheus/pull/71) as an independent package and the next major version should be good to go. As many of the other maintainers are holiday, I also did a bunch of [opam-repository](https://github.com/ocaml/opam-repository) merging, I also fixed [oxcaml/opam-repository\#59](https://github.com/oxcaml/opam-repository/pull/59) to let upstream dune 3.24.2, eio 1.4 and mdx 2.6.0 through the [guards](https://anil.recoil.org/notes/oxcaml-opam-guards) so all three now build with OxCaml. My [website monorepo](https://anil.recoil.org/notes/bushel-lives) continues to build on `ox-minus39` so I can update this website, yay\! ## Fun links - There's a brilliant mobile game on being a prime minister simulator, [No 10: Full Confidence](https://www.bbc.co.uk/news/articles/cq56pzqy6jvo), which briefly ranked above Minecraft on the UK App Store chart (!). I managed two and a half years before being ousted by my own cabinet. - [Frank Feng](https://www.cst.cam.ac.uk/people/zf281) sent on a report from China that they've been using in-house [Moffett S40 Computing Card](https://www.eet-china.com/vote/p/2/27/37?lang=en) to accelerate Tessera inference significantly. I'd heard about these when I went with the Royal Society delegation to China in 2023, but it's the first time I'm seeing sparse-weight AI accelerators 'in the wild'. - Fascinating article on "[Why Are Rivers So Mathematical?](https://www.quantamagazine.org/why-are-rivers-so-mathematical-20260810/)" in Quanta. I've been thinking about how to model roads and rivers generatively using Tessera to help with flood predictions, so this article is well timed\! - It's been an insanely fast week of progress on local LLMs, with [Qwen 3.8](https://artificialanalysis.ai/models/qwen3-8-27b) performing incredibly, and I've been trialling [Deepseek v4 Pro 813](https://github.com/antirez/ds4/issues/807) as well. More on that next week\! Synopsis: TESSERA 1.0 is now fully available as Zarr with global RGB previews, a weather downscaling preprint, and Evidence TAP gets a public website with progress on the downloader and parser. Words: 2179 Discussion: - Bluesky: - LinkedIn: - Mastodon: - Twitter: ## Related - [Using Forester to turn Foundations of CS into interactive evergreen lectures](https://anil.recoil.org/notes/forester-teaching-notes) (note, 2026-08-16) - [Improving local weather forecasts using Tessera embeddings](https://anil.recoil.org/notes/weather-downscaling-tessera) (note, 2026-08-13) - [.plan-26-32: Finally a use for serverless and found the Forester for the trees](https://anil.recoil.org/notes/2026w32) (note, 2026-08-09) - [Updating to the OxCaml 5.2.0-minus39 opam packaging](https://anil.recoil.org/notes/oxcaml-opam-guards) (note, 2026-08-08) - [Earth observation embeddings are effective sub-grid descriptors for probabilistic weather downscaling](https://anil.recoil.org/papers/2026-weather-downscaling) (paper, 2026-08-01) - [Package Managers à la Carte: A Formal Model of Dependency Resolution](https://anil.recoil.org/papers/2026-package-calculus) (paper, 2026-07-01) - [TESSERA v2: Scaling Pixel-wise Earth Foundation Models](https://anil.recoil.org/papers/2026-tessera-v2) (paper, 2026-07-01) - [Tessera 1.0 to 1.1 embeddings](https://anil.recoil.org/videos/297de7c9-9cea-4051-8b27-041fffa90e72) (video, 2026-06-12) - [Tessera v1.1 released, with smoother and temporally stable embeddings](https://anil.recoil.org/notes/tessera-v11-out) (note, 2026-06-12) - [Streaming millions of TESSERA tiles over HTTP with Zarr v3](https://anil.recoil.org/notes/tessera-zarr-v3-layout) (note, 2026-03-14) - [Enki, a Dashboard of Life on Earth](https://anil.recoil.org/projects/enki) (project, 2026-01-01) - [AI-assisted Living Evidence Databases for Conservation Science](https://anil.recoil.org/papers/2025-evidence-tap) (paper, 2025-10-01) - [Arise Bushel, my sixth generation oxidised website](https://anil.recoil.org/notes/bushel-lives) (note, 2025-01-29) - [TESSERA, a pixelwise geospatial foundation model](https://anil.recoil.org/projects/tessera) (project, 2025-01-01) - [The potential for AI to revolutionize conservation: a horizon scan](https://anil.recoil.org/papers/2024-ai-conhorizon) (paper, 2024-12-01) - [Realizing the social value of impermanent carbon credits](https://anil.recoil.org/papers/2023-ncc-permanence) (paper, 2023-11-01) --- Canonical: https://anil.recoil.org/notes/2026w33 Type: note Tags: tessera, ocaml, conservation, ai, academia