.plan-26-38: Near, far, wherever you are, Tessera infers on

1 Tessera now has wall to wall embeddings over nine years!

The big news this week is that Tessera community member dClimate announced that global wall-to-wall v1.1 Tessera embeddings are now available on AWS for 2017-2025! This marks the first time that we have complete embeddings coverage for so many years for any of our models, which is a giant milestone for the project. This inference was possible thanks to a grant that dClimate received from AWS Open Data.

This also highlights just how cool working in the open is, since dClimate reimplemented the Tessera inference engine to specialise it to high-performance processing on cloud infras. We've been exchanging tips since May on our Zulip and clarifying various details in their codebase. Having an independent validation of our inference methods is as much of a big deal as the embeddings themselves, given the importance of these to such a variety of downstream tasks from which important policy decisions may be driven.

Mark Elvers and I have been working this week on integrating all this smoothly into GeoTessera so that our users can take advantage of the new embeddings. Here are some of my working notes on the topic on cross-validating the two v1.1 runs, checking their coverage, converting Icechunk to Zarr v3 and weaving the matrix into GeoTessera. I haven't had the time to cover my whole week, but here's also a dashboard of threatened species near you and some fun photos!

1.1 Cross validating the new v1.1 embeddings and sources

The first thing we did was to cross-validate the embeddings to make sure there weren't big differences in performance. One notable change resulting from the inference mechanism running on Amazon is that they switched to a slightly different satellite data source.

While both are derived from the same ESA Sentinel-2 L2A products, they aren't quite identical. On AWS, Element 84 runs "Earth Search" over its own cloud-optimised GeoTIFFs, whereas the Planetary Computer maintains its own archive and until mid-2024 ran Sen2Cor directly to produce the L2A data.

The inventories therefore differ subtly; e.g. one user found ~98k scenes on Element 84 vs ~76k on MPC for the same UK fields and dates. The data formats also need different handling due to encoding differences. For example, ESA adds a +1000 offset to every reflectance value so that -ve reflectance can be represented. However, AWS removes this offset in the Element84 dataset, but MPC serves the values raw. dClimate found that getting it wrong makes every pixel silently too bright or too dark by the same amount!

In order to test this all, we did a few lightweight tests. Our usual 'go to' has been requesting James G. C. Ball to run his Trentino tree species mapping but he was on a well deserved vacation this week. So instead Mark Elvers used the ESA WorldCover map to label a 100m grid over Cayenne, and then used 80% of the points to fit a linear classifier and predict classes. Both performed identically.

Then Sadiq Jaffer ran his tiny solar farm segmentation model and also found equivalent performance. However, one important finding from Sadiq is that we cannot use the differently inferred v1.1 embeddings in the same analysis, since both were inferred from different sources:

Sadiq's checks above show that each set of v1.1 embeddings works equally well on its own, but a model trained on one and tested on the other loses between 0.06 and 0.11 IoU, and its variance rises sharply. Although both runs used the same v1.1 model, the embeddings are computed from different copies of the input imagery, with different scene inventories and offset handling.

We will therefore keep 1.1-cam and 1.1-dclimate as separate variants in GeoTessera, and you should be careful to pick one and stick with it for any given analysis (but swapping them wholesale should make no practical difference).

1.2 Checking equivalent coverage across the two

One persistent problem with most GeoFMs is that some areas of the world have many fewer observations than others, and so the quality of inference can vary if the number of annual satellite observations is very low. Our Tessera v1 paper ran ablations to show that performance dropped sharply if n<10 for S2.

The dClimate folks decided to take the route of not generating embeddings for those areas for which we have low coverage, to keep a consistent quality bar. Robert showed that this is only a very small percentage of areas, so most users should hopefully never notice. Kristian Bodolai from Space Intelligence started a discussion about what to do with these embeddings, and apparently they are still getting excellent results doing palm oil classification even in those low coverage areas, possibly thanks to the S1 coverage still holding up the quality of the inferred embedding.

We'll make more progress on this next week, and I hope to reach consensus on what to do. Even though the 'Cambridge embeddings' do cover those areas, remember that we can't mix them safely in the same task.

1.3 Converting from Icechunk to Zarr v3

The Amazon Sustainability Data Initiative and the AWS Open Data Sponsorship Program that sponsored the inference also cover the hosting of the result, which dClimate publish as an Icechunk repository. Icechunk adds database-style transactions to Zarr, which matters at this scale as a truncated write to a normal Zarr hierarchy on object storage will be undetectable. Mark Elvers and I have been converting it over to plain Zarr v3 on Source Coop to allow non-Icechunk clients (like my OCaml code!) to access the data.

Mark has been running the conversion on AWS Fargate Spot as we did before, so it all runs as containers that have been churning through the world at around 113 GB/minute, and is about two-thirds done at the time of writing.

One possible screwup I might have made during this conversion is picking too small a chunk size for the Zarr. The Source Coop stores keep the (1,128,32,32) inner chunks inside 4096×4096 shards that I chose back in March. The dClimate store uses much larger 256×256 inner chunks inside 2048×2048 shards, and so a single pixel fetch is about 8.65 MB of traffic.

In practice, scattered point lookups are expensive in the dClimate store, but windowed reads are more expensive per pixel in my Source Coop store. There's therefore a tradeoff between streaming into a browser and streaming for the cloud, which we've been discussing on Zulip. I suspect we'll just settle on different chunk sizes for the Icechunk and Zarr v3 stores in the end to accommodate different clients. Source Coop feels more appropriate for the mobile use case due to the Cloudflare edge caching it provides. More research needed!

1.4 Weaving the matrix of embeddings into a client

I've also been teaching GeoTessera to handle this matrix of models. In the upcoming version, the v1.1 becomes the default model now that we have so much coverage. While the Zarr conversion is ongoing, streamed reads go straight to the dClimate Icechunk store.

Michael Dales also reported occasionally unusable source.coop performance, which I'm tracing to some possible instability in the Source Coop Rust proxy. More on this when I investigate next week too!

2 Dash of Life finds threatened species near you

Shane Weisz's Dash of Life from our biodiversity mapping project now has a near me page that lists the threatened species records around a point, along with the corresponding threats that their IUCN Red List assessments cite. I added a button to zoom straight into your current location (PR #555) so you can learn more about your local region with a quick browser bookmark.

Within 10 km of Cambridge, the list shows the common pochard, horse chestnut and European turtle dove as vulnerable, the European rabbit as endangered, and the European eel as critically endangered!

3 Fun photos

Cyrus Omar has made his way to Cambridge to start his sabbatical, and Carl Henrik Ek and I rolled out the red carpet at the Mill for him!

And I enjoyed seeing some gorgeous bikes and races at the Goodwood Revival, making the most of the September good weather!