Imagine you're out on a field trip in the Peruvian Amazon and come across a mesmerising butterfly species you don't recognise. You could ask iNaturalist's AI, but its identification models have biases and many missing species. The latest computer vision model recognises just over 120,000 taxa, out of roughly 2 million described species on Earth. So how else would you go about working out what species it is?
One approach is to take an image of the species and then find a local field guide on Peruvian butterflies. You would flip through the pages, compare your image to the pictures and text descriptions for each species in the book, and reason over which seems like the best match.
This project would do something similar for few-shot species image classification. The research question is whether multi-modal reasoning models doing agentic search over local field guides can improve classification performance over existing approaches. The existing paradigm is to collect a huge labelled dataset and train a big supervised neural net. That approach suffers from class imbalance and long-tail problems, and so performs poorly on less common species. In theory, the field guide approach could mitigate some of those weaknesses. Rare species are also precisely where the biodiversity data gaps are worst, so this is a good application of tool-using agentic LLMs.
For evaluation, we could start with one of the existing benchmark datasets for species image classification. The TreeOfLife-200M dataset is a great starting point, with 200M images spanning nearly 1M taxa. Another good candidate is the recent Open-Insect benchmark for open-set recognition of novel species, from Sara Beery and colleagues. We could then compare the accuracy of the field guide, agentic search and reasoning approach against the supervised approaches, while using only a fraction of the training data.
As far as we can tell, this approach has not been tried yet, which makes it a timely experiment. Ultimately, better identification of rare species could help address important biodiversity data gaps. It would also feed into the observation pipelines behind our Enki dashboard project and Shane Weisz's work on a living IUCN Red List.