Shane Weisz become a certified IUCN Red List assessor last November, and then wondered whether Claude Code could pass the same exam he had just sat. It turned out that it could, with a similar mark to what Shane himself achieved! That was a big discovery at the time, but we kind of got used to frontier models being really good.
The advance of local models since then has been so rapid that they can now pass the exam as well alongside frontier models, and by running on a desktop-class machine! This is detailed in our new preprint on evaluating AI agents on the IUCN Red List assessor exam.
1 Why bother testing on an exam?
Conducting accurate species assessments is painstaking work. The IUCN Red List is the world's most widely used system for assessing the extinction risk of species, and it underpins conservation policy and action everywhere. But the assessments are severely resource constrained; <20% of plant species have been assessed at all, and over 30% of the assessments that do exist haven't been revisited in 10+ years. One bottleneck is the number of trained assessors available, which is why Shane's PhD and his Dash of Life project is looking at how to inject modern tooling to the Red List to help them out.
Modern AI can help assessors increase both the throughput and the quality of assessments, but it raises very reasonable concerns about trust and reliability, data security, cost, and the environmental impact of the datacentres doing the inference. The official assessor training exam is a convenient starting point for examinging these tradeoffs. The exam covers species categories and criteria, and mapping standards for extent and area of occupancy, and also regional assessment protocols. The questions are hard enough that Shane found them tough, and I didn't know the answers to most things!
2 What we found
We built a benchmark of exam questions and executed a spread of agents against it, each with the same official IUCN guidance documents that a human candidate has access to. We found that frontier models from Anthropic and OpenAI reliably clear the 75% pass mark, with the best of them (Claude Opus 5) reaching 96% mean accuracy, rising to 99% with a majority vote over five attempts.
The breaking result is that open-weight models running on local hardware now get a pass as well. Qwen3.8 Flash, self-hosted on a desktop Mac Studio on Shane's desk, attains 81% mean accuracy (87% with majority voting). That is a comfortable pass mark, which means assessment agents could plausibly be deployed on offline and mobile devices in the field. For many Red List partners, the location of sensitive species data (think the coordinates of the last few individuals of a poached orchid) is a blocker to using AI, so having a usable local alternative could be a game changer for conservation practitioners.
The exam is of course only a start. Passing it doesn't make a model a "species expert" any more than it makes a newly certified human one, and we're keen to encourage the responsible use of AI assistants within assessors' workflows (such as quality and consistency checks on draft assessments) rather than as a replacement for the humans.
3 Assessors in the loop
Our hope is that AI responsibly deployed, with assessors-in-the-loop and firmly in control, could be a powerful tool to meet the scale of the resourcing challenge in global biodiversity. But we need to collectively work through issues of transparency, traceability and auditability of anything an agent produces. Just as importantly, the human communities of assessors and specialist groups who do this work need to end up stronger and better connected as a result of these technologies, not more fragmented.
The paper is a collaboration between Shane Weisz, Michael Dales, Sadiq Jaffer, Steve Bachman at Kew. You can read the preprint on EcoEvoRxiv, and comments and thoughts are extremely welcome, to any of us.