Evaluating RAG pipelines for conservation evidence

This is an idea proposed in 2024 as a good starter project, and has been completed by Radhika Iyer. It was co-supervised with Sadiq Jaffer.

At the Conservation Evidence Copilots project, we are interested in finding and synthesising evidence for conservation interventions. Once we have this evidence, it needs to be synthesised into a form that can be used to inform policy and practice via natural language query interfaces. One way to do this is to use a RAG (Retrieval Augmented Generation) pipeline, which can automatically retrieve relevant information from a large corpus of documents, analyse it to extract key information relevant to CE, and generate a summary of the key findings.

This project involved involve evaluating the performance of RAG pipelines for conservation evidence, comparing different models, configurations and benchmark sets, and identifying areas for improvement. The goal is to develop a set of best practices for using RAG pipelines in the context of conservation evidence, and to make recommendations for future work in this area.

A first preprint on this work titled "Careful design of Large Language Model pipelines enables expert-level retrieval of evidence-based information from syntheses and databases" is now available.

Related Reading

• Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation, arXiv:2405.13622v1, May 2024