This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is available for being worked on. It will be supervised by Anil Madhavapeddy and Sadiq Jaffer as part of my Conservation Evidence Copilots project.

Summary

At the Conservation Evidence Copilots project, we are interested in finding and synthesising evidence for conservation interventions. Once the text has been retrieved, it needs to be summarised in a way that is accurate, concise and relevant. This is particularly important for conservation evidence, where the key findings need to be communicated clearly to inform policy and practice.

This project investigates how to verify the accuracy of summaries generated by LLMs and RAG pipelines for CE literature. Our goal is to develop a pipeline that can reliably go from extracting relevant information from text to a summary that is verifiably (by a human) correct.

Related Reading

• The Ragas framework for RAG evaluation
• CheckEmbed: Effective Verification of LLM Solutions to Open Ended Tasks, arxiv:2406.02524v2, June 2024
• Calibrating Sequence Likelihood Improves Conditional Language Generation, arxiv:2210.00045, September 2000

Related Ideas

• Assessing mangrove literature for conservation evidence
  cosupervised with Sadiq Jaffer and Tom Worthington (Available, Part II)
• Evaluating RAG pipelines for conservation evidence
  by Radhika Iyer cosupervised with Sadiq Jaffer in 2024 (Ongoing)
• Crawling grey literature for conservation evidence
  by Shrey Biswas and Kacper Michalik cosupervised with Sadiq Jaffer in 2024 (Ongoing)