Updated preprint on LLMs for evidence-based decision support
We have just updated our preprint on using LLMs for evidence decision support with more evaluation results and corrections from peer review.
Our findings suggest that, with careful domain-specific design, LLMs could potentially be powerful tools for enabling expert-level use of evidence syntheses and databases. However, general LLMs used "out-of-the-box" are likely to perform poorly and misinform decision-makers. By establishing that LLMs exhibit comparable performance with human synthesis experts on providing restricted responses to queries of evidence syntheses and databases, future work can build on our approach to quantify LLM performance in providing open-ended responses.
See also the fantastic EEG seminar talk that the student group who worked on this over the summer gave towards the end of last year.
Radhika Iyer, Alec Christie, Anil Madhavapeddy, Sam Reynolds, Bill Sutherland and Sadiq Jaffer.
Working paper at Research Square.
papers