Summary
A preprint reports that four large language models more often matched clinical questions to supporting evidence when given access to a larger corpus of precision evidence-based findings. The evaluation used 209 synthetic questions and measured evidence alignment and faithful reporting.
A preprint introducing Precision Evidence Bench reports that four large language models more often produced answers matched to relevant clinical evidence when given access to a larger evidence corpus. The evaluation used 209 synthetic clinical questions and assessed both whether cited evidence fit the question and whether the answer faithfully represented its sources.
The benchmark focuses on a problem in applying research to individual clinical decisions: evidence may not cover a patient's relevant characteristics, such as comorbidities, previous treatments or laboratory results. The authors tested whether models could find and report evidence aligned with those details, rather than simply produce an answer on the same general topic.
How the benchmark evaluates answers
Each synthetic question specified PICO elements: population, intervention, comparator and outcome. These describe who or what a study concerns, the treatment or exposure being considered, what it is compared with, and the outcome being assessed. Matching those elements matters because evidence about a different patient group, comparison or outcome may not answer the clinical question at hand.
The authors assessed answers with a five-item rubric. It considered whether a single cited source matched the question's PICO elements and whether the answer reported its sources faithfully. Item-level verdicts were combined into one of five ratings, from “Precision match” to “Unsupported claim”. The approach does not require a single pre-written reference answer; it evaluates the relationship between the question, cited evidence and response.
Larger evidence corpus, higher match rates
The researchers first evaluated four frontier large language models using web search across all 209 questions. They then repeated the evaluation with access to a corpus of precision evidence-based findings, or pEBFs, designed to address gaps in evidence applicability. They tested two corpus sizes.
Across the models, the share of answers rated “Precision match” was 13–18% with web search alone. With access to 100 million pEBFs, rates were 17–28%; with 500 million, they were 41–56%. The reported ranges reflect results across the four models.
The result suggests that a model's ability to answer a narrowly specified clinical question depends in part on whether it can access evidence that matches the question's details. The benchmark therefore evaluates evidence retrieval and reporting alongside the model's generated answer—a distinction that matters when an answer sounds plausible but its supporting study concerns a different population or treatment comparison.
This was a computational evaluation using synthetic questions, not a study of clinicians, patients or treatment outcomes. The paper is a medRxiv preprint, so its findings should be read as preliminary. The authors say the benchmark data are publicly available.