Summary
A medRxiv preprint found that language models often produced numbers consistent with clinical evidence but struggled to adapt those numbers to patient-specific cases and actions. Explicit inputs, calculations and complete decision rules substantially improved performance in controlled tests.
A medRxiv preprint reports that language models can often reproduce numbers consistent with clinical evidence but have more difficulty applying those numbers to a particular patient case. In the study, explicit patient inputs, calculation requests and complete decision rules substantially improved performance in controlled experiments.
The researchers evaluated the benchmark, called NUMBERS, using 16 model configurations and 1,300 questions linked to public clinical evidence. The experiments examined three related abilities: updating prevalence-dependent quantities, producing patient-specific estimates and selecting a clinical action from a stated rule.
How the benchmark exposed the gap
Prevalence is the proportion of a population with a condition. Clinical estimates that depend on prevalence can change when the underlying frequency of that condition changes, so applying evidence to a case requires more than recalling a fixed number.
The models updated prevalence-dependent quantities correctly in 78.5% of comparisons when they were given the relevant inputs and asked to perform a calculation. When the same type of task was presented using clinical wording, accuracy fell to 33.7%.
A related experiment compared patient-level answers with a near-target standard across 2,340 pairs. Supplying the inputs and explicitly requesting a calculation increased the share of near-target answers from 29.7% to 78.8%.
The study also examined diagnostic estimates against the source evidence. Among estimates that fell outside the source-result tolerance, 77.1% remained within the evidence's central 80% predictive range. Together, these results indicate that numerical output can appear broadly consistent with evidence while still being difficult for a model to adapt reliably to the details of a case.
Complete rules improved clinical action selection
The direct-action experiment tested whether models could move from recalling a cutoff to selecting the action associated with it. It covered 50 rules, five models and 7,500 calls.
In the recall condition, models sometimes stated a cutoff that implied the correct action but then selected the wrong action. This occurred in 413 of 3,750 calls, or 11.01%, with a 95% confidence interval of 8.93 to 13.17.
Providing the complete rule produced a large improvement. Action accuracy rose from 85.63% when the model had to recall the relevant rule to 99.41% when the full rule was supplied. The reported improvement was 13.79 percentage points, with a 95% confidence interval of 11.65 to 15.95.
The pattern matters because clinical reasoning frequently requires several linked steps: identifying the relevant evidence, adjusting an estimate for the case and applying a threshold or rule to choose an action. The benchmark suggests that a model's ability to state a clinical fact does not automatically ensure consistent use of that fact in a decision.
Evidence level and study context
This was a preprint study using public, aggregate, non-identifiable human data and published results that were available before the research began. The experiments used controlled prompts rather than prospective patient care. The authors conclude that explicit inputs, calculations and complete rules substantially improved performance under those test conditions.
The findings therefore describe a specific property of the tested model configurations and prompt designs. How the improvements would transfer to real clinical workflows, where information may be incomplete and decisions involve broader context, was not tested in the reported experiments.