Summary
A medRxiv preprint tested eight general-purpose large language models on 100 neurology board-style questions. Confidence estimates were often too high, while agreement across repeated answers best distinguished correct from incorrect responses.
A medRxiv preprint reports that eight general-purpose large language models answered between 45% and 92% of 100 validated neurology board-style multiple-choice questions correctly. The study also compared three ways of estimating whether an answer was right: asking the model how confident it was, using a separate model instance to critique the answer, and checking whether repeated responses were consistent.
Across the models, confidence was generally higher than performance warranted, particularly among models with lower accuracy. The authors argue that evaluations of clinical AI should report confidence calibration alongside answer accuracy. The work is a preprint, and its results concern an exam-style question set rather than clinical care.
Accuracy varied across models
The lowest reported accuracy was 45% for Llama 3.1 8B. Gemini 3 Flash and Gemini 3 Pro each reached 92%. These results describe performance on this particular set of 100 questions; they are not a measure of how well the systems perform across every neurology task.
Accuracy and calibration capture different things. Accuracy is the share of answers that are correct. Calibration concerns whether stated confidence corresponds to the likelihood of being correct—for example, whether answers assigned a given confidence level are correct about that often. A system can answer many questions correctly while still expressing unjustifiably high confidence in some answers.
Repeated answers were a stronger confidence signal
Consistency-based confidence—the agreement of responses across repeated sampling—was better at distinguishing correct from incorrect answers than either self-reported confidence or confidence inferred from a critique by another model instance. Its pooled area under the receiver operating characteristic curve (AUROC) was 0.74. AUROC measures how well a signal ranks correct answers above incorrect ones across possible thresholds; it does not, on its own, show that confidence estimates match actual probabilities of correctness.
Despite that stronger discrimination, the researchers found no statistically significant differences in calibration between the three confidence methods. The result therefore points to a useful distinction for evaluations: a confidence signal may help sort more reliable answers from less reliable ones without being better calibrated overall.
The benchmark used 100 board-style questions, so its findings speak to performance on that assessment format, not to diagnosis or treatment in real patient encounters. The authors say their calibration data and analysis scripts are available, while the question text, answer options and answer key are restricted educational materials.