Summary
A blinded preprint compared five large language models on 17 synthetic emergency-department cases. All named the reference diagnosis, but triage assignments and clinician ratings of response quality differed.
A blinded comparison of five large language models found that each named the reference diagnosis in all 17 synthetic emergency-department vignettes, but their triage assignments and response-quality ratings varied. The medRxiv preprint reports that two emergency physicians assessed the models’ 85 responses with model identities concealed. The authors caution that results from these selected cases do not establish any model’s clinical safety.
How the models were compared
The researchers submitted the same 17 vignettes to Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3 through consumer interfaces on 11–12 August 2026. Two emergency physicians independently assessed each response. They considered agreement with the reference diagnosis, the Canadian Triage and Acuity Scale (CTAS) triage level, response appropriateness and overall quality. Appropriateness and quality were rated on five-point scales.
Every model correctly named the reference diagnosis in all 17 cases. CTAS exact agreement was lower, ranging from 14 to 15 cases per model. The reported comparison of triage agreement found no statistically significant difference between models (p=0.938). Across 13 discordant triage assignments, nine were undertriage—assigning less urgency than the reference—and four were overtriage.
Response ratings and potential harm flags
The physicians’ appropriateness and overall-quality ratings differed across models (p<0.001 for both measures). Kimi K3 had the highest mean scores, at 4.82 for appropriateness and 4.74 for overall quality. Grok 4.5 had the lowest, at 3.91 and 3.71, respectively; after Holm adjustment, it scored significantly lower than each of the other four models on both scales.
Reviewers flagged nine responses for potential harm: four from Grok 4.5, two each from Gemini 3.6 Flash and GPT-5.6 Sol, one from Claude Fable 5, and none from Kimi K3. The reported between-model comparison for these flags was not statistically significant (p=0.119).
The results show why diagnostic naming alone is an incomplete measure of emergency-care performance. A response can give the reference diagnosis yet still assign a different urgency level, and the study recorded undertriage as well as potential-harm flags. Because the comparison used 17 synthetic vignettes rather than clinical encounters, its findings describe performance on this test set—not established safety in patient care. The preprint was posted on 23 October 2026.