Summary
A medRxiv preprint reports a remote, voice-based method for testing patient-facing AI intake chatbots with standardized-patient encounters. In a pilot, the chatbot covered fewer critical history elements in harder cases, while summary ratings remained near ceiling.
A medRxiv preprint describes a remote, voice-based method for evaluating patient-facing AI intake chatbots through simulated clinical conversations. In a pilot involving ChatGPT 4o, the chatbot asked about and satisfactorily addressed a smaller share of critical history elements in harder versions of four primary-care cases than in easier versions. Standardized patients also reported missed information and rushed pacing, even as ratings of the chatbot’s summaries were near ceiling.
The approach is intended to assess more than the quality of a final written summary: it examines what the chatbot asks during the conversation and how it responds. The authors say this can help identify conversational gaps in pre-clinical evaluation.
How the evaluation worked
A standardized patient is a person trained to portray a clinical case consistently, allowing performance to be assessed across repeatable scenarios. The researchers developed four primary-care cases, each with an easy and a difficult variant, and tested the method in three phases.
In Phase 1, five standardized patients completed 40 encounters with internal medicine residents. The residents rated how faithfully the cases were portrayed; the average fidelity score was 4.52 out of 5. In Phase 2, the standardized patients completed 80 encounters with a ChatGPT 4o chatbot. They recorded whether critical elements were asked about and addressed satisfactorily, rated the resulting summaries, and took part in four debriefing interviews. In Phase 3, two blinded clinicians assessed transcripts and summaries from 101 encounters: 32 resident-led and 69 chatbot-led.
Harder cases exposed weaker history coverage
The chatbot asked about 59.3% of critical elements in easy variants, compared with 50.2% in difficult variants. The adjusted odds ratio was 1.49 (95% CI, 1.15 to 1.93). It satisfactorily addressed 73.3% of elements in easy cases and 64.6% in difficult ones (OR 1.47; 95% CI, 1.12 to 1.93). Summary-quality ratings were near ceiling and did not differ between the two levels of difficulty.
In an exploratory comparison, residents asked about 12.5 percentage points more critical elements than the chatbot (95% CI, 7.0 to 18.1) and satisfactorily addressed 9.2 percentage points more (95% CI, 3.8 to 14.6). The reported totals did not differ. These comparisons are part of the pilot rather than a broad assessment of clinician and chatbot performance.
The authors also found that agreement between reviewers depended on the measure: agreement on individual element judgments ranged from moderate to substantial (AC1 0.49 to 0.70), while agreement on totals was poor (ICC 0.10 and 0.21). That distinction matters when interpreting scores produced by the evaluation method. The study is a preprint pilot based on four simulated cases and one chatbot; it offers a way to examine conversational performance, not evidence about patient outcomes in clinical use.