Summary

A Nature Medicine study describes a fully on-premise clinical AI agent that combines diagnostic reasoning with repeated-run consistency checks. On a seven-disease benchmark, the system retained 98.9% accuracy among cases selected for autonomous handling at a defined consistency threshold, while routing the rest for review.

Researchers have developed and evaluated a fully on-premise medical AI agent that combines local processing with decision-time reliability checks. In simulations using clinical reasoning benchmarks, the system used the consistency of repeated diagnostic runs to identify cases suitable for autonomous handling and route less stable cases to clinician review.

The study, published in Nature Medicine, reported 90.04% accuracy on a seven-disease benchmark and 83.8% on a four-disease abdominal-diagnosis benchmark. At a behavioural-consistency threshold of 0.90, the agent retained 49.4% of cases while achieving 98.9% accuracy in that selected group.

Contents

A local agent with a reliability gate

The system used two interacting components: a Physician Agent powered by an on-premise large language model and a Patient Agent that simulated the clinical encounter. The Physician Agent could gather information through structured tools, including laboratory results, urinalysis, radiology reports, microbiology findings and physical-examination data. It then produced a diagnosis and a reasoning trace.

All processing took place in an institutionally governed local environment. The study authors describe this as operational trust: control over health data, model versions, auditability and deployment. A separate requirement was decisional trust—estimating whether a particular output was stable enough to support autonomous handling.

For that second function, the researchers measured three types of signal: internal model likelihood, uncertainty expressed in language and behavioural stability across repeated runs. Behavioural stability measured whether independent stochastic encounters led to semantically consistent diagnoses.

Benchmark performance and reliability signals

The primary MIRA-v2 benchmark contained 551 cases covering seven conditions: appendicitis, cholecystitis, diverticulitis, pancreatitis, pneumonia, pulmonary embolism and urinary tract infection. Using the same agent architecture, Qwen-3.5 achieved the highest on-premise result at 90.0% in the detailed benchmark analysis, compared with 90.7% for the cloud baseline used by the researchers.

On the separate Clinical Decision Making benchmark, which contained 2,400 cases across four abdominal conditions, Qwen-3.5 reached 83.8% accuracy and GLM-4.5-Air reached 81.2%.

The strongest diagnostic reliability signal was ConsistencyDx, the stability of the final diagnosis across repeated runs. On MIRA-v2, it discriminated correct from incorrect diagnoses with an area under the receiver-operating-characteristic curve of 0.860. Internal probability produced an AUC of 0.747. In an external PubMed-derived benchmark called VivaBench, consistency remained the strongest evaluated signal, although its discrimination was lower, with an AUC of 0.719.

A blinded physician review of 181 MIRA-v2 cases judged 81.8% valid for both the agent diagnosis and the electronic-health-record label. In 4.4% of cases, physicians considered the agent diagnosis clinically valid even though it differed from the record label, illustrating how single-label benchmarks can treat clinically plausible alternatives as errors.

How selective autonomy worked

The researchers tested a triage policy in which cases above a consistency threshold were retained for autonomous handling and the rest were deferred for review. At ConsistencyDx ≥0.90, 272 of 551 MIRA-v2 cases were retained. The selected cases had 98.9% diagnostic accuracy, with three residual errors; the other cases were routed to human review.

The same signal also responded to poorer information. When the simulated patient’s initial history was withheld and the Patient Agent could provide unreliable testimony, overall accuracy fell from 90.6% to 70.2%. Consistency declined sharply and still separated correct from incorrect cases, producing an AUC of 0.875. By contrast, internal probability and language-based certainty could remain high even as accuracy fell, including in pneumonia cases.

These results position repeated-run consistency as a routing signal rather than a guarantee of correctness. It can concentrate unstable outputs in a review queue, but the study’s triage simulation still contained errors in the autonomous stream.

Deployment trade-offs

Consistency estimation required five independent multi-turn encounters per case. This increased Physician Agent token use by approximately five times compared with single-pass inference, although the runs could be parallelised. The threshold was also configuration-dependent: decoding temperature, the number of repeated runs and the semantic-embedding model changed the balance between retained accuracy and coverage.

The evidence came from retrospective simulations using deidentified MIMIC-IV data and physician-curated cases, not from prospective clinical use. The main benchmarks were derived from a single institutional data source, and the evaluation focused on text-based diagnostic reasoning rather than native medical-image interpretation. The authors also observed lower accuracy in older age groups in descriptive subgroup analyses, making dedicated bias audits relevant before any clinical deployment.

The study therefore presents a framework for governed, confidence-based escalation rather than a ready-to-use autonomous diagnostic system. Its central result is that local control and decision-time reliability estimation can be combined, with behavioural consistency providing the most useful of the evaluated signals for selecting cases for further automation.

Sources