Summary
An Insilico Medicine preprint reports AUROCs of 0.828 for efficacy and 0.809 for safety/toxicity endpoint prediction on a test set of trials initiated in 2021 or later. It also introduces a benchmark of 2,040 ongoing trials for prospective evaluation as outcomes become available.
A medRxiv preprint by researchers affiliated with Insilico Medicine describes inClinico Agent, an AI system for predicting whether clinical trials will meet their primary endpoints. On a test set of trials initiated in 2021 or later, the researchers report an area under the receiver operating characteristic curve (AUROC) of 0.828 for efficacy predictions and 0.809 for safety or toxicity predictions. The work is a preprint, and the authors also introduce a benchmark intended to track predictions on ongoing trials as their results become available.
How the system makes predictions
The system produces a Predictive Trial Index (PTI), a score from 0 to 1 that the authors describe as reflecting the probability of trial success. It uses separate models for trials whose primary endpoints concern efficacy and those focused on safety or toxicity, spanning phases I through IV.
The models use information intended to be available when a trial is registered: target and mechanism-of-action precedent, drug modality and novelty, trial design and operational characteristics, and literature-derived embeddings related to target choice. The authors say this registration-time approach is designed to prevent lookahead bias and outcome leakage—using information that would not have been available when a prediction was made.
InClinico Agent combines these scoring models with large language models, retrieval tools and specialized clinical skills. The reported AUROCs describe how well the models discriminate between trials that do and do not meet their primary endpoints: an AUROC of 0.5 corresponds to chance-level ranking, while 1.0 represents perfect discrimination. AUROC measures ranking performance, not whether a score such as 0.8 corresponds to an 80% success rate.
Reported results and the prospective benchmark
For efficacy, the reported AUROC was 0.828, with a 95% confidence interval of 0.812–0.845, across 2,347 trials. For safety or toxicity, it was 0.809 (95% confidence interval 0.787–0.831) across 1,485 trials. These results are from the models’ test set; the abstract does not describe them as a prospective evaluation.
For that longer-term evaluation, the researchers created ClinBench, comprising 2,040 ongoing trials. They report running inClinico Agent and five frontier large language models on the benchmark, with predictions intended to be assessed as trials reach completion. The abstract does not provide results from that follow-up, so ClinBench is a framework for prospective measurement rather than evidence here of how accurately the systems forecast those ongoing trials.
The report is a medRxiv preprint. All authors are affiliated with Insilico Medicine, which the preprint describes as a commercial AI company, and the curated trial database used to train the models is not publicly available. The authors say the prospective ClinBench predictions are provided in Supplementary Table 1.