Summary
A South African benchmark of 4,999 verbal-autopsy records found supervised AI models improved cause-of-death assignment over the InSilicoVA comparator. Combining interview narratives with symptom questionnaires produced different gains depending on the metric.
A medRxiv preprint reports that AI models using verbal-autopsy records improved cause-of-death assignment over the InSilicoVA automated comparator in a benchmark of 4,999 deaths in South Africa. The records covered people aged 12 years or older, and physician-assigned causes of death served as the reference. All supervised model pipelines performed better than InSilicoVA on both individual assignments and estimates of the population distribution of causes.
In a verbal autopsy, information about a death is collected through a structured symptom questionnaire and a free-text narrative. Such records are important where deaths lack medical certification: the study notes that fewer than 10% of deaths in Africa are medically certified. Existing automated verbal-autopsy algorithms were designed around structured responses, while narratives could inform physician assignments but were not used in algorithmic assignment.
Why verbal-autopsy narratives matter
The researchers compared models that used each data type separately with approaches that combined them. They fine-tuned pretrained language models on narratives, trained machine-learning models on questionnaire responses, and combined the two sources at the data, feature or decision level. They also tested a locally deployed generative large language model by prompting it with the inputs, without updating its parameters.
The narrative-only and questionnaire-only pipelines performed similarly overall, but their strengths differed by cause. Of the deaths, 63.9% received the same prediction from both approaches; 11.6% were correctly classified only by the narrative pipeline and 13.6% only by the questionnaire pipeline. Narratives were more accurate for noncommunicable causes, while structured responses did better for communicable causes. The results suggest the two sources can provide complementary signals.
Different models led on different measures
Every supervised pipeline exceeded InSilicoVA by 8.5–15.9 percentage points in top-1 accuracy—the share of records assigned the reference cause as the model’s leading prediction—and by 3.5–7.1 points in cause-specific mortality fraction (CSMF) accuracy, which measures how closely estimated cause shares match the reference population distribution.
Decision-fusion models, which combine model outputs at the decision stage, added 4.2–5.4 points in top-1 accuracy over unimodal approaches, with gains concentrated among common causes. Feature fusion using cross-modal attention performed better on less common causes, adding 6.6 points in balanced accuracy (95% CI, 1.7–11.5) and 5.5 points in macro F1 (95% CI, 0.6–10.3). It achieved the highest CSMF accuracy, at 95.7%. However, paired differences against the better unimodal model ranged from −0.2 to +3.4 points, with all intervals including zero.
The prompted large language model had the highest balanced accuracy among the tested pipelines, at 60.6%; supervised models had higher CSMF accuracy. Supervised gains over InSilicoVA held across all nine held-out South African provinces for top-1 accuracy and in nearly every comparison for macro F1 and CSMF accuracy.
These findings come from a national benchmark, not evidence of performance in every health system. The reference was physician-assigned cause of death, and the authors identify validation across other settings and age groups as a step toward broader implementation. The work is a preprint.