Summary
A medRxiv preprint introduces KampoBench, a physician-revised benchmark testing 10 large language models on 74 Japanese Kampo consultation scenarios. The results point to a gap between naming relevant information and applying it to a specific case.
A team of Japanese physicians has introduced KampoBench, a benchmark for testing how large language models (LLMs) respond to consultation scenarios in Kampo, Japan’s traditional medicine. In a medRxiv preprint posted on 11 October 2026, the researchers report that 10 models were assessed against 74 scenarios and 310 conversation-specific criteria. The results suggest that a model’s ability to name relevant information can differ from its ability to apply that information to an individual case.
Testing consultation judgements, not recall alone
Many existing evaluations use examination questions, which test whether a model can recall information. KampoBench instead asks whether a response meets criteria tailored to a consultation scenario. These include advising when care should be sought and connecting a known risk associated with a crude drug to the circumstances in the case.
The scenarios, criteria and reference answers were first drafted by an LLM, then revised by board-certified Kampo physicians. The researchers evaluated 10 models from three vendors. To account for the possibility that an LLM grader may favour models from its own vendor, each response was assessed by a model from each of the three vendors.
Scores varied with the grader
The highest-scoring model achieved 74.8% to 86.3%, depending on which grader assessed it. The physician-revised reference answers scored 87.9% to 92.4% across graders. The researchers report that model rankings were similar across the different graders, with Spearman’s rho ranging from 0.952 to 0.976. Their estimate of grader preference for its own model family was 0.4 to 0.6 percentage points—below the bootstrap standard deviation of one measurement. Grader leniency itself varied by 14.3 percentage points, so a score is most informative when considered alongside the grader that produced it.
Performance differed across types of judgement. Models scored highest on criteria about advising that care be sought, at 83.6%. They scored lowest—17.5%—on criteria requiring a known crude-drug risk to be tied to the particular case. In 79.2% of responses to those criteria, the model named the drug. The contrast points to a challenge beyond identifying relevant information: connecting it to the patient’s situation.
The study’s authors argue that consultation-style evaluation captures a capability that examination questions alone cannot measure: applying knowledge in context. KampoBench is a test of model responses to scenarios, however, rather than a study of patient outcomes or clinical use. The findings are from a medRxiv preprint; its benchmark files are restricted until journal publication, while the authors say evaluation code and measurement conditions are available on GitHub.
The authors also disclosed financial relationships with Kampo-related companies, including lecture fees and collaborative research funding.