Summary
A randomized experiment at ICML 2026 found near-zero effects of LLM-use policies on paper decisions, scores and reviewer confidence. In a post-study survey, reviewers reported substantial noncompliance with both restrictive and permissive rules.
A randomized experiment at ICML 2026 found that assigning reviewers different rules for using large language models (LLMs) had near-zero effects on final paper decisions, paper scores and reviewer confidence. Reviews written under the more permissive policy were 5.5–7% longer. A post-study survey also found that reviewers reported using LLMs in ways their assigned policy prohibited.
The findings are reported in an arXiv preprint about LLM use in peer review. ICML 2026 involved more than 24,000 papers and 17,000 reviewers, but the experiment randomized policies only among a subset of main-track papers and reviewers.
What the policy comparison found
Reviewers in the conservative group were prohibited from using LLMs. The permissive policy allowed limited assistance. The researchers compared outcomes following random assignment to these policies, including final paper decisions, scores and reviewers’ confidence.
The authors describe the policies’ effects on those outcomes as near-zero. That finding suggests that, in the part of the review process studied, changing the formal rules did not materially shift the measured outcomes. It does not mean the policies were followed uniformly: the survey responses point to a gap between assigned rules and reported practice.
Reviewers reported violations under both policies
Among 1,486 post-survey respondents, 22.5% of reviewers assigned to the conservative policy said they had used an LLM despite the prohibition. In the permissive group, 36.5% reported at least one use that was explicitly disallowed under that policy.
These figures are reviewers’ survey reports, rather than direct measurements of every reviewer’s activity. The experiment’s randomized comparison and the survey answer different questions: the former measures the effect of policy assignment on review outcomes, while the latter records what respondents said they did.
Together, the results give conference organisers evidence to consider when designing LLM rules and tools for peer review. The study’s authors discuss implications for both. Because the randomized comparison covered only a subset of ICML 2026’s main-track papers and reviewers, its results describe that experiment rather than every review setting.