A third of ICML was reproduced in three weeks

Between 15 July and 2 August, 1,221 people attempted to reproduce 2,226 papers from ICML 2026 — about a third of the 6,352 accepted from 23,918 submissions. The exercise produced 6,816 published logbooks and verdicts on 35,908 separate claims.

The headline split is uncomfortable reading. At least one claim was independently verified in 51% of the papers examined, or 1,103 of them, with 3,978 individual claims confirmed by experiment. Against that, 23% — 496 papers — had at least one claim falsified or contested, and 49 papers had every claim falsified.

A further group sits in neither camp: 502 papers yielded only toy-scale evidence, and 280 produced no established result at all, most often because artefacts were missing.

The number that should worry reviewers

In 242 cases, independent teams reached opposite verdicts on the same paper. That is not a finding about the papers so much as about the reproduction process itself: two competent attempts at the same claim can disagree, which means a single reproduction result carries less weight than it appears to.

The organisers are explicit that unattended agents were not sufficient. Runs were carried out with Claude Code, Codex, Cursor and OpenResearch's orx, with an automated judge issuing verdicts, but human oversight was required throughout — agents struggled in particular to notice when behaviour depended on scale, and needed steering to produce reliable results.

For anyone building on published benchmarks, the practical reading is that a claim in an accepted paper is a hypothesis about a result, not the result. The rate at which claims fail here is high enough that citing a number without checking it is now a measurable risk rather than a theoretical one.