Which AI model is truly the best: how reliable are public benchmarks of frontier AI models?
Authors: Miloš Kankaraš
Venue / Source: Zenodo (CERN European Organization for Nuclear Research) (2026)
Frontier AI leaderboards cannot statistically distinguish top-ranking models because typical lead margins fall well inside standard item-sampling noise.
Key takeaways
- Most AI benchmark leaderboards cannot statistically distinguish among their top five models due to substantial item-sampling error.
- Benchmark resolution adheres strictly to the square-root law of test length: halving measurement error requires quadrupling the number of items.
- Evaluating thousands of models does not reduce score error or improve resolution if the underlying item bank remains fixed.
- Changes in evaluation harness versions can shift agent benchmark scores by fifteen points, dwarfing the model differences being ranked.
- Leaderboards should report confidence intervals and statistical ties rather than presenting unverified deterministic rank orders.
Executive Summary (TL;DR)
Authors' abstract (original)
Public leaderboards order AI models by a point score and the public debate about them turns on leads of one or two points. We ask how many of those leads the benchmarks can detect. Using only the item-level results that the benchmarks themselves publish, we computed a psychometric report card for 61 public leaderboards drawn from HELM, the Desai et al. release, SWE-bench Verified, tau2-bench and the archived Open LLM Leaderboard, applying one instrument throughout: exact paired tests on shared items with false-discovery control, item-sampling error, and a tier count down each ranking. 55 of the 58 leaderboards that discriminate at all cannot fully separate their own top five models, 16 separate none of those ten pairs, and in 40 the leader cannot be told apart from at least one other model. On the benchmarks that decide current headlines, Gemini 3 Pro and GPT-5 are statistically tied on GPQA; on SWE-bench Verified, with the agent held fixed, Claude Opus 4.5 and 4.6 are indistinguishable; among the newest models on tau2-bench no named head-to-head is a real difference. Precision follows the square-root law in the number of items (slope -0.49), and a cohort 84 times larger does not improve resolution. We also find that a change of harness build moves one model's tau2-bench score by 14.9 points, more than the gaps it is used to rank, and that published leaderboard data carry defects that can change a model's standing. We recommend that leaderboards report ties and error bars, fix and disclose their harness, and size their benchmarks to the differences they are used to adjudicate.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a quantitative psychometric template for auditing LLM performance as test-takers on institutional or standardized examinations.
- Highlights why educational assessment redesign cannot rely uncritically on vendor-reported benchmark leads when selecting generative models.
- Demonstrates exact statistical methods (Clopper-Pearson, paired McNemar with FDR correction, bootstrap resampling) for evaluating AI score shifts and detecting genuine capability drift.
Methodology and Dataset
Secondary psychometric re-analysis of item-level response data from 61 public leaderboards (HELM [17 suites], Desai et al. [40 suites], Open LLM Leaderboard v1 archive [1 suite], SWE-bench Verified [2 suites], tau2-bench banking [1 suite]). Sample sizes ranged from 11 to 6,721 models (including Gemini 3 Pro, GPT-5, Claude Opus versions, o3, Grok 4) across test banks ranging from 97 to 14,556 items. No human students were evaluated.
Why Worth Reading
For assessment researchers and psychometricians, this paper demonstrates the direct relevance of classical testing theory, item-sampling error, and multiple comparisons to frontier AI evaluation. It provides educational measurement specialists with rigorous empirical ammunition against the deterministic interpretation of LLM evaluation benchmarks, illustrating how unadjusted leaderboards suffer from severe measurement noise, harness drift, and lack of resolution at the top tier.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- No adversarial testing: not mentioned in the text
- Focuses strictly on rank resolution and measurement error without auditing the construct validity of whether benchmarks assess their intended latent skills.
- Dependent on maintainer-published raw test runs; 24 candidate benchmarks were omitted due to unreleased item-level logs, continuous metrics, or small model cohorts.
- Treats item banks as random samples from hypothetical super-populations of tasks rather than assessing strictly fixed-test sample mastery.
- supports Validates earlier work showing that SWE-bench has converged and can no longer resolve top entries. «Our SWE-bench cards confirm that claim with the same instrument used everywhere else, add a comparison with the agent held fixed, and extend the analysis of agentic benchmarks to harness versions on tau2-bench», p. 2, para 6
- supports Confirms findings that evaluation harness discrepancies create major variance in benchmark outcomes. «A change of harness build moves one model's tau2-bench score by 14.9 points, more than the gaps it is used to rank», p. 1, para 6
- [method / conceptual blindspot]: Standard AI leaderboards treat point differences as deterministic rankings, whereas classical psychometrics treats individual test scores as random variables subject to standard measurement error.
- [metric / underpowered resolution]: Public benchmarks are severely undersized (e.g., 97 to 500 items) relative to the narrow margin of differences (1-3%) claimed by frontier LLM developers.