Which AI model is truly the best: how reliable are public benchmarks of frontier AI models?
Miloš Kankaraš · Zenodo (CERN European Organization for Nuclear Research) (2026) · doi:10.5281/zenodo.22998204 · gemini-3.7-flash
TL;DR
AI leaderboard rankings often celebrate a model leading by one or two percentage points as the new best system. This study applies standard educational testing statistics to the results of 61 major public benchmarks. It shows that almost none of the leaderboards have enough test questions to reliably separate their top-performing models from runner-ups.
Method and sample: Secondary psychometric re-analysis of item-level response data from 61 public leaderboards (HELM [17 suites], Desai et al. [40 suites], Open LLM Leaderboard v1 archive [1 suite], SWE-bench Verified [2 suites], tau2-bench banking [1 suite]). Sample sizes ranged from 11 to 6,721 models (including Gemini 3 Pro, GPT-5, Claude Opus versions, o3, Grok 4) across test banks ranging from 97 to 14,556 items. No human students were evaluated.
#psychometrics#benchmarking#measurement_error#leaderboards#item_sampling_error#LLM_evaluation#McNemar_test#harness_drift