Auditing LLM Benchmarks with Item Response Theory
Authors: Sander Land, Daniel M. Bikel
Venue / Source: arXiv (2026)
Item Response Theory surfaces hidden test bank errors at ninety-five percent precision by detecting when top-performing models systematically reject answer keys.
Key takeaways
- Unsupervised Item Response Theory accurately isolates test key errors by modeling where high-ability test-takers systematically diverge from the reference answer.
- Standard test accuracy masks underlying answer key corruption, which propagates through derivative assessment datasets and artificially distorts evaluator scoring.
- Automated rating models frequently specialize in stylistic traits like brevity or structure rather than factual correctness when grading complex text responses.
- Answer key audit techniques using forced-ceiling psychometric models provide a scalable quality-control pipeline for maintaining large-scale digital item banks.
Executive Summary (TL;DR)
Authors' abstract (original)
LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Demonstrates how modern psychometric modeling (4PL Item Response Theory) detects validity threats, keying errors, and item compromise in large digital assessment banks.
- Provides statistical methods for evaluating whether automated AI raters/judges evaluate substantive construct accuracy or superficial stylistic markers.
- Shows how differential item performance patterns reveal test bank corruption and contamination.
Methodology and Dataset
Evaluated 114 Large Language Models (spanning 2023–2026 releases from 0.8B parameters to frontier flagships, including 6 dedicated reward models) across 20,986 test items spanning 32 subsets from 7 preference and multiple-choice benchmarks (RewardBench 1, RewardBench 2, RM-Bench, JudgeBench, GPQA Diamond, MATH-MC, GSM-MC). Evaluated against 2PL IRT, plain 4PL IRT, GLAD, top-k disagreement heuristics, and supervised XGBoost classifiers.
Why Worth Reading
This paper demonstrates a powerful application of modern Item Response Theory (4PL IRT) to audit digital test banks and identify compromised, ambiguous, or miskeyed items. For assessment researchers and educators using automated grading or LLM judges, it provides a mathematical framework for detecting answer key degradation and understanding the systematic biases of automated raters.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Unrepresentative sample: not mentioned in the text
- Single detector: not mentioned in the text
- No adversarial testing: not mentioned in the text
- Monolingual: not mentioned in the text
- Evaluation relies on a GPT-5.4 label aggregator as a proxy for gold labels due to the impracticality of humanly re-annotating 20,000 items.
- Assumes conditional independence between model responses given latent ability, whereas models sharing pretraining data or architectures exhibit residual correlations.
- Cannot distinguish between genuine answer key mislabels and items where weaker models guess while all strong models converge on a highly persuasive distractor.
- supports Rodriguez et al. (2021) applied 4PL IRT upper asymptote parameters to measure item feasibility in natural language processing test sets. «Rodriguez et al. (2021) use a 4PL upper asymptote as a per-item feasibility score to filter unsolvable items.», p. 2, para 1
- supports Label errors are pervasive in widely used benchmark test sets and cause downstream distortions in model evaluations. «Northcutt et al. (2021a,b) introduce Confident Learning to detect supervised label errors and show that such errors are pervasive in widelyused test sets, with downstream effects on model rankings.», p. 2, para 3
- supports IRT item parameterization uncovers internal test structures obscured by aggregate accuracy statistics. «Vania et al. (2021) showing that IRT reveals structure invisible to aggregate accuracy.», p. 2, para 1
- [metric / psychometric modeling / methodological extension]: Extends traditional 2PL and 3PL IRT psychometric applications by framing item-key error detection as a forced-ceiling likelihood ratio contrast (4PL delta log-likelihood).
- [sample / assessment domain / domain transfer to human education]: While evaluated on LLM benchmark datasets, the mathematical formulation directly applies to auditing human standardized assessment item banks for answer key errors and compromised items.