Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset
Authors: Mayank Sharma, Savira Nadela, Tyler Matteson
Venue / Source: arXiv (2026)
Widely used LLM benchmarks like HLE often rely on domain subscores that, upon psychometric analysis, turn out to measure only general reasoning.
Key takeaways
- HLE domain subscores do not represent empirically distinct latent capabilities, functioning instead as a measure of general reasoning.
- The benchmark lacks sufficient measurement precision to differentiate effectively among high-performing frontier models.
- IRT-based dimensionality analysis requires careful calibration against a simulated null distribution when sample sizes are small.
- Engineering items provide the most discrimination for top-tier models, despite representing a tiny fraction of the benchmark.
- Benchmark evaluation should prioritize model coverage and psychometric validation to ensure reported metrics represent meaningful skill profiles.
Executive Summary (TL;DR)
Authors' abstract (original)
Humanity's Last Exam (HLE) is widely used to evaluate frontier language models. HLE organizes its questions into eight subject-domain categories, whose subscores are often interpreted as evidence of distinct capabilities. However, no study has assessed whether these labels correspond to empirically separable latent constructs, nor whether the benchmark effectively differentiates between models of similar ability. We evaluate 29 LLMs on the text-only multiple-choice subset of HLE and apply psychometric methods to assess both the dimensionality of the benchmark and the distribution of its measurement precision. Fitting a two-parameter logistic IRT model, we find that HLE is dominated by a single general reasoning factor: domain labels explain only 3.5\% of variance in the leading principal components of item responses, and within- and between-domain residual correlations are nearly identical (Cohen's $d = 0.016$). A simulation study with the same number of models ($N = 29$) shows that our analyses would detect clearly distinct domain abilities, so their absence is informative; however, small domain-specific differences cannot be ruled out, and model rankings vary across domains somewhat more than a single factor predicts. A separate analysis of the test information function reveals that measurement precision concentrates at moderate ability levels and drops sharply above $θ= 0$, where the strongest models sit. These findings suggest that the subset's domain subscores do not warrant distinct capability interpretations and that its ability to discriminate among the strongest models might be limited.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a methodology for auditing the validity of assessment instruments in an AI-saturated context.
- Demonstrates how to evaluate the 'measurement precision' of tests, which is critical for identifying grade inflation or ceiling effects.
- Highlights the risks of relying on subscore categories without psychometric proof of dimensionality.
Methodology and Dataset
The study evaluates 29 contemporary LLMs (including OpenAI o-series, Claude Opus/Sonnet/Haiku, Gemini, DeepSeek, and various open-weight models) on 428 multiple-choice items from the HLE test, using a two-parameter logistic (2PL) IRT model and dimensionality analysis calibrated with power simulations.
Why Worth Reading
This paper provides a rare, rigorous psychometric audit of a high-stakes AI benchmark, challenging the common practice of using domain-specific subscores in leaderboards and offering a template for evaluating future benchmarks.
Key Claims & Verified Quotes
No verified quotes available for this paper.
Limitations & Study Design Flags
- Monolingual: not mentioned in the text
- The analysis is restricted to the text-only multiple-choice subset (21% of total HLE), excluding exact-match and multimodal items.
- The model sample size (N=29) is small and non-random, limiting the scope for formal measurement invariance testing.
- The study assumes a deterministic response per item, which does not account for within-model response variability.
- The small sample size limits the statistical power to detect weak domain-specific differences.
- supports Standard reporting of aggregate and domain subscores in benchmarks lacks empirical justification for assuming distinct latent factors. «Most benchmark studies, including HLE, report aggregate accuracy and domainspecific subscores without testing whether the reported domains correspond to empirically distinct latent factors.», p. 1, para 8
- [method / sample size]: The authors identify that standard psychometric methods, like CFA, were impossible due to a small model sample (N=29) compared to the item pool (J=428).