Evaluating Large Language Models through Multidimensional Item Response Theory: A Comprehensive Case Study on ENEM
Authors: L. Taschetto, Renato Fileto
Venue / Source: Brazilian Symposium in Information and Human Language Technology (2025)
Raw test accuracy conceals massive language model skill gaps that multidimensional item response theory readily exposes across subject domains.
Key takeaways
- Raw percentage accuracy misrepresents model capability by treating guessing, easy items, and highly discriminative hard questions identically.
- Multidimensional item response theory separates model proficiencies across distinct academic domains, exposing severe quantitative deficits in models with high overall scores.
- Cross-domain covariance re-weights latent proficiency estimates compared to isolated unidimensional models, penalizing uneven model performance.
Executive Summary (TL;DR)
Authors' abstract (original)
LLM evaluations on tasks like high-stakes multidisciplinary tests still rely on raw accuracy, a metric that weights easy and difficult questions equally and ignores guessing. To help bridge this methodological gap, we repurpose the official three-parameter logistic Item Response Theory (IRT) calibration that the Brazilian education authority (INEP) uses to score humans on the Exame Nacional do Ensino Médio (ENEM), and apply it to LLM responses. We then fit a four-dimensional 3-PL model aligned with ENEM’s knowledge domains. Results show that similar accuracies can mask proficiency gaps exceeding one standard deviation across domains. Mathematics remains the toughest domain for both humans and models, whereas questions on Human Sciences are systematically easier for both.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly models LLM performance using item response theory (3-PL unidimensional and multidimensional), providing empirical tools for detecting AI-vulnerable exam items.
- Exemplifies how psychometric calibration scales can place human examinees and artificial agents onto an aligned, latent-trait continuum.
- Supplies quantitative evidence that test-level aggregate scores fail Messick's structural and substantive validity aspects when LLMs are test takers.
Methodology and Dataset
Psychometric evaluation evaluating 10 LLMs (DeepSeek R1, DeepSeek V3, GPT-4o, O1, O3, Llama 4 Maverick, Llama 4 Scout, GPT-4o Mini, O3 Mini, O4 Mini) on 358 items across the 2022 and 2023 Brazilian ENEM exams (45 items per domain: Mathematics, Natural Sciences, Human Sciences, Languages; 2 items annulled). LLMs were evaluated across few-shot and chain-of-thought conditions yielding 19,332 item-level responses (54 configurations). Human baseline calibration used ENEM microdata from 665,421 candidates who took identical booklet codes. Latent traits were estimated using 3-PL unidimensional IRT and a four-factor 3-PL multidimensional IRT (MIRT) model via Quasi-Monte Carlo Expectation-Maximization in the R 'mirt' package.
Why Worth Reading
This paper provides a direct, methodologically rigorous psychometric bridge between educational measurement and generative AI benchmarking. For assessment specialists and university examiners, it demonstrates concretely why raw test scores fail Kane's validity argument when evaluating LLMs, and proves how 3-PL multidimensional item response theory can diagnose domain-specific LLM performance and item vulnerability under high-stakes conditions.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
- Evaluations were restricted to multiple-choice items and omitted the ENEM open-ended essay component.
- Greedy decoding was fixed at temperature zero, omitting stochastic response variation.
- The analysis focused strictly on Portuguese-language items from the 2022 and 2023 ENEM administrations.
- supports Previous IRT evaluations of LLMs on university admission tests relied on unidimensional Rasch models that ignored discrimination, guessing, and multidimensionality. «Both studies relied exclusively on uni-dimensional 1-PL IRT models. However, as discussed in Section 2, the uni-dimensional IRT may not adequately capture the interdisciplinary nature of university-admission exams», p. 4, para 2
- [metric / methodological divergence]: Standard benchmark pipelines report raw accuracy, which conflates successful guessing and mastery of easy items with true latent proficiency on highly discriminating tasks.
- [definition / construct underrepresentation]: Unidimensional psychometric models collapse multidisciplinary performance into a single scalar trait, masking cross-domain deficits such as severe quantitative reasoning failures.