Evaluating Large Language Models through Multidimensional Item Response Theory: A Comprehensive Case Study on ENEM
L. Taschetto, Renato Fileto · Brazilian Symposium in Information and Human Language Technology (2025) · doi:10.5753/stil.2025.37846 · gemini-3.7-flash
TL;DR
Evaluating AI models on standardized tests by counting correct answers hides their real strengths and weaknesses. By using the same psychometric scoring model applied to college applicants, researchers found that AI models with identical overall scores can differ by an entire letter grade in mathematics. This shows that standard test accuracy gives educators a misleading picture of AI capabilities.
Method and sample: Psychometric evaluation evaluating 10 LLMs (DeepSeek R1, DeepSeek V3, GPT-4o, O1, O3, Llama 4 Maverick, Llama 4 Scout, GPT-4o Mini, O3 Mini, O4 Mini) on 358 items across the 2022 and 2023 Brazilian ENEM exams (45 items per domain: Mathematics, Natural Sciences, Human Sciences, Languages; 2 items annulled). LLMs were evaluated across few-shot and chain-of-thought conditions yielding 19,332 item-level responses (54 configurations). Human baseline calibration used ENEM microdata from 665,421 candidates who took identical booklet codes. Latent traits were estimated using 3-PL unidimensional IRT and a four-factor 3-PL multidimensional IRT (MIRT) model via Quasi-Monte Carlo Expectation-Maximization in the R 'mirt' package.
#item response theory#multidimensional IRT#LLM benchmark#ENEM#psychometrics#higher education assessment#construct validity