psychometric_modeling

Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation

Authors: Gili Lior, Tzviel Frostig, Gabriel Stanovsky, Matan Eyal
Venue / Source: arXiv (2026)

Decomposing psychometric difficulty and discrimination across languages reveals translation flaws and cultural bias that raw test scores routinely hide.

Key takeaways

Multilingual-IRT allows leaving up to 60% of multilingual test items unobserved with negligible loss in ability estimation and score prediction.
Language-specific difficulty deviations distribute translation error detection evenly across all languages, whereas accuracy-based metrics heavily over-concentrate on a few low-resource languages.
Separating language discriminability from content discriminability identifies culture-specific questions that are invisible to raw score differences.
The learned cross-lingual correlation matrix autonomously recovers linguistic family clusters and resource gradients without any external supervision.

Executive Summary (TL;DR)

In plain words In the paper's terms

Evaluating AI models across dozens of translated tests is expensive and prone to hidden translation mistakes and cultural bias. This paper develops an advanced testing model that separates pure subject mastery from language and translation difficulties. The method accurately predicts test scores from sparse data and pinpoints faulty translations that traditional percentage scores miss.

Parallel multilingual large language model benchmarks suffer from high evaluation costs, undetected automated translation errors, and conflated cultural knowledge. To address this, the authors develop Multilingual-IRT, extending Item Response Theory with language-specific difficulty deviations, split content-versus-language discriminability, and correlated per-language ability residuals. Evaluated on 25 models across 29 languages in MMLU-Pro-X, the model predicts held-out instances with 11–16% lower binary cross-entropy than accuracy baselines while effectively surfacing translation flaws and culture-specific items.

Authors' abstract (original)

Multilingual benchmarks are central to evaluating large language models (LLMs) across languages, but they suffer from three issues: exhaustive evaluation scales linearly with the number of languages, automatic translation introduces errors that are easily missed at scale, and some items conflate general and culture-specific knowledge. We address all three with a unified statistical framework, Multilingual-IRT, which extends Item Response Theory with per-language difficulty deviations, split discriminability separating content from language effects, and per-language ability residuals. Fitting Multilingual-IRT on 25 LLMs across 29 languages of MMLU-Pro-X, we show that its fitted parameters support three practical applications: predicting unobserved (item, LLM, language) instances with 11-16% lower binary cross-entropy than the strongest accuracy-based baseline, surfacing candidate translation errors distributed across all 28 non-English languages, whereas accuracy-based baselines concentrate detections in a few languages, and recovering culture-specific items that accuracy-based baselines miss.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

7.5/10

Consensus Challenge

6.5/10

Topic Relevance

7.0/10

Relevance for Educators & Research

Provides a mathematical template for modeling differential item functioning (DIF) and construct-irrelevant variance when assessments are presented across diverse cohorts or translated formats.
Demonstrates how Item Response Theory can quantify item parameter drift and benchmark contamination introduced by automated translation or generative AI artifacts.
Offers a principled alternative to naive percentage-correct metrics for evaluating LLM test performance, establishing validity arguments around language vs. construct mastery.

Methodology and Dataset

25 instruction-tuned and reasoning open-source LLMs (2B to 122B parameters across Aya, Qwen3.5, OLMo-3, Gemma-3, Llama-3, Phi-4, GPT-OSS); MMLU-Pro-X benchmark comprising 11,759 multiple-choice items across 29 languages (~8.5 million item responses); fitted using stochastic variational inference (SVI) in NumPyro; compared against parametric IRT ablations (Indep-IRT, Parallel-IRT, Coupled-IRT) and non-parametric accuracy baselines; validated via LLM-as-a-judge (Gemini-2.5-Flash, GPT-5.4 mini, Claude Haiku 4.5) and human expert annotations.

Why Worth Reading

The paper provides a mathematically elegant psychometric framework for separating construct ability from linguistic artifacts in parallel assessments. For assessment researchers grappling with generative AI, it demonstrates how multi-parameter IRT formulations can quantify differential item functioning, isolate construct-irrelevant translation noise, and audit benchmark integrity without requiring exhaustive, expensive evaluation.

Key Claims & Verified Quotes

Multilingual-IRT reaches near-maximal predictive accuracy from a small fraction of observed test instances.

Quote: “Multilingual-IRT reaches nearly its full predictive performance from a small fraction of observed instances.” — 10.48550/arxiv.2606.15643, p. 5, para. 20 (exact)

Sixty percent of (item, LLM, language) instances can be left unobserved with negligible loss in prediction quality.

Quote: “In practice, this means 60% of the instances can be left unseen with negligible loss in prediction quality.” — 10.48550/arxiv.2606.15643, p. 5, para. 20 (exact)

The asymmetric formulation of Multilingual-IRT resolves rotational indeterminacy and is identifiable and reproducible across random seeds.

Quote: “Together these confirm that the Multilingual-IRT formulation we suggest is not just identifiable in principle but also reproducible in practice.” — 10.48550/arxiv.2606.15643, p. 4, para. 15 (exact)

Limitations & Study Design Flags

Unrepresentative sample: not mentioned in the text
Monolingual: not found
No adversarial testing: not mentioned in the text
Single detector: not mentioned in the text

Requires aligned parallel item translations across languages, preventing application to independent, non-parallel multilingual corpora.
Estimation precision of per-language difficulty deviations (dil) and off-diagonal covariance entries is constrained by evaluating only 25 LLMs.
Error detection and cultural validation rely heavily on LLM-as-a-judge proxies, though partially corroborated by human annotators.

Model inference (gaps & contradictions)

[definition / construct underrepresentation]: Standard multilingual evaluations treat test items as uniform across translations, ignoring construct distortion caused by semantic drift, translation loss, or unintended language shortcuts.
[metric / metric inconsistency]: Raw accuracy metrics fail to separate item difficulty from language ability, over-flagging errors in low-resource languages while missing subtle translation flaws and cultural specificity in high-resource languages.

#Item Response Theory #Multilingual Evaluation #Differential Item Functioning #Benchmark Validity #Translation Error Detection #LLMs as Test Takers #Psychometrics