psychometric_modeling

Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

Authors: Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Hendrik Drachsler, Ulf Kroehne
Venue / Source: arXiv (2026)

Language models with matching overall grading accuracy can diverge dramatically when scoring difficult, semantically ambiguous student responses.

Key takeaways

Language models with similar aggregate accuracy diverge substantially in their robustness against difficult student responses.
On high-difficulty answers, AI graders systematically collapse toward the intermediate partially correct label under semantic ambiguity.
Grading difficulty correlates strongly with semantic misalignment, contradiction against reference answers, and embedding-space isolation.
Testlet response theory models reliably isolate AI grading ability from answer difficulty, facilitating selective human-in-the-loop routing.

Executive Summary (TL;DR)

In plain words In the paper's terms

Standard test scores mask whether an AI grader fails when faced with challenging or ambiguous student answers. By applying psychometric testing theory to evaluate AI graders, this study shows that even top-performing models collapse on difficult answers, defaulting to partially correct labels. This framework provides an interpretable way to identify which student answers need human review rather than automated grading.

Automated short answer grading typically relies on aggregate metrics that mask how model accuracy changes across items of varying difficulty. To address this, the authors fit a 1PL testlet response theory model treating 17 open-weight language models as graders and student responses as items across two science benchmarks. The findings reveal that models with similar overall performance exhibit markedly divergent degradation slopes on difficult responses, which systematically induce an intermediate-label collapse toward the partially correct incomplete category.

Authors' abstract (original)

Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen's kappa. However, these metrics provide limited insight into how grading performance varies across student responses of differing grading difficulty. We introduce an evaluation framework for LLM-based ASAG based on item response theory (IRT), which models grading correctness as a function of latent grader ability and response grading difficulty. This formulation enables response-level analysis of where LLM graders succeed or fail and reveals robustness differences that are not visible from aggregate scores alone. We apply the framework to 17 open-weight LLMs on the SciEntsBank and Beetle benchmarks. The results show that even models with similar overall performance differ substantially in how sharply their grading accuracy declines as response difficulty increases. In addition, confusion patterns show that errors on difficult responses concentrate disproportionately on the \texttt{partially\_correct\_incomplete} label, indicating a tendency toward intermediate-label collapse under ambiguity. To characterize difficult responses, we further analyze semantic and linguistic correlates of estimated difficulty. Across both datasets, higher difficulty is associated with weaker semantic alignment to the reference answer, stronger contradiction signals, and greater semantic isolation in embedding space. Overall, these results show that item response theory offers a useful framework for evaluating LLM-based ASAG beyond aggregate performance measures.

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.5/10

Engineering Countermeasures

6.5/10

Consensus Challenge

6.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a formal psychometric modeling framework (IRT / testlet response theory) for evaluating LLMs as raters in constructed-response assessment.
Demonstrates how to measure item and response difficulty to detect systematic rater distortion and label collapse.
Offers an empirical methodology for auditing AI grading resilience and structuring selective human-in-the-loop scoring.

Methodology and Dataset

Empirical psychometric modeling applying a 1PL testlet response theory model via regularized negative log-likelihood (L-BFGS) to 17 open-weight LLMs (0.8B to 14B parameters: Gemma, Qwen, Phi, Llama, Ministral, OpenChat) on the 5-way SciEntsBank and Beetle benchmarks (SemEval-2013 Task 7). Student demographics comprise undergraduate/tutorial science responses from the benchmark corpora; evaluations are deterministic with greedy decoding (temperature=0).

Why Worth Reading

This paper directly bridges psychometric item response theory and generative AI assessment. It shifts the evaluation of LLMs as automated graders from crude summary accuracy to a nuanced measurement framework, showing how and why AI raters fail on ambiguous student constructed responses.

Key Claims & Verified Quotes

Models with similar overall accuracy show substantially different degradation slopes across response difficulty tiers.

Quote: “The results show that even models with similar overall performance differ substantially in how sharply their grading accuracy declines as response difficulty increases.” — 10.48550/arxiv.2605.00238, p. 1, para. 7 (exact)

High response difficulty causes LLM graders to disproportionately misclassify responses into the partially correct category.

Quote: “In addition, confusion patterns show that errors on difficult responses concentrate disproportionately on the partially_correct_incomplete label, indicating a tendency toward intermediate-label collapse under ambiguity.” — 10.48550/arxiv.2605.00238, p. 1, para. 7 (exact)

Response difficulty is consistently linked to lower semantic overlap and reference contradiction rather than basic surface length. overclaim

Quote: “Across both datasets, higher difficulty is associated with weaker semantic alignment to the reference answer, stronger contradiction signals, and greater semantic isolation in embedding space.” — 10.48550/arxiv.2605.00238, p. 1, para. 7 (exact)

Limitations & Study Design Flags

Single detector: not mentioned in the text

Evaluation is restricted to 17 open-weight LLMs without commercial proprietary baselines like GPT-4 or Claude.
Analyses are confined to two science benchmarks using a specific five-way labeling taxonomy.
Grading correctness is binarized in the 1PL IRT model rather than using polytomous IRT formulations for multi-class error types.
Correlational feature analyses characterize difficulty properties without establishing underlying causal mechanisms.

From the article text (quote verified)

supports Aggregate metrics obscure item difficulty and model robustness in NLP evaluation. «Prior work shows that benchmark-level averages can obscure substantial variation in item properties such as difficulty and discriminability, while offering limited separability among strong models (Lalor et al., 2016, 2018).», p. 2, para 10
supports Item response theory effectively models latent grader abilities and evaluation instances. «Prior work has shown that IRT can support more informative evaluation in NLP when benchmark instances vary substantially in difficulty (Lalor et al., 2016, 2018).», p. 1, para 12

Model inference (gaps & contradictions)

[metric / methodological gap]: Aggregate ASAG metrics (accuracy, F1, quadratic weighted kappa) mask model-specific failure rates on boundary cases and ambiguous student answers.
[definition / conceptual contradiction]: Treating difficulty as an inherent property of student text rather than an LLM confidence artifact provides an interpretable anchor for human-in-the-loop triaging.

#item response theory #automated short answer grading #LLMs as raters #testlet response theory #rater drift #measurement validity