psychometric_modeling

LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora

Authors: Veerendra Kumar Sunkavalli
Venue / Source: arXiv (2026)

Standard agreement metrics for AI graders hide massive rater severity effects that distort student outcomes in absolute assessment contexts.

Key takeaways

LLM judges display significant, systematic differences in scoring severity that are entirely missed by standard rank-based agreement statistics.
Model version updates act as loud instrument changes, causing unpredictable shifts in grading standards that require re-calibration.
Simple replication (repeated scoring) improves score stability but does not improve validity or align the AI grader with human standards.
Calibration against a human-anchored sample is essential for absolute decisions, such as passing/failing, to mitigate unfair classification rates.

Executive Summary (TL;DR)

In plain words In the paper's terms

Large language models used to grade essays exhibit significant, hidden differences in how harshly they score student work. Common methods for evaluating these AI graders only check for general agreement, missing the fact that these models are unstable instruments that change their scoring standards whenever they are updated. Educators must use calibration and monitoring instead of just relying on simple agreement scores when using AI for grading.

This study conducts a pre-registered rater-effects audit of 12 LLMs acting as essay graders across two public educational corpora. It reveals that judge severity is idiosyncratic and large (a 219-point range on a 1000-point scale), effectively invisible to standard agreement metrics. Furthermore, model version updates function as significant instrument changes that necessitate re-calibration of absolute scores.

Authors' abstract (original)

Large language models (LLMs) are increasingly used as essay graders in learning analytics, evaluated almost exclusively with agreement statistics. Educational measurement warns that raters also differ in severity, show halo, and drift as instruments. We treat LLM judges as raters and run a pre-registered rater-effects battery (many-facet Rasch severity, residual halo, generalizability/decision studies, cross-version shifts, differential functioning) on public corpora in two languages (ENEM/Essay-BR; ASAP): 2,377 essays, 12 judges, 4 providers, 5 version contrasts, replicated cells, released as a score tensor. Judge severity spans 219 points on ENEM's 0-1000 scale; on ASAP the panel spread is 15-33% of the score range against a between-trained-human gap near 1%. Judge-human correlations sit in an undiscriminating .47-.56 band. All five version contrasts shift severity beyond a family-wise permutation null (up to 133 points), and one judge was deprecated mid-study, caught by identity canaries. Two pre-registered tests returned honest nulls: severity-adjusted leaderboard reversals did not survive a permutation null, and "silent drift" was refuted: agreement moved with severity in four of five contrasts. Replication yields self-consistency (phi>=.80 at k<=2) but not human-level accuracy, and a same-instrument check overturned our own halo comparison: matched on instrument and calibration, we find no credible evidence that judge halo exceeds the trained-human range.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

9.5/10

Engineering Countermeasures

9.0/10

Consensus Challenge

8.0/10

Topic Relevance

10.0/10

Relevance for Educators & Research

Provides a methodology for auditing automated grading systems using classical psychometrics (MFRM).
Establishes a template for pre-registration and canary-based monitoring of AI agents in high-stakes assessment.
Highlights the unreliability of commercial model versioning in longitudinal educational data.

Methodology and Dataset

Crossed, replicated rater-effects audit on 2,377 essays (977 ENEM, 1,400 ASAP). 12 judge models across 4 providers (Anthropic, Amazon, Meta, Qwen). The study employed many-facet Rasch measurement, G-theory, and permutation tests for version drift.

Why Worth Reading

It is the first pre-registered, psychometrically rigorous audit of LLMs-as-graders using a crossed rater-effects framework, moving beyond superficial agreement metrics to address the validity of absolute scores.

Key Claims & Verified Quotes

No verified quotes available for this paper.

Limitations & Study Design Flags

Language, rubric structure, and human-rater regimes are confounded in the cross-corpus comparison, limiting generalizability.
Contamination probes (continuation-based) are weak detectors and do not definitively prove the absence of training data memorization.
Findings are limited to the specific commercial hosted API endpoints and mutable model aliases tested.

Model inference (gaps & contradictions)

[metric / conceptual]: The paper demonstrates that agreement metrics (QWK/Pearson) are systematically blind to the largest rater effect (severity).

#psychometrics #automated-essay-scoring #rater-effects #validity #LLM-judge