LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora
Veerendra Kumar Sunkavalli · arXiv (2026) · doi:10.48550/arxiv.2608.29517 · gemini-3.1-flash-lite
TL;DR
Large language models used to grade essays exhibit significant, hidden differences in how harshly they score student work. Common methods for evaluating these AI graders only check for general agreement, missing the fact that these models are unstable instruments that change their scoring standards whenever they are updated. Educators must use calibration and monitoring instead of just relying on simple agreement scores when using AI for grading.
Method and sample: Crossed, replicated rater-effects audit on 2,377 essays (977 ENEM, 1,400 ASAP). 12 judge models across 4 providers (Anthropic, Amazon, Meta, Qwen). The study employed many-facet Rasch measurement, G-theory, and permutation tests for version drift.
#psychometrics#automated-essay-scoring#rater-effects#validity#LLM-judge