Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
Delip Rao, Chris Callison-Burch · arXiv (2026) · doi:10.48550/arxiv.2606.00093 · gemini-3.8-flash
TL;DR
When automated AI graders are tested against human educators, researchers often report agreement numbers that seem reassuring. However, unstated evaluation choices—such as how to handle skipped questions or pool criteria—can make the exact same AI appear either highly accurate or no better than chance. Standardizing reporting protocols and confusion tables is essential before LLM graders can be trusted in educational assessment.
Method and sample: Theoretical proof of metric equivalences, asymptotic delta-method derivations, Manski partial-identification bounds, and reanalysis of three published empirical datasets: (1) 2,238 paired human-GPT-4 self-preference comparisons from Wataoka et al. (2024); (2) 4,718 test pairs evaluating an abstaining judge cascade from Jung et al. (2025); and (3) 2,007 criterion decisions across 223 human-AI dialogues rated on 9 rubric criteria by humans and GPT-3.5-turbo-16k from Hashemi et al. (2024).
#LLM-as-judge#Automated Grading#Psychometrics#Inter-Rater Reliability#Measurement Validity#Rubrics#Manski Bounds