Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim · arXiv (2026) · doi:10.48550/arxiv.2602.00521 · gemini-3.7-flash
TL;DR
This study tests whether AI models can act as reliable graders by applying established psychological testing models to automated grading prompts. The researchers found that no single AI model grades consistently across all evaluation criteria, and minor prompt changes can alter grades substantially. While larger models grade text more reliably when given clear rubrics, AI image evaluators often measure completely different traits than human graders.
Method and sample: Psychometric evaluation using Item Response Theory Graded Response Model (GRM; 2PL for binary items) fitted via PyMC with NUTS sampler (4 chains, 1,000 warmup, 1,000 sampling iterations). Evaluation used 7 models (Gemini 2.5 Flash, GPT-4o, GPT-4o-mini, Qwen3-30B, Qwen3-235B, LLaMA-4-Maverick, LLaMA-4-Scout) evaluated across NLP benchmarks (SummEval, TopicalChat, HelpSteer-2) and vision benchmarks (ImageHub/VIEScore across CIG, MIE, and TIE subsets) under three prompt perturbations (typo, newline, paraphrase) against human annotation baselines.
#Item Response Theory#LLM-as-a-Judge#Graded Response Model#Automated Scoring#Rater Reliability#Construct Validity#Prompt Robustness