Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne · arXiv (2026) · doi:10.48550/arxiv.2610.02877 · gemini-3.1-flash-lite
TL;DR
AI judges often agree with human scores on average, but this study shows they don't necessarily struggle with the same specific assignments that humans do. By applying a psychometric tool, researchers found that AI models and human raters have different 'blind spots' that depend on the specific task, like checking for factual consistency versus judging flow. Identifying these differences helps educators design better systems where humans handle the hardest AI-evaluation cases.
Method and sample: The study analyzes 1,600 system summaries from the SummEval dataset, rated across four dimensions (coherence, consistency, fluency, relevance) by eight human annotators. These are compared against ratings from 17 open-weight LLMs (0.8B-14B parameters) including Llama 3, Qwen 3.5, Phi-4, and Mistral models. Psychometric analysis uses Many-Facet Rasch Models (MFRM).
#LLM-as-a-judge#psychometrics#Many-Facet Rasch Model#assessment validity#residual hardness#human-AI collaboration