Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
Authors: Delip Rao, Chris Callison-Burch
Venue / Source: arXiv (2026)
Unstated choices in handling skipped items can swing an LLM rubric grader's reported accuracy by thirty-five percentage points.
Key takeaways
- Five popular binary agreement metrics—Pearson, Spearman, Kendall, phi, and MCC—are mathematically identical, so reporting multiple coefficients offers zero corroborating evidence.
- Excluding abstentions inflates apparent accuracy and leaves full-set performance identified only within a sharp interval as wide as the omitted fraction.
- Macro-averaging rubric criteria versus micro-pooling decisions can shift chance-corrected agreement by a factor of four on identical grading output.
- Publishing full two-by-two confusion tables and marginal base rates is mandatory to diagnose whether an LLM judge is systematically lenient or strict.
Executive Summary (TL;DR)
Authors' abstract (original)
Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels. Yet the same verdicts can support wildly varying agreement numbers, depending on seemingly minor choices: the judgment scale, the retained cases, the handling of abstentions and invalid outputs, and the pooling of verdicts across items and rubric criteria. The statistics that settle these choices are established, but in psychometrics, econometrics, and corpus annotation rather than in the evaluation practice that needs them. We treat the choices as a measurement protocol that fixes what the reported number estimates before any metric is computed, assemble the relevant results into a single source-attributed analysis, and apply it to three published LLM-judge evaluations. For non-degenerate binary verdicts, Pearson's $r$, Spearman's $ρ$, Kendall's $τ_b$, the phi coefficient, and the Matthews correlation coefficient are exactly the same statistic, so reporting several repeats one number under different names. Cohen's $κ$ differs from them only through a marginal-mismatch factor in $(0,1]$ and shares their asymptotic variance when judge and human assign the positive verdict equally often. Under exclusion, accuracy over all cases is pinned down only to a worst-case interval as wide as the uncovered fraction. On a rubric benchmark carrying per-criterion human labels, protocol choice alone moves reported accuracy from $0.551$ to $0.899$ and carries $κ$ across zero, without altering a single verdict. We distill the analysis into a reporting checklist that makes agreement claims reconstructible and comparable.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly informs the psychometric validity of LLMs as raters and automated rubric graders in higher education assessment.
- Provides quantitative bounds and statistical corrections for rater drift, leniency bias, and selective abstention when substituting AI for human graders.
- Delivers a formal Kane-aligned measurement protocol checklist for defending automated assessment score interpretation.
Methodology and Dataset
Theoretical proof of metric equivalences, asymptotic delta-method derivations, Manski partial-identification bounds, and reanalysis of three published empirical datasets: (1) 2,238 paired human-GPT-4 self-preference comparisons from Wataoka et al. (2024); (2) 4,718 test pairs evaluating an abstaining judge cascade from Jung et al. (2025); and (3) 2,007 criterion decisions across 223 human-AI dialogues rated on 9 rubric criteria by humans and GPT-3.5-turbo-16k from Hashemi et al. (2024).
Why Worth Reading
It provides essential psychometric grounding for automated grading and LLM-as-judge systems, exposing how published agreement metrics can be manipulated by up to 35 percentage points through routine preprocessing decisions.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- No adversarial testing: not mentioned in the text
- Monolingual: not mentioned in the text
- The literature scan of 24 papers was descriptive, non-exhaustive, and coded by a single author without independent inter-coder reliability checks.
- Closed-form identification bounds are derived for accuracy, whereas bounds for other metrics require full combinatorial enumeration over latent contingency tables.
- The analysis does not evaluate construct validity beyond agreement with a single human reference.
- Cluster-resampling corrections are recommended for hierarchical criteria within items but were not formally modeled across multiple human rater pools.
- supports Under nominal disagreement on complete judge panels, Krippendorff's alpha and Fleiss's kappa differ only by a finite-sample correction factor. «That Do is the complement of Fleiss’ observed agreement, and that the expected disagreements differ only by this factor, are established by Artstein and Poesio (2008).», p. 10, para 32
- [metric / conceptual confusion]: Educational automated scoring literature frequently treats Cohen's kappa and Pearson/Spearman correlation as independent evidence of rater agreement, whereas for binary rubric criteria they share numerators and only differ by marginal mismatch.
- [method / omitted variable]: Studies benchmarking LLM automated graders against human professors routinely omit ungradable or abstained student submissions from reported accuracy without calculating Manski identification intervals.