assessment_design

A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

Authors: Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong
Venue / Source: arXiv (2026)

Standard judge agreement metrics are insufficient because they fail to distinguish between reliable invariance and a fundamental blindness to the construct being measured.

Key takeaways

Reliability (invariance) and validity (sensitivity) are independent dimensions; a stable judge can be entirely invalid.
Judges exhibit a significant sensitivity asymmetry, detecting claim scope expansion more easily than increases in claim commitment strength.
High agreement between judges and existing label sets is often an artifact of provenance leakage rather than evidence of construct understanding.
Validation power, not raw agreement, should be the primary metric for evaluating assessment-related benchmarks.
Assessment designers should use human-verified construct-changing edits rather than assuming benchmark labels are ground truth.

Executive Summary (TL;DR)

In plain words In the paper's terms

Researchers have created a new way to test if AI 'judges' actually understand the content they are evaluating or are just reacting to superficial features. They found that these AI judges are often good at ignoring unimportant changes, like font or length, but they frequently miss critical changes in the strength or accuracy of the arguments. This means high consistency scores do not guarantee that an AI judge is actually valid or accurate.

This paper establishes a two-dimensional construct validity profile for LLM-as-a-judge evaluation, defined by invariance to surface edits and sensitivity to construct-relevant edits. It proves that these coordinates are independent and that high invariance can mask complete failure to detect construct changes. The empirical audit of seven judge families reveals a systematic sensitivity gap where judges detect scope expansion but largely fail to register increased commitment strength.

Authors' abstract (original)

LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile: invariance S, the probability that a verdict is unchanged under construct-preserving edits, and construct sensitivity R, the probability that it changes under minimal construct-changing edits. We show that S and R are independent and that no scalar summary preserves all relevant comparisons. We measure the profile across 7 judges and 4 domains using 7 construct-changing intervention types and 5 register-only controls, with intervention direction determined by human annotators and generation, verification, and judging assigned to disjoint model families. At matched invariance S >= 0.90, judges average S = 0.945 but R = 0.319. Sensitivity also differs between scope and strength edits: R_scope = 0.383 versus R_strength = 0.262, a +0.121 gap with the same sign for all 7 judges. We further audit five public label sets and find that surface-only predictors reproduce 55%-67% of labels in paired mode, including 67.4% of MT-Bench human votes. These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated, motivating joint reporting of invariance and sensitivity and auditing the validation set itself.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

8.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a methodology for quantifying assessment validity in LLM-mediated contexts.
Offers a framework (S, R) for evaluating the fairness and sensitivity of automated assessment tools.
Highlights the risks of using provenance-tainted datasets for validating AI raters.

Methodology and Dataset

The study tested 7 distinct LLM judge families across 4 domains. It used 478 human-annotated items (288 construct-changing, 190 construct-preserving) involving 12 intervention types. Human annotation was conducted with 3 independent raters to ensure a high agreement bar (0.75+). Baselines included 5 public label sets (RewardBench, RM-Bench, HelpSteer2, MT-Bench).

Why Worth Reading

This paper provides a rigorous measurement-theoretic framework that forces the field to move beyond simplistic 'judge agreement' metrics. It introduces a practical harness for assessment designers to validate whether their AI-assisted grading systems are actually sensitive to the construct they claim to measure.

Key Claims & Verified Quotes

No verified quotes available for this paper.

Limitations & Study Design Flags

The scope/strength decomposition is not exhaustive of all ways a scientific claim can overreach its evidence.
The study focuses on scientific claim specificity and may not generalize perfectly to other assessment domains.
The per-axis form of the validation power prediction remains untested on existing public sets.

From the article text (quote verified)

contradicts The unification of scope and strength into one dimension is a theoretical assumption that is refuted by the current findings. «We treat the unification as a hypothesis with a stated refutation condition, and it does not survive.», p. 1, para 9

Model inference (gaps & contradictions)

[Method / Measurement]: Standard benchmarks conflate absolute and relative properties in their labels, leading to misinterpreted validation power.

#AI evaluation #construct validity #psychometrics #LLM-as-a-judge #assessment design #automated assessment