Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
Authors: Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra
Venue / Source: arXiv (2026)
Rubric content, not the model grading it, dictates scoring strictness, and automated self-review secretly degrades evaluation standards.
Key takeaways
- Scoring strictness is driven almost entirely by rubric checklist items rather than judge model choice or rendering format.
- Unsupervised LLM self-review causes severe rubric drift by imposing unprompted requirements and removing broad coverage criteria.
- Under all-must-pass grading, each check's Shapley attribution has an exact closed-form solution computable in linear time.
- Gating revisions by pairing Shapley false-fail blame checks with provenance freezes protects grading validity across models.
Executive Summary (TL;DR)
Authors' abstract (original)
Checklist rubrics applied by large language model (LLM) judges have become a standard instrument for evaluating open-ended outputs and for rewarding them during training. Two assumptions surround this practice: that leniency or strictness is a property of the judge model, and that a rubric can be improved by asking a model to review it. We test both in a controlled grading task with planted flaws and on two human-labeled datasets, SummEdits and RAGTruth. Holding the judge and the outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing the judge model or the rendering of the same checks moves it by a few points. Model self-review then drifts the rubric toward the reviser's own standard: it adds requirements the task never stated and deletes checks that human labels had taught, with three reviser models and on all three datasets. To measure and control this, we show that under all-must-pass aggregation the Shapley value of each check has an exact closed form, which yields per-check catch credit, false-fail blame and unique catches in linear time. Attribution shows that self-review deletes the checks that do unique work and keeps the checks that cause false fails. A change-control gate that combines these quantities with a provenance freeze on label-derived checks holds nearly every drifting revision chain at its starting quality, with identical decisions under three judge models.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly models automated rater drift and construct representation under generative AI scoring rubrics.
- Provides a formal, closed-form Shapley decomposition for checklist scoring rubrics to evaluate rater reliability and error attribution.
- Warns assessment designers against using unsupervised AI self-refinement loops to generate or polish scoring rubrics.
Methodology and Dataset
Evaluated across three testbeds: Recon-72 (72 reports: 24 correct, 48 flawed across 12 synthetic cloud environments) and Recon-108 (108 reports: 36 correct, 72 flawed including unplanned misses); SummEdits (146 summaries from 38 conversations; 73 consistent); and RAGTruth (160 LLM responses across 40 sources; 80 faithful). Evaluator and reviser models: Claude 3.5 Haiku, Sonnet, and Opus (Sonnet default judge, Opus default reviser). Baselines included free-form verdicts, holistic paragraph standards, and generic 1-5 scales.
Why Worth Reading
For researchers and psychometricians deploying LLMs as automated raters or item evaluators, this paper overturns the common assumption that judge model strength or prompt harshness controls rater leniency. It establishes that checklist rubrics fundamentally encode the construct standard, provides an exact linear-time Shapley formulation to trace rater error to individual criteria, and offers actionable engineering gates to stop automated rubric drift.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Single detector: not mentioned in the text
- All judge, author, and reviser LLMs belonged to a single model family (Claude), though judged texts in RAGTruth came from external LLMs.
- Sample sizes in specific experimental subsets were small (24 to 36 correct reports and 8 to 12 unplanned misses per comparison).
- Gating evaluations assumed local independence among checklist items when replaying answer matrices from cache.
- Text similarity matching during revision gating occasionally misclassified rewritten checks as brand new additions.
- supports Rubric text alone predicts judge labels on HealthBench and ResearchRubrics. «Bagaria et al. [2] report that rubric text alone predicts judge labels on HealthBench and ResearchRubrics, consistent with the rubric carrying the standard.», p. 2, para 13
- supports Criterion-preserving rubric edits can substantially shift LLM judge preferences across target domains. «Ding et al. [6] show that criterion-preserving rubric edits that pass standard validation can shift a judge’s preferences on target domains by up to 27.9% and that the shift propagates through alignment», p. 3, para 6
- supports LLM self-review struggles without an external ground-truth reference signal. «That self-review fails without an external signal echoes results on self-correction in reasoning [13] and contrasts with the gains reported for self-refinement of outputs [19].», p. 3, para 6
- [metric / metric]: Standard benchmark accuracy on imbalanced defect datasets disguises severe rubric dysfunction; reporting balanced accuracy or separating correct pass from flawed caught is essential.
- [method / methodological]: LLM self-correction loops fail to maintain construct validity in evaluation settings because models generate test cases that share their own systematic biases.