Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
Saeid Asgari, Rui Ying, Sara Malvar, Rakshanda Agarwal, Bruce Sun, Claire Wang, Leonardo de Oliveira Nunes, Ranveer Chandra · arXiv (2026) · doi:10.48550/arxiv.2610.24971 · gemini-3.8-flash
TL;DR
Automated grading by AI depends heavily on the specific wording and rules in the rubric rather than the AI model doing the scoring. Letting an AI edit its own rubric actually makes grading worse by deleting critical criteria and adding unneeded demands. Combining game-theoretic scoring checks with strict revision controls prevents grading standards from silently drifting.
Method and sample: Evaluated across three testbeds: Recon-72 (72 reports: 24 correct, 48 flawed across 12 synthetic cloud environments) and Recon-108 (108 reports: 36 correct, 72 flawed including unplanned misses); SummEdits (146 summaries from 38 conversations; 73 consistent); and RAGTruth (160 LLM responses across 40 sources; 80 faithful). Evaluator and reviser models: Claude 3.5 Haiku, Sonnet, and Opus (Sonnet default judge, Opus default reviser). Baselines included free-form verdicts, holistic paragraph standards, and generic 1-5 scales.
#LLM-as-a-judge#automated assessment#rubric drift#Shapley attribution#construct validity#rater drift#checklist grading