RubricWorld: World Model-Guided Rubric Optimization for LLM Graders
Authors: Hongji Pu, Hang Li, Yucheng Chu, Jiliang Tang
Venue / Source: arXiv (2026)
Predicting specific grading corrections and degradations at ordinal boundaries optimizes LLM scoring rubrics far more effectively than standard validation accuracy.
Key takeaways
- Evaluating rubric edits by net validation accuracy conceals localized degradation where new grading errors offset corrections.
- A world model predicting paired correction and degradation rates at ordinal score boundaries allows screening candidate edits without expensive LLM re-scoring.
- Independent Clopper-Pearson statistical certification on held-out samples prevents over-fitting and controls error accumulation across optimization rounds.
- Conditioning edit predictions on the current rubric state is critical because early prompt additions alter how subsequent rules affect LLM scoring behavior.
Executive Summary (TL;DR)
Authors' abstract (original)
Rubric optimization improves LLM grading by revising scoring instructions while keeping the grader's parameters fixed. Existing optimizers often judge edits by aggregate validation performance, limiting the supervision they retain about which errors an edit corrects or introduces. Testing each candidate also requires rescoring validation answers, which limits how many edits can be explored within a fixed budget. We propose RubricWorld, a framework that learns from previous edit evaluations to guide further search. Its world model predicts an edit's correction and degradation rates at score boundaries, conditioned on the current rubric state. The training signal comes from paired grading results: scores for the same answers before and after an edit. Predictions select one candidate for direct evaluation, and held-out confidence bounds determine whether it corrects enough target errors while keeping new errors within prespecified limits. Measured effects update the model even when the edit is rejected. Across EIR (National), ASAP2.0, and ASAP-SAS with three frozen LLM graders, RubricWorld achieves the highest score in 20 of 27 metric comparisons. On ASAP2.0, it raises gpt-5-mini accuracy from 18.20% with the original rubric to 46.70%, exceeding the strongest baseline, APO, by 22.60 percentage points. code will be released upon acceptance.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Demonstrates theoretical and practical methods for auditing and optimizing LLMs as raters in educational assessment.
- Provides a formal quantitative methodology for modeling rater drift and score distortion across ordinal performance boundaries.
- Establishes statistical certification methods (Clopper-Pearson bounds) for ensuring rubric updates do not degrade assessment validity.
Methodology and Dataset
Evaluated on three essay and short-answer automated scoring datasets: EIR National (N=1,373 private teacher short-answer responses across 5 items, scores 0-3/0-4), ASAP2.0 (N=17,307 student essays from PERSUADE 2.0 corpus, scores 1-6), and ASAP-SAS Set 1 (N=1,673 short answer items, scores 0-3). Evaluated using three frozen LLM raters: Meta-Llama-3.1-8B-Instruct-AWQ-INT4, QwQ-32B-Preview, and OpenAI gpt-5-mini. Baselines included Zero-shot, Few-shot, APO, GradeOpt, and GEPA across 3 random seeds.
Why Worth Reading
This paper establishes a rigorous statistical and computational paradigm for automated rater calibration. Instead of fine-tuning LLM weights or relying on heuristic prompt optimization, RubricWorld provides a principled framework using ordinal regression boundaries, world models, and Clopper-Pearson confidence bounds to iteratively optimize scoring rubrics while preventing rater drift and degradation.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Requires sufficient empirical answer examples at each score boundary; rare score categories or small round-specific sample sizes prevent edit certification.
- Relies on human reference scores as ground truth, which may contain noise or human rater bias.
- Evaluation focus is limited to educational automated scoring tasks and may not transfer directly to open-domain text evaluation or safety alignment.
- [metric / conceptual divergence]: Standard prompt optimizers maximize aggregate metrics (QWK/ACC), which allows large localized boundary errors to hide behind net performance gains.