Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards
Authors: Xuefeng Jin, Jiashuo Zhang, Teng Cao, Bin Yang
Venue / Source: arXiv (2026)
This paper presents a reinforcement learning framework that solves the feedback degradation problem by gating rubric-based rewards, ensuring AI-generated feedback remains high-quality while maintaining accurate automated scoring.
Key takeaways
- Optimizing AI essay scoring through standard reinforcement learning alone can degrade the quality of generated feedback.
- The RFE framework enables automated, fine-grained evaluation of essay feedback by converting rubric criteria into scalar reward signals.
- Adaptive Gated Feedback Optimization (AGFO) significantly reduces the computational costs of using an LLM-as-judge without sacrificing scoring or feedback performance.
- Adjacent Contrastive Reasoning (ACR) improves model sensitivity to fine-grained quality differences by forcing the model to distinguish between adjacent score boundaries.
- Joint training across different writing prompts preserves scoring accuracy while simplifying deployment and maintenance.
Executive Summary (TL;DR)
Authors' abstract (original)
Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG). However, existing studies rely primarily on prompt engineering or supervised fine-tuning, while systematic research on reinforcement learning (RL) post-training and automated evaluation of feedback quality remains limited. We propose RLAES, a unified LLM framework that jointly optimizes essay scoring and feedback generation through RL. To make feedback quality measurable, interpretable, and usable for training, we introduce Rubric-based Feedback Evaluation (RFE), an essay-grounded feedback evaluation framework comprising 166 fine-grained binary rubric items and an LLM-as-judge. Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO), which activates rubric-based feedback rewards on demand during RL, reducing evaluation overhead while improving feedback quality. We also propose Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration by explicitly contrasting adjacent score levels. Experimental results show that the RFE framework captures essay-feedback consistency, exhibits strong pairwise discriminative power, and closely aligns with expert preferences. On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance among LLM-based methods (QWK = 0.803), while maintaining feedback quality comparable to GPT-5.5 and avoiding the feedback degradation observed under score-only RL. Code and datasets are publicly available at https://github.com/hellomuyi/RLAES.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a scalable framework for integrating pedagogical rubrics into automated assessment models.
- Offers a quantitative methodology to detect and mitigate 'feedback drift' in automated grading systems.
- Discusses the trade-offs between computational overhead and feedback quality in RL-based AI assessment.
Methodology and Dataset
The study uses the ASAP dataset (12,978 essays across 8 prompts). The model is based on Qwen3.5-9B, utilizing LoRA for fine-tuning. Baselines include GPT-3.5, GPT-4, Llama-3-8B-Instruct, and various SFT models. Evaluations employ GPT-5.5 as the reference judge for the RFE framework.
Why Worth Reading
It provides a rare, rigorous solution to the problem of 'reward hacking' in AI-based educational assessment, where models optimize for the grade at the expense of pedagogical feedback quality.
Key Claims & Verified Quotes
Limitations & Study Design Flags
-
Unrepresentative sample:
found
“Evaluation is also restricted to prompt-specific English essays from ASAP.” (p. 7)
-
No adversarial testing:
found
“RLAES remains limited by its dependence on LLM judges, whose differing preferences can cause reward mismatch” (p. 7)
-
Monolingual:
found
“Evaluation is also restricted to prompt-specific English essays from ASAP.” (p. 7)
- The framework relies on LLM-based judges, which may suffer from reward mismatch due to inconsistent preferences.
- Evaluation is restricted to English-language essays from the ASAP dataset, limiting immediate cross-lingual or cross-domain generalizability.
- Feedback evaluation incurs financial and computational costs when using large, high-performing LLM judges.
- [method / missing adversarial testing]: While the authors include 'perturbed' feedback (adjacent scores), they do not test the RFE framework against adversarial feedback designed specifically to mimic high-quality pedagogical markers.