From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring
Jodi M. Casabianca, Daniel F. McCaffrey, Matthew S. Johnson, Naim Alper, Vladimir Zubenko · arXiv (2026) · doi:10.48550/arxiv.2603.19280 · gemini-3.8-flash
TL;DR
Automated scoring systems using generative AI do not grade student essays in transparent or perfectly repeatable ways. When tested on over 13,000 student essays, popular AI models gave noticeably harsher scores than human graders and traditional automated engines. Educational institutions must demand extensive psychometric validity evidence before using generative AI to assign high-stakes test scores.
Method and sample: Empirical evaluation on N = 13,032 persuasive essays from the PERSUADE 2.0 corpus written by US grade 6–12 students across 8 prompts, evaluated on a split sample of N = 6,514. Scoring systems compared: double human ratings (SAT 1–6 rubric with third-rater adjudication), a 10-macrofeature e-rater linear regression model, GPT-4o (two runs, temp=0), Gemini 1.5 Pro (two runs, temp=0), and Gemini 2.5 Pro (one run).
#automated essay scoring#generative AI#validity framework#e-rater#rater drift#psychometrics