Beyond Response Scoring: An AI-aware Cognitive Supervision Framework for Automated Mathematical Short Answer Grading
Shengran Guo, Min Chi, Feifei Qiao, Varad Patwardhan · arXiv (2026) · doi:10.48550/arxiv.2610.41215 · gemini-3.6-flash
TL;DR
Traditional automated grading systems struggle to evaluate step-by-step math reasoning and can be fooled by AI-assisted student answers. This study introduces a framework that grounds automated grading in specific mathematical skills while using an auxiliary detector to down-weight AI-generated answers in skill mastery estimates. Evaluated on nearly 50,000 university student responses, the system dramatically improves grading precision while maintaining fair scoring for all students.
Method and sample: The dataset contains 49,402 instructor-graded mathematical short-answer responses from 3,861 university student-semester records across 19 semesters (Fall 2014 to Fall 2024). Evaluated models include AICG (RoBERTa multi-task encoder with G-DINA cognitive supervision and contrastive AI detector), BERT, RoBERTa regression, ASQ-DBN, zero-shot GPT-4o, and GradeOpt (LLM multi-agent rubric optimization). Evaluation uses a 10-year expanding-window temporal split.
#Automated Short Answer Grading#Cognitive Diagnosis Models#G-DINA#Generative AI Detection#Assessment Validity#Multi-Task Learning#Item Response Theory