Beyond Response Scoring: An AI-aware Cognitive Supervision Framework for Automated Mathematical Short Answer Grading
Authors: Shengran Guo, Min Chi, Feifei Qiao, Varad Patwardhan
Venue / Source: arXiv (2026)
Jointly supervising automated math grading with psychometric skill models and soft AI detection significantly improves accuracy and resilience against GenAI assistance.
Key takeaways
- Integrating psychometric skill models with neural encoders drastically reduces grading error compared to score-only neural regression models.
- Soft reliability weighting of suspected AI-assisted text protects cognitive skill estimates without altering or penalizing response scores.
- Pretest and posttest mastery targets provide stronger supervision for automated short-answer grading than learning-gain targets alone.
- Generative AI detectors can serve as continuous confidence weights for psychometric modeling rather than binary misconduct filters.
Executive Summary (TL;DR)
Authors' abstract (original)
Transformer-based language models have achieved remarkable success in automated short answer grading (ASAG). However, current methods in ASAG struggle to comprehend, reason about and evaluate complex mathematical answers involving symbols and formulas. These score-only methods fail to generalize across diverse responses and skill states associated with identical scores. Moreover, they ignore the potential upward bias introduced by generative Artificial Intelligence (AI) usage in assessing students' performance and knowledge mastery. To address these limitations, we propose AICG (AI-aware Cognitive-supervised Grading), a two-branch multi-task framework. The first branch combines RoBERTa-based regression, ordinal, and classification heads for individual-answer grading. The second branch uses an expert-designed Q-matrix to associate responses with assessed skills and jointly predicts G-DINA-derived skill mastery and learning gain. Through the shared encoder, these skill-level targets provide cognitive supervision for answer grading beyond score labels. AICG further uses a Gen-AI detector, trained through supervised contrastive pretraining followed by linear classification, to estimate assistance probabilities and convert them into reliability weights, attenuating unreliable cognitive evidence during learning assessment. We conduct experiments on 49,402 instructor-graded responses spanning 19 academic semesters. Relative to RoBERTa regression, AICG reduces mean absolute error (MAE) by 50.3% and improves quadratic weighted kappa (QWK) from 0.798 to 0.927. Ablations identify Q-matrix-guided cognitive supervision as the primary source of improvement, while AI-aware weighting further reduces overall MAE by 4.2% over the unweighted cognitive model and consistently improves MAE and QWK across the four most recent post-Gen-AI semesters.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Demonstrates how generative AI tools distort psychometric inferences (skill mastery estimation) without necessarily changing response scores.
- Provides an engineering countermeasure to score validity degradation under unproctored/unsupervised conditions by applying continuous reliability weighting.
- Connects automated short-answer grading with formal item response and cognitive diagnosis modeling (G-DINA).
Methodology and Dataset
The dataset contains 49,402 instructor-graded mathematical short-answer responses from 3,861 university student-semester records across 19 semesters (Fall 2014 to Fall 2024). Evaluated models include AICG (RoBERTa multi-task encoder with G-DINA cognitive supervision and contrastive AI detector), BERT, RoBERTa regression, ASQ-DBN, zero-shot GPT-4o, and GradeOpt (LLM multi-agent rubric optimization). Evaluation uses a 10-year expanding-window temporal split.
Why Worth Reading
This paper presents a novel paradigm for automated assessment under generative AI: instead of using AI detectors to penalize students or reject submissions, it uses soft AI-detection probabilities to attenuate corrupted psychometric evidence while maintaining standard grading. It offers a mathematically grounded approach to combining neural language models with cognitive diagnosis models (G-DINA) for resilient educational measurement.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
-
Unrepresentative sample:
found
“Future work will learn Q-matrices from response text, improve detector robustness across LLMs, evaluate cross-course and multi-institution generalization, and use mastery and learning-gain profiles for personalized feedback.” (p. 7)
- No adversarial testing: not mentioned in the text
- Requires a human-expert Q-matrix mapping assessment items to underlying cognitive skills.
- Evaluated on a single university course domain (introductory probability principles).
- Synthetic AI-assisted data for detector training relied on masking and reconstruction using a single LLM family (DeepSeek).
- supports Current language models remain unreliable in domain-specific symbolic and compositional math reasoning. «Current language models remain unreliable in domain-specific symbolic and compositional reasoning (Latif and Zhai 2024; Jiang and Bosch 2024; Ferreira Mello et al. 2025) and may misinterpret valid intermediate or mathematically equivalent work.», p. 1, para 11
- [method / conceptual gap]: Demonstrates that LLMs used as direct zero-shot or rubric-optimized graders (GPT-4o, GradeOpt) underperform specialized fine-tuned multi-task Transformer models grounded in psychometric cognitive representations.