Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution
Authors: Giulio Corsi, Maryam Abo-Tabik, Yael Benn, Georgiana Thorpe Apreutesei, Roni Tibon, Alexandru Marcoci, Deborah Talmi
Venue / Source: arXiv (2026)
Frontier AI essay graders share systematic errors, compressing grade distributions and rewarding sentence complexity that human university markers completely ignore.
Key takeaways
- AI marking agreement fails to generalize across institutions, with quadratic weighted kappa dropping from 0.616 at Cambridge to 0.345 at Manchester Metropolitan.
- All tested models systematically compress grades towards a central attractor, overvaluing failing work and undervaluing first-class assignments by more than 11 points.
- Frontier models from different providers share correlated errors, meaning multi-model panels and ensembling fail to remove evaluative bias.
- Machine grades depend roughly three times more on surface linguistic features than human marks, specifically rewarding sentence complexity that human assessors ignore.
- Institutional validation of automated scoring cannot be borrowed from other universities and must be re-established locally whenever underlying models update.
Executive Summary (TL;DR)
Authors' abstract (original)
Large language models (LLMs) promise to automate one of higher education's most resource-intensive activities: assessing student writing. Existing evaluations are typically limited to a single model, a single prompt and a single institution. Using 761 undergraduate psychology assignments from three UK universities, we evaluated three frontier LLMs, individually and in ensemble, on accuracy, prompt sensitivity, error independence, generalisability and evidential basis. Agreement between human and machine marks varied widely, with no model or prompt optimal across institutions. All models compressed marks towards a common central tendency, overvaluing the weakest assignments and undervaluing the strongest. Because this error was shared, ensembling failed to remove it. Machine marks also rewarded linguistic features more strongly than human marks. Since human-machine agreement was a joint product of model, prompt and institution, we conclude that current LLMs lack evaluative capacity transferable across contexts, and that single-context validation overstates readiness for deployment.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Demonstrates severe threats to Kane's validity argument when employing LLMs as raters, particularly regarding the evaluation and generalization inferences.
- Provides a quantitative template (Kitagawa decomposition, partial correlation of errors, PCA feature regression) for auditing rater drift and construct-irrelevant variance in AI-assisted grading.
- Shows that AI tools reward surface linguistic markers, incentivizing students using generative AI to produce overly complex, stylized text to game automated scoring.
Methodology and Dataset
Corpus of 761 summative undergraduate psychology assignments (Cambridge n=133, Manchester Metropolitan n=456, Nottingham n=172) across 50 modules and 87 distinct prompts, marked on UK 0-100 percentage scales. Evaluated zero-shot GPT-5.4, Claude Opus 4.6, and Gemini 3 Flash under a 3x3x3 factorial prompt design (criterion specificity, calibration base rates, scoring strategy) split into a 20% calibration set (n=153) and 80% held-out test set (n=608). Baselines included four ensemble aggregation strategies, linear/affine calibrations, and documented human inter-marker reliability benchmarks (r = 0.47 to 0.93).
Why Worth Reading
This paper provides one of the most methodologically rigorous warnings against the plug-and-play adoption of LLMs as raters in higher education. By combining multi-site authentic data, factorial prompt experimentation, error correlation analysis, and linguistic PCA decomposition, the authors expose fundamental threats to construct validity: algorithmic monoculture, grade compression, and susceptibility to stylistic features.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Single detector: not found
- The corpus is limited to undergraduate psychology assignments within three UK institutions, which may not generalize to STEM, humanities, or non-UK grading scales.
- Assignments carry single moderated institutional marks rather than independent double marks, preventing within-corpus measurement of human-human inter-rater reliability.
- Evaluations were restricted to zero-shot prompting; few-shot exemplar calibration or domain-specific fine-tuning was not assessed.
- Analyses reflect proprietary frontier model snapshots from March 2026 subject to silent vendor updates.
- supports Machine marks cluster towards the middle of the scale and treat extremes unevenly. «machine marks have clustered towards the middle of the scale (Lundgren, 2024) and, on undergraduate psychology essays, were more lenient than instructors' for weaker work and harsher for stronger work (Wetzler et al., 2025).», p. 2, para 5
- supports Automated scoring systems reward surface linguistic and lexical features rather than core assessed constructs. «The evidential basis of machine marks has likewise received little direct attention, despite a construct-validity literature predating LLMs, which established that automated systems could match human marks while rewarding essay length and lexical sophistication», p. 3, para 3
- [method / contradiction]: Standard LLM-as-a-judge literature assumes ensembling independent LLMs averages out idiosyncratic variance, but the empirical error partial correlation of r = 0.74-0.78 refutes this independence assumption for authentic student essay grading.
- [definition / methodological gap]: Previous automated essay scoring validations relying on single-institution benchmarks confounded cohort grade distributions with rater accuracy, masking severe within-band degradation across contexts.