Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals
Yupei Wang, Renfen Hu, Zhe Zhao · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (2024) · doi:10.18653/v1/2024.findings-emnlp.520 · gemini-3.6-flash
TL;DR
Standard automated essay grading tools often give accurate scores for the wrong reasons, such as ignoring essay organization entirely. By altering specific features of student essays like grammar, vocabulary, or paragraph order, researchers tested what AI models actually look at when grading. They found that fine-tuned Large Language Models evaluate essays much more like human teachers than traditional transformer models do.
Method and sample: Evaluated automated essay scoring models on two datasets: TOEFL11 (N = 12,100 essays; 9,900 train, 1,100 val, 1,100 test across 11 native language backgrounds) and ELLIPSE (N = 6,482 essays from 8th-12th grade English learners; 2,568 test). Models tested include fine-tuned BERT, RoBERTa, and DeBERTa, as well as zero-shot, few-shot, and supervised fine-tuned GPT-3.5 Turbo, GPT-4 Turbo, Llama-3-8B, and Llama-3-70B. Intervention methods combined rule-based edits (nlpaug word swapping, sentence shuffling) and GPT-4 Turbo text modifications across 8 counterfactual conditions.
#Automated Essay Scoring#LLMs as Raters#Construct Validity#Counterfactual Interventions#Rationale Alignment#Item Scoring Sensitivity