Automated L2 Proficiency Scoring: Weak Supervision, Large Language Models, and Statistical Guarantees
Aitor Arronte Alvarez, Naiyi Xie Fincham · Workshop on Innovative Use of NLP for Building Educational Applications (2025) · doi:10.18653/v1/2025.bea-1.30 · gemini-3.6-flash
TL;DR
Scoring language learners automatically often relies on small human-graded datasets or unreliable AI ratings. This study introduces a statistical framework that uses a small set of expert-graded student responses to safely correct and expand predictions across larger unannotated practice sessions. The resulting method expands usable assessment data by over 150% and improves scoring accuracy without compromising statistical validity.
Method and sample: Sample: 1,721 text-based conversational sessions generated by 121 intermediate-level (CEFR B1-B2) English language learners interacting with an AI agent over 3 months (average 8.9 turns per session). Gold-standard ground truth consisted of 590 human-scored sessions (445 train, 145 evaluation) collaboratively annotated by expert raters on CEFR vocabulary range (1-3) and grammatical accuracy (1-4) rubrics. Models evaluated: XGBoost and LightGBM trained on 9 extracted lexical/syntactical features, compared against zero-shot LLM baselines (GPT-4o and GPT-4o-mini). debiasing protocol applied Prediction-Powered Inference (PPI) under weakly supervised (WSL) and semi-supervised setups.
#Automated Scoring#L2 Proficiency#Prediction-Powered Inference#Weak Supervision#Large Language Models#Assessment Validity#Semi-Supervised Learning#CEFR