Automated L2 Proficiency Scoring: Weak Supervision, Large Language Models, and Statistical Guarantees
Authors: Aitor Arronte Alvarez, Naiyi Xie Fincham
Venue / Source: Workshop on Innovative Use of NLP for Building Educational Applications (2025)
Prediction-Powered Inference enables semi-supervised scoring models to safely expand assessment sample size by 157% with guaranteed statistical validity.
Key takeaways
- Naively relying on LLM outputs as ground-truth scores invalidates statistical inference because models assign high confidence to erroneous ratings.
- Applying Prediction-Powered Inference to semi-supervised machine learning models expands usable assessment data by 157% with 97% empirical coverage.
- Combining small gold-standard human datasets with Prediction-Powered Inference yields up to a 23.2% accuracy boost in automated L2 vocabulary scoring.
- Gradient boosting models trained on extracted linguistic features outperform zero-shot LLMs when integrated into debiased statistical protocols.
Executive Summary (TL;DR)
Authors' abstract (original)
Weakly supervised learning (WSL) is a machine learning approach used when labeled data is scarce or expensive to obtain.In such scenarios, models are trained using weaker supervision sources instead of human-annotated data.However, these sources are often noisy and may introduce unquantified biases during training.This issue is particularly pronounced in automated scoring (AS) of second language (L2) learner output, where high variability and limited generalizability pose significant challenges.In this paper, we investigate the analytical scoring of L2 learner responses under weak and semi-supervised learning conditions, leveraging Prediction-Powered Inference (PPI) to provide statistical guarantees on score validity.We compare two approaches: (1) synthetic scoring using large language models (LLMs), and (2) a semi-supervised setting in which a machine learning model, trained on a small gold-standard set, generates predictions for a larger unlabeled corpus.In both cases, PPI is applied to construct valid confidence intervals for assessing the reliability of the predicted scores.Our analysis, based on a dataset of L2 learner conversations with an AI agent, shows that PPI is highly informative for evaluating the quality of weakly annotated data.Moreover, we demonstrate that PPI can increase the effective sample size by over 150% relative to the original human-scored subset, enabling more robust inference in educational assessment settings where labeled data is scarce.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a formal statistical framework (Prediction-Powered Inference) for debiasing automated scoring models when evaluating student dialogue or text under AI-assisted environments.
- Offers empirical evidence against treating raw LLM evaluations as valid assessment scores due to severe rater overconfidence and poor uncertainty calibration.
- Demonstrates how semi-supervised learning combined with small human-annotated anchor datasets can expand assessment sample size by over 150% without compromising Messick/Kane validity arguments.
Methodology and Dataset
Sample: 1,721 text-based conversational sessions generated by 121 intermediate-level (CEFR B1-B2) English language learners interacting with an AI agent over 3 months (average 8.9 turns per session). Gold-standard ground truth consisted of 590 human-scored sessions (445 train, 145 evaluation) collaboratively annotated by expert raters on CEFR vocabulary range (1-3) and grammatical accuracy (1-4) rubrics. Models evaluated: XGBoost and LightGBM trained on 9 extracted lexical/syntactical features, compared against zero-shot LLM baselines (GPT-4o and GPT-4o-mini). debiasing protocol applied Prediction-Powered Inference (PPI) under weakly supervised (WSL) and semi-supervised setups.
Why Worth Reading
This paper addresses a fundamental validity challenge in automated educational scoring: how to scale assessment using AI or machine learning without introducing unquantified bias or invalidating score inference. By introducing Prediction-Powered Inference (PPI) to L2 assessment, it demonstrates a practical, mathematically grounded framework that allows educators and researchers to safely augment small human-graded datasets with unlabeled or AI-generated data while maintaining rigorous statistical guarantees on score reliability.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Single detector: not mentioned in the text
- No adversarial testing: not mentioned in the text
- Assumes human-scored gold-standard data are independently and identically distributed from a normal distribution.
- Evaluated on a single low-resource cohort of 121 intermediate English language learners across three sessions of one undergraduate course.
- Relies on feature engineering of tabular syntactic/lexical metrics rather than end-to-end deep neural text representations.
- supports GPT-4 produces holistic scores that correlate highly with human evaluations in L2 essay scoring. «In L2 assessment in particular, GPT-4 has shown to produce holistic scores that are highly correlated to human evaluation in written essays and have moderate to high inter-rater reliability (Tate et al., 2024).», p. 2, para 2
- supports LLMs exhibit poor calibration and overconfidence regarding the uncertainty of their own outputs. «Nonetheless, LLMs exhibit overconfidence in their incorrect assessments, indicating a poor understanding of uncertainty, a concern also noted in a recent work (Pawitan and Holmes, 2025).», p. 8, para 4
- [method / methodological gap]: Standard weak supervision frameworks in educational NLP treat LLM labels as proxy ground truth, whereas this paper demonstrates that uncorrected LLM labels violate coverage guarantees due to systemic verbalized overconfidence.
- [sample / sample constraint]: While general machine learning studies assume large gold-standard validation sets, educational assessment contexts typically suffer from extreme human label scarcity, making bounded semi-supervised sampling necessary.