psychometric_modeling

Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals

Authors: Yupei Wang, Renfen Hu, Zhe Zhao
Venue / Source: Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (2024)

High scoring agreement masks AES flaws: BERT models ignore essay organization, whereas fine-tuned LLMs align with human rubric rationale.

Key takeaways

High score agreement with human raters does not guarantee an automated essay scoring model uses valid rubrics.
BERT-like models ignore paragraph and essay organization perturbations, remaining insensitive to logical flow and text coherence.
Zero-shot LLMs exhibit strict grading biases but show broader alignment across conventions, complexity, and organization rubrics.
Few-shot prompting and supervised fine-tuning with 200 to 400 essays elevate LLM score agreement to match BERT models while preserving rationale alignment.
LLM-generated diagnostic feedback explicitly reflects counterfactual essay interventions, enhancing evaluation transparency.

Executive Summary (TL;DR)

In plain words In the paper's terms

Standard automated essay grading tools often give accurate scores for the wrong reasons, such as ignoring essay organization entirely. By altering specific features of student essays like grammar, vocabulary, or paragraph order, researchers tested what AI models actually look at when grading. They found that fine-tuned Large Language Models evaluate essays much more like human teachers than traditional transformer models do.

Automated essay scoring (AES) models often achieve high agreement with human raters, but standard correlation metrics like Quadratic Weighted Kappa mask whether models evaluate genuine writing constructs. Using linguistically informed counterfactual interventions generated via rules and LLMs, this study diagnoses the decision-making rationale of BERT-like models and LLMs across essay conventions, complexity, and organization. Results show BERT-like models ignore essay organization and coherence, whereas fine-tuned LLMs demonstrate holistic rationale alignment alongside high scoring agreement.

Authors' abstract (original)

While current Automated Essay Scoring (AES) methods demonstrate high scoring agreement with human raters, their decision-making mechanisms are not fully understood.Our proposed method, using counterfactual intervention assisted by Large Language Models (LLMs), reveals that BERT-like models primarily focus on sentence-level features, whereas LLMs such as GPT-3.5,GPT-4 and Llama-3 are sensitive to conventions & accuracy, language complexity, and organization, indicating a more comprehensive rationale alignment with scoring rubrics.Moreover, LLMs can discern counterfactual interventions when giving feedback on essays.Our approach improves understanding of neural AES methods and can also apply to other domains seeking transparency in model-driven decisions 1 .

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

7.5/10

Engineering Countermeasures

7.0/10

Consensus Challenge

7.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Demonstrates how to evaluate the validity argument (Kane/Messick framework) for LLM-based raters using counterfactual experimental interventions.
Provides a methodology for testing whether automated scoring systems rely on construct-relevant dimensions versus superficial linguistic cues.
Offers operational recipes for fine-tuning open and closed LLMs to achieve human-level agreement while maintaining interpretability.

Methodology and Dataset

Evaluated automated essay scoring models on two datasets: TOEFL11 (N = 12,100 essays; 9,900 train, 1,100 val, 1,100 test across 11 native language backgrounds) and ELLIPSE (N = 6,482 essays from 8th-12th grade English learners; 2,568 test). Models tested include fine-tuned BERT, RoBERTa, and DeBERTa, as well as zero-shot, few-shot, and supervised fine-tuned GPT-3.5 Turbo, GPT-4 Turbo, Llama-3-8B, and Llama-3-70B. Intervention methods combined rule-based edits (nlpaug word swapping, sentence shuffling) and GPT-4 Turbo text modifications across 8 counterfactual conditions.

Why Worth Reading

This paper addresses a fundamental validity concern in automated writing assessment: whether AI raters grade based on legitimate educational constructs or construct-irrelevant surface patterns. By introducing a model-agnostic counterfactual framework, it proves that conventional transformer models can be right for the wrong reasons (e.g., completely ignoring essay organization), while fine-tuned LLMs achieve both high agreement and rubric-aligned reasoning.

Key Claims & Verified Quotes

Large Language Models display superior alignment with human scoring rubrics compared to BERT-like models and reflect counterfactual perturbations in feedback.

Quote: “LLMs, although have lower score agreement than traditional models, display a superior alignment with human experts' reasoning during scoring and can also address counterfactual interventions in their feedback.” — 10.18653/v1/2024.findings-emnlp.520, p. 2, para. 16 (exact)

Supervised fine-tuning or few-shot learning allows LLMs to achieve both high score agreement and strong rationale alignment.

Quote: “Through few-shot learning or fine-tuning, LLMs can achieve both high scoring agreement and rationale alignment.” — 10.18653/v1/2024.findings-emnlp.520, p. 2, para. 16 (exact)

Limitations & Study Design Flags

Small sample: not found
Unrepresentative sample: not found
Single detector: not mentioned in the text
No adversarial testing: not found
Monolingual: found
“To identify the target concepts for AES scenarios, we reviewed scoring rubrics from major standardized English tests” (p. 3)

Content relevance and task requirement fulfillment were excluded from counterfactual interventions because content cannot be altered without changing linguistic features.
Evaluations were restricted to English language learner corpora (TOEFL11 and ELLIPSE), limiting cross-lingual generalizability.
Evaluation of textual feedback was conducted on a subsample of 200 ELLIPSE essays rather than the full test set.

Model inference (gaps & contradictions)

[metric / conceptual gap]: Standard agreement metrics (QWK) obscure structural invalidity in automated essay scoring; models with high QWK can completely ignore construct-relevant features like text organization.

#Automated Essay Scoring #LLMs as Raters #Construct Validity #Counterfactual Interventions #Rationale Alignment #Item Scoring Sensitivity