psychometric_modeling

From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring

Authors: Jodi M. Casabianca, Daniel F. McCaffrey, Matthew S. Johnson, Naim Alper, Vladimir Zubenko
Venue / Source: arXiv (2026)

Zero-shot generative language models underperform traditional feature-based engines in essay scoring agreement and exhibit non-deterministic drift even at zero temperature.

Key takeaways

Traditional feature-based engines showed higher agreement with human raters (QWK 0.84) than zero-shot LLMs (QWK 0.52 to 0.68).
Generative AI models consistently assigned lower average scores than human raters, introducing significant severity bias.
Setting model temperature to zero does not guarantee deterministic scoring across repeated evaluations of identical essays.
Validity arguments for generative scoring require novel evidence on pretraining corpora, non-determinism, and prompt injection gaming.

Executive Summary (TL;DR)

In plain words In the paper's terms

Automated scoring systems using generative AI do not grade student essays in transparent or perfectly repeatable ways. When tested on over 13,000 student essays, popular AI models gave noticeably harsher scores than human graders and traditional automated engines. Educational institutions must demand extensive psychometric validity evidence before using generative AI to assign high-stakes test scores.

Generative AI scoring lacks explicit ties to construct definitions, deterministic reproducibility, and transparent feature weights present in traditional automated scoring engines. The authors formulate a comprehensive validity framework across five Standard-aligned domains and test it using human ratings, e-rater macrofeatures, GPT-4o, and Gemini models. Empirical evaluation shows feature-based models achieve superior human agreement (QWK 0.84) compared to zero-shot LLMs (QWK 0.52–0.68), which exhibit substantial score severity drift.

Authors' abstract (original)

The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from a large corpus of independent argumentative essays written by 6-12th grade students demonstrate the collection of validity evidence for different types of scoring systems and highlight the numerous complexities and considerations when making a validity argument for these scores.

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

7.0/10

Consensus Challenge

7.5/10

Topic Relevance

8.5/10

Relevance for Educators & Research

Directly maps Messick and Kane validity arguments onto generative AI scoring engines.
Provides quantitative metrics (QWK, SMD, intra-rater drift) for assessing LLM rater reliability and systematic severity shift.
Highlights vulnerability of LLM raters to prompt-injection adversarial gaming in constructed response assessments.

Methodology and Dataset

Empirical evaluation on N = 13,032 persuasive essays from the PERSUADE 2.0 corpus written by US grade 6–12 students across 8 prompts, evaluated on a split sample of N = 6,514. Scoring systems compared: double human ratings (SAT 1–6 rubric with third-rater adjudication), a 10-macrofeature e-rater linear regression model, GPT-4o (two runs, temp=0), Gemini 1.5 Pro (two runs, temp=0), and Gemini 2.5 Pro (one run).

Why Worth Reading

This paper establishes an authoritative psychometric validity framework bridging the 2014 AERA/APA/NCME Standards to generative AI constructed response scoring, exposing key pitfalls of off-the-shelf LLM raters compared to traditional NLP engines.

Key Claims & Verified Quotes

Generative language models produce non-deterministic scores across repeated evaluations even when generation temperature is set to zero.

Quote: “The empirical example discussed later in this manuscript using GPT-4o and Gemini 1.5 Pro shows variability in scores even with temperature = 0.” — 10.48550/arxiv.2603.19280, p. 9, para. 4 (exact)

Limitations & Study Design Flags

No adversarial testing: not found

LLM scoring was evaluated in a zero-shot prompting configuration without few-shot in-context examples or task-specific supervised fine-tuning.
External test validity criteria and broader educational covariates were absent from the PERSUADE 2.0 dataset.
The corpus is limited to 6th–12th grade US argumentative essays evaluated under a single holistic SAT rubric.

From the article text (quote verified)

supports Essays with greater length reliably correlate with higher human ratings. «Longer essays are consistently associated with higher quality essays in many studies», p. 16, para 3

Model inference (gaps & contradictions)

[method / conceptual divergence]: Standard NLP engines optimize explicit feature weights against human targets, whereas zero-shot prompting relies on latent next-token probabilities that misalign with score distributions.

#automated essay scoring #generative AI #validity framework #e-rater #rater drift #psychometrics