psychometric_modeling

Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark

Authors: Eunjeong Song, Sehee Hong
Venue / Source: arXiv (2026)

Large language models provide deceptively high internal consistency while failing to capture the low-end variation essential for validating rating scales.

Key takeaways

Large language models demonstrate high internal consistency yet fail to capture the variance characteristic of human response patterns at the lower end of the ability scale.
The deterministic nature of synthetic respondents results in infit statistics that may falsely indicate model overfit rather than identifying poor item performance.
Relying on synthetic data for scale development may lead practitioners to incorrectly merge response categories or misjudge the targeting of assessment instruments.
Efforts to improve synthetic response diversity through persona enrichment or specific prompting strategies may be insufficient to replicate the complexity of human respondent behavior.
The study highlights the critical need for psychometric validation of synthetic respondents, as superficial metrics like internal consistency do not ensure construct validity.

Executive Summary (TL;DR)

In plain words In the paper's terms

Researchers tested whether AI could replace human participants in the early stages of survey design. While the AI provided consistent answers, it failed to mimic the full range of human responses, particularly those of lower-skilled users. This gap would lead developers to make incorrect decisions about how to structure and score their tests.

The authors calibrate a human-based Rasch model on 14 digital skill items to assess if LLM-generated persona responses can serve as valid pilot data. They find that LLMs consistently fail to represent the lower end of the human response distribution despite high internal consistency. Consequently, LLM data misrepresents category functioning and targeting, necessitating caution when using synthetic respondents for scale development.

Authors' abstract (original)

Large language models (LLMs) are proposed as synthetic respondents for pilot testing, but their usefulness depends on whether they supply the evidence assessment development requires. We calibrated rating scale models on 14 digital-use skill items from 6,245 adults and used the human item parameters to evaluate responses generated for 1,300 demographically matched personas. LLM responses had high internal consistency ($α\approx .94$) but used the lowest category in 0.2-0.3% of responses versus 13.7-21.5% for humans, and no persona chose it on every item. These gaps changed the response-category judgment on both subscales and the PC targeting judgment; on the human-calibrated scales, 12 of 14 items had infit below 0.70, indicating responses more predictable than the Rasch model expects. Preregistered changes to the prompt, category order, and persona information did not restore the human lower range. High internal consistency is insufficient evidence that LLM responses can replace human pilot data.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

6.0/10

Consensus Challenge

5.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a framework for validating synthetic data against human-calibrated reference scales.
Defines specific indicators (all-items-lowest patterns, item infit) for detecting LLM distribution bias.
Challenges the validity argument for using AI as a stand-in for human pilots in educational assessment development.

Methodology and Dataset

Dataset consists of N=6,245 adults (age 19+) from the 2025 Survey on the Digital Divide (South Korea). Synthetic responses were generated for 1,300 demographically matched personas using the NVIDIA Nemotron-Personas Korea profiles and the gpt-oss-120b model. Baselines included human responses under the same item set and preregistered prompt/persona variations.

Why Worth Reading

This paper provides a rigorous psychometric critique of the 'synthetic respondent' trend. It moves beyond superficial global metrics like Cronbach's alpha to evaluate LLM validity through the lens of Rasch-based assessment development, demonstrating that high internal consistency is not evidence of construct validity.

Key Claims & Verified Quotes

LLM responses fail to replicate human lower-range response behavior, regardless of demographic matching.

Quote: “Preregistered changes to the prompt, category order, and persona information did not restore the human lower range.” — 10.48550/arxiv.2609.25059, p. 1, para. 17 (exact)

High internal consistency in LLM responses is not evidence of valid performance as a synthetic respondent.

Quote: “High internal consistency is insufficient evidence that LLM responses can replace human pilot data.” — 10.48550/arxiv.2609.25059, p. 1, para. 18 (exact)

Limitations & Study Design Flags

The study examined only one instrument, one model, and one cultural context.
Construct-relevant persona information was limited in depth, which may have contributed to the missing lower-range representation.
The study does not establish performance for objective tests, children, or youth populations.
The analysis utilized joint maximum likelihood estimation (JMLE) without bias correction.

Model inference (gaps & contradictions)

[metric / methodological oversight]: High Cronbach's alpha in LLM data masks severe distribution failures in the tail, contradicting the reliance on alpha as a proxy for respondent validity.

#psychometrics #Rasch_model #synthetic_data #assessment_development #construct_validity