Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark
Eunjeong Song, Sehee Hong · arXiv (2026) · doi:10.48550/arxiv.2609.25059 · gemini-3.1-flash-lite
TL;DR
Researchers tested whether AI could replace human participants in the early stages of survey design. While the AI provided consistent answers, it failed to mimic the full range of human responses, particularly those of lower-skilled users. This gap would lead developers to make incorrect decisions about how to structure and score their tests.
Method and sample: Dataset consists of N=6,245 adults (age 19+) from the 2025 Survey on the Digital Divide (South Korea). Synthetic responses were generated for 1,300 demographically matched personas using the NVIDIA Nemotron-Personas Korea profiles and the gpt-oss-120b model. Baselines included human responses under the same item set and preregistered prompt/persona variations.
#psychometrics#Rasch_model#synthetic_data#assessment_development#construct_validity