The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7
Qian Shen, Yu Han · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) (2026) · doi:10.18653/v1/2026.clpsych-1.7 · gemini-3.7-flash
TL;DR
Researchers tested whether AI models could act as stand-ins for humans when pilot-testing questionnaires. Although the AI answers appeared remarkably consistent on the surface, statistical tests showed they did not capture real psychological traits. As a result, using AI to test assessment scales before administering them to real people can severely distort test design.
Method and sample: Four open-weight LLMs (Llama-3.1-8B-Instruct, Mistral-7B-Instruct, Llama-3.3-70B-Instruct, GPT-OSS-120B) were tested under a 2x3x2 factorial design (2 temperatures, 3 persona severity levels, 2 prompt conditions) across 500 iterations per condition, generating 6,000 valid responses per model per instrument on PHQ-9 and GAD-7. The human comparison baseline comprised 6,000 randomly sampled human responses from the 2017-2019 NHANES (PHQ-9) and 2019 NHIS (GAD-7). Evaluation used unidimensional CFA via robust maximum likelihood (MLR), multi-group CFA for measurement invariance, and ordinal logistic regression for differential item functioning (DIF).
#measurement invariance#synthetic respondents#differential item functioning#confirmatory factor analysis#construct validity#psychometrics#LLM evaluation