The Reliability Illusion in Synthetic Patients: Psychometric Misalignment of Open-weight LLMs on PHQ-9 and GAD-7
Authors: Qian Shen, Yu Han
Venue / Source: Workshop on Computational Linguistics and Clinical Psychology (CLPsych) (2026)
High reliability in LLM-generated survey data is an illusion: synthetic respondents mimic consistency while completely distorting latent measurement structure.
Key takeaways
- High internal consistency in synthetic data reflects superficial semantic matching rather than authentic latent psychological traits.
- No tested language model achieved scalar measurement invariance against authentic human baseline response data.
- Synthetic respondents showed extreme prompt fragility, altering underlying measurement geometry when prompt instructions varied slightly.
- Automated scale pre-testing using language models risks generating fundamentally misleading psychometric parameter estimates.
Executive Summary (TL;DR)
Authors' abstract (original)
Globally, the incidence of depression and anxiety continues to rise, and the importance of mental health assessment scales as diagnostic tools has grown accordingly.Researchers are increasingly employing generative AI to produce large volumes of items and entire scales, which in turn elevates the costs of validating their reliability and validity.In this study, we used four open-weight LLMs to complete the GAD-7 and PHQ-9, varying prompts, sampling temperature, and dynamic contextual scenarios to emulate realistic human response patterns.Using multi-group confirmatory factor analysis, differential item functioning analyses, and other psychometric methods, we evaluate the factor structure of LLM-generated responses and assess measurement invariance relative to human responses.Our findings reveal a critical paradox: although open-weight LLMs exhibit exceptionally high internal consistency, they demonstrate severe structural mismatch and fail to achieve scalar measurement invariance against human baselines.Furthermore, pervasive differential item functioning and extreme prompt fragility indicate that these models rely on superficial, stereotype-driven semantic matching rather than simulating stable latent psychological dynamics.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly relevant to research investigating LLMs as synthetic test takers, automated item piloting, and synthetic examinees.
- Provides a reproducible methodological blueprint using multi-group CFA and DIF to test measurement invariance between human examinees and generative AI.
- Serves as an empirical warning for educational psychometricians attempting to calibrate item banks or examine validity arguments using AI responses.
Methodology and Dataset
Four open-weight LLMs (Llama-3.1-8B-Instruct, Mistral-7B-Instruct, Llama-3.3-70B-Instruct, GPT-OSS-120B) were tested under a 2x3x2 factorial design (2 temperatures, 3 persona severity levels, 2 prompt conditions) across 500 iterations per condition, generating 6,000 valid responses per model per instrument on PHQ-9 and GAD-7. The human comparison baseline comprised 6,000 randomly sampled human responses from the 2017-2019 NHANES (PHQ-9) and 2019 NHIS (GAD-7). Evaluation used unidimensional CFA via robust maximum likelihood (MLR), multi-group CFA for measurement invariance, and ordinal logistic regression for differential item functioning (DIF).
Why Worth Reading
This paper rigorously dismantles the dangerous assumption that synthetic LLM respondents can substitute for human test-takers or examinees during assessment pre-testing, proving that surface-level reliability hides catastrophic psychometric invalidity.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- No adversarial testing: not mentioned in the text
- Small sample: not found
- Evaluated only open-weight models rather than frontier proprietary closed-source models like GPT-4 or Gemini.
- Simulated persona conditions enforced balanced mental health proportions, whereas clinical or community prevalence is naturally unbalanced.
- Did not systematically ablate the independent contributions of dynamic situational modifiers versus prompt phrasing.
- Compared synthetic datasets to only a single sampled cross-section of human survey data.
- supports LLM synthetic respondents only partially reproduce human psychometric patterns in survey and scale contexts. «This suggests that LLM-generated data may be useful for early group-level psychometric prototyping, but cannot replace individual-level validation.», p. 2, para 3
- supports LLMs idiosyncratically process ordinal scales by mechanically concentrating probability on arbitrary numeric tokens. «This mechanical token bias provides a critical mechanistic context for why synthetic responses may mimic human averages but fail structural validation.», p. 2, para 3
- [method / methodological gap]: Studies relying merely on correlation or Cronbach's alpha to claim LLM-human equivalence overlook factor structure and measurement invariance.
- [definition / conceptual divergence]: High Cronbach's alpha in LLMs stems from stereotype-driven semantic matching across prompts rather than human-like continuous latent traits.