psychometric_modeling

Reproducibility is not construct validity: LLM measurement of institutionally situated communication

Authors: Veronika Batzdorfer, Carlo Romano Marcello Alessandro Santagiustina
Venue / Source: arXiv (2026)

Reproducible LLM annotations of public text do not necessarily measure the same construct as survey responses, revealing systematic institutional communication biases.

Key takeaways

High procedural reproducibility of LLM annotations does not inherently demonstrate construct validity when measuring complex communication.
Automated coding of public text may capture institutional rhetoric and strategic incentives rather than the private beliefs of individual respondents.
Discrepancies between model outputs and survey data often follow systematic patterns, suggesting that measurement misalignment is context-dependent and geographically clustered.
Researchers should exercise caution, as consistent LLM metrics may lack convergence with traditional psychometric measures, potentially conflating surface-level discourse with latent constructs.
Findings are limited by the specific policy domain and model evaluated, indicating a need for further testing across diverse linguistic and regulatory contexts.

Executive Summary (TL;DR)

In plain words In the paper's terms

Researchers often assume that if an AI model gives the same answer every time, it is accurately measuring what they want to study. By comparing AI-coded public submissions to direct survey responses from the same people, this study shows that the AI is actually picking up on how different organizations talk publicly, rather than their true private beliefs. This means AI-based scores can be perfectly consistent while failing to actually measure the intended concept.

High LLM annotation reproducibility does not guarantee construct validity when measuring institutionally situated communication. The authors demonstrate a profound dissociation between reproducible LLM scores and survey-reported stakeholder concerns. This divergence is systematically patterned by stakeholder type and geographic region, revealing how public discourse artifacts reflect institutional incentives rather than just underlying latent attitudes.

Authors' abstract (original)

High annotation reproducibility does not necessarily imply that an LLM-inferred measure captures the construct it is intended to measure. We test this distinction using a dataset from the European Commission's AI Act consultation, linking structured survey responses to free-text consultation submissions from the same stakeholders. LLM annotations of consultation submissions are highly reproducible (intraclass correlations > 0.99), yet show limited convergence with survey-reported measures of the nominal construct they were intended to approximate. Divergence between survey-and LLM-inferred text-based measures varies systematically across stakeholder groups: business associations express greater concern about AI risks in text-based consultations than in survey responses ({g} = +1.0), whereas public authorities and several nonbusiness groups show smaller or negative divergences. Divergences between scores suggest positive spatial autocorrelation across European countries (Moran's I = 0.347, p = 0.036), indicating that stakeholders from neighboring countries tend toward more similar text-based stances towards AI safety concerns. Despite divergence, survey-reported concerns remain strongly associated with support for explainability across all divergence levels. These results demonstrate that LLM annotation reproducibility can coexist with poor construct correspondence and motivate validation procedures that distinguish reproducibility, construct validity, and communication context variation when LLMs are used as measurement instruments.

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

5.0/10

Consensus Challenge

8.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a critical framework for evaluating LLM-based assessment instruments in education.
Highlights the risk of collider-stratification bias when using text-derived features as controls in downstream educational modeling.
Emphasizes the distinction between procedural reproducibility and construct validity, essential for high-stakes assessment audits.

Methodology and Dataset

Dataset of 857 stakeholder submissions from the European Commission’s AI Act public consultation (2020-2021). The final validation sample includes N=348 submissions that contain both free-text responses and structured survey data (safety concern, rights concern, explainability importance). The model used for annotation is Qwen3.5-397b-A17b.

Why Worth Reading

This paper offers a rigorous psychometric critique of the 'text-as-data' paradigm in computational social science. It provides a foundational warning for educators and researchers using LLMs for automated grading or assessment, demonstrating that procedural consistency (reproducibility) is a necessary but insufficient condition for construct validity in complex communicative environments.

Key Claims & Verified Quotes

LLM annotation reproducibility does not guarantee construct validity when applied to institutionally situated communication.

Quote: “High annotation reproducibility does not necessarily imply that an LLM-inferred measure captures the construct it is intended to measure.” — 10.48550/arxiv.2609.19866, p. 1, para. 5 (exact)

Limitations & Study Design Flags

Monolingual: not mentioned in the text

Analysis restricted to a single policy domain (EU AI regulation) and a single consultation process.
The findings are based on a single model (Qwen3.5-397b-A17b); generalizability across other LLMs remains to be tested.
The prompt methodology may reflect Anglo-American textual conventions that do not perfectly align with European regulatory and precautionary rhetoric.
The study does not establish an epistemic hierarchy; survey measures may also suffer from social desirability bias.

From the article text (quote verified)

supports LLM annotation reproducibility is distinct from construct validity. «Reproducibility is not construct validity: LLM measurement of institutionally situated communication», p. 1, para 1
supports Strong performance in matching human annotations does not establish substantive validity. «But as pointed out by Baden et al. [14], strong performance in matching human annotations or given ground truths does not establish substantive validity», p. 3, para 4

Model inference (gaps & contradictions)

[method / Measurement Equivalence]: LLM annotators infer 'salience/presence' while survey items measure 'intensity', leading to systematic score compression at the high end of the scale for the LLM proxy.

#construct validity #computational social science #psychometrics #LLM annotation #institutional communication