Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Veronika Batzdorfer, Carlo Romano Marcello Alessandro Santagiustina · arXiv (2026) · doi:10.48550/arxiv.2609.19866 · gemini-3.1-flash-lite
TL;DR
Researchers often assume that if an AI model gives the same answer every time, it is accurately measuring what they want to study. By comparing AI-coded public submissions to direct survey responses from the same people, this study shows that the AI is actually picking up on how different organizations talk publicly, rather than their true private beliefs. This means AI-based scores can be perfectly consistent while failing to actually measure the intended concept.
Method and sample: Dataset of 857 stakeholder submissions from the European Commission’s AI Act public consultation (2020-2021). The final validation sample includes N=348 submissions that contain both free-text responses and structured survey data (safety concern, rights concern, explainability importance). The model used for annotation is Qwen3.5-397b-A17b.
#construct validity#computational social science#psychometrics#LLM annotation#institutional communication