Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Manuel Pita · arXiv (2026) · doi:10.48550/arxiv.2606.28574 · gemini-3.1-flash-lite
TL;DR
AI models can correctly label text while using the wrong reasoning, effectively 'cheating' by relying on surface patterns rather than the actual concepts they are supposed to measure. The authors propose a new validation method called grain calibration, which requires models to break down concepts into smaller, evidence-based steps to prove they are 'thinking' about the right things. This helps researchers ensure their AI tools are truly measuring the intended educational or psychological constructs.
Method and sample: This is a theoretical and methodological paper. It does not conduct original empirical experiments with primary data. It provides a synthetic review and analysis of 58 existing studies, literature, and benchmarks from computational social science and psychometrics, focusing on LLMs used as coding instruments for theoretical constructs.
#assessment validity#construct validity#AI measurement#computational social science#theory-naivete#grain calibration