theoretical

Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

Authors: Manuel Pita
Venue / Source: arXiv (2026)

Researcher's Note:

Якщо просити модель пояснити результат - це не означає що модель покаже своє розуміння області. Що буде збагачувати контекст - результат сформує пояснення чи пояснення формує результат - є недетермінованим. Стаття далека від освіти, але важлива для розуміння що ми можемо очікувати від LLM, а що не варто.

Large language models can produce accurate codes for the wrong reasons, making reliability an insufficient indicator of valid construct measurement.

Key takeaways

High inter-rater agreement between LLMs and humans is not sufficient evidence of construct validity.
Theory-naive instruments often rely on surface features that correlate with a construct without actually measuring it.
The 'grain calibration' procedure forces a model to test specific construct components and provide supporting evidence for each.
Validation should shift from scoring outputs against humans to proving the model follows the theoretical process defined by the construct.
Construct decomposition helps uncover hidden biases and 'shortcuts' where models latch onto irrelevant variables like demographics.

Executive Summary (TL;DR)

In plain words In the paper's terms

AI models can correctly label text while using the wrong reasoning, effectively 'cheating' by relying on surface patterns rather than the actual concepts they are supposed to measure. The authors propose a new validation method called grain calibration, which requires models to break down concepts into smaller, evidence-based steps to prove they are 'thinking' about the right things. This helps researchers ensure their AI tools are truly measuring the intended educational or psychological constructs.

Large language models often achieve high agreement with human annotators by relying on surface-level statistical correlates rather than theoretical construct definitions. The paper introduces 'grain calibration,' a method that decomposes a theoretical construct into independently testable, clause-level components to ensure the model's inferential path matches the construct's nomological network. Findings suggest that validity requires evidence of process engagement, which cannot be inferred from agreement scores alone.

Authors' abstract (original)

When a large language model (LLM) codes a construct in text as a human annotator would, that agreement makes the LLM a reliable coder. Yet reliability leaves construct validity untouched. The instrument may be theory-naive, reaching the code through a correlate that meets none of the demands the construct's theory makes, and no current method tells that apart from genuine measurement. We propose grain calibration as a method that closes the gap. It decomposes a construct into clause-level components, tests each against the text with extractive evidence, and combines the results through an explicit, theory-derived rule. Because the rule is stated rather than lodged in one opaque pass, its structure is evidence about the process rather than the output. It shows which components settled a code, and, when the code is wrong, whether a component was missed or an adjacent construct mistaken for it. Validation shifts from scoring an instrument's outputs against an annotator to showing that the instrument runs on the construct its theory specifies.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

7.0/10

Engineering Countermeasures

7.0/10

Consensus Challenge

8.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a theoretical foundation for assessment redesign under generative AI.
Offers a methodology to prevent 'shortcut learning' in AI-assisted grading systems.
Helps define what 'construct validity' looks like in automated, AI-driven evaluation.

Methodology and Dataset

This is a theoretical and methodological paper. It does not conduct original empirical experiments with primary data. It provides a synthetic review and analysis of 58 existing studies, literature, and benchmarks from computational social science and psychometrics, focusing on LLMs used as coding instruments for theoretical constructs.

Why Worth Reading

It provides a rigorous conceptual framework (grain calibration) that addresses a fundamental failure in current AI-based assessment: the confusion of reliability with validity. It is essential for researchers designing automated assessment pipelines who need to move beyond simple agreement metrics to ensure their models are measuring the correct theoretical constructs.

Key Claims & Verified Quotes

Agreement-based validation of LLMs as coders is fundamentally insufficient because it ignores the 'substantive aspect' of validity.

Quote: “A theory-naive instrument has not earned Messick’s substantive aspect.” — 10.48550/arxiv.2606.28574, p. 6, para. 5 (exact)

Limitations & Study Design Flags

Grain calibration requires the target construct to be 'codable' (defined a priori), meaning it cannot be applied to constructs emergent from grounded theory.
The method relies on a human in the loop, introducing potential for researcher bias in defining the decomposition.
The paper does not provide empirical results on the performance of grain calibration, as it is a theoretical proposal.
The method may not be necessary for simple constructs where distributional competence is sufficient, such as basic sentiment polarity.

Model inference (gaps & contradictions)

[method / operationalization]: The paper critiques existing validity frameworks like ValiText for focusing on pre-measurement justifications rather than empirical process evidence.
[metric / validity definition]: Standard agreement metrics (F1, Cohen's kappa) are identified as measuring reliability, which is conflated with validity in current AI literature.

#assessment validity #construct validity #AI measurement #computational social science #theory-naivete #grain calibration