psychometric_modeling

A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments

Authors: Jing Huang, Jihong Zhang, Hua-Hua Chang
Venue / Source: arXiv (2026)

Automated item similarity analysis using LLMs provides a more precise and psychometrically coherent way to mitigate construct-irrelevant redundancy in large-scale assessments than traditional text-matching methods.

Key takeaways

The framework successfully distinguishes between structural item features and deeper semantic relatedness to detect incidental content redundancy.
LLM-derived similarity metrics show stronger correspondence with psychometrically defined local item dependencies compared to BLEU or cosine similarity.
Integrating LLM-based similarity constraints into CAT selection algorithms improves the stability of ability estimates with minimal trade-offs in accuracy.
Traditional automated similarity metrics fail to capture the multi-dimensional nature of incidental content in educational items, necessitating a more nuanced approach.
The proposed method provides a scalable, automated alternative for item bank curation, reducing the manual burden on subject matter experts.

Executive Summary (TL;DR)

In plain words In the paper's terms

This study introduces a new AI-based method to measure how repetitive test items are by analyzing both their structure and meaning. Traditional tools only count shared words, but the proposed system catches deeper similarities that cause issues like student fatigue. Using this approach to select test questions improves the stability and fairness of computerized tests.

This study proposes a dual-dimensional LLM framework for Automated Item Similarity Analysis that decomposes incidental content into structural and semantic components. By operationalizing these dimensions through prompt engineering, the framework yields similarity metrics that align significantly better with construct-irrelevant local dependencies than traditional BLEU or cosine-based measures. Simulations in Computerized Adaptive Testing demonstrate that integrating these LLM-derived similarity constraints effectively balances psychometric accuracy with content diversity, improving estimation stability.

Authors' abstract (original)

The rapid expansion of large-scale assessments and the growing adoption of automatic item generation have intensified concerns about incidental content redundancy, where construct-irrelevant elements such as wording or contextual framing become unintentionally repetitive across items. Traditional similarity metrics like BLEU or cosine similarity, often fail to capture the nuanced structural and semantic layers that drive perceived redundancy simultaneously. This study proposes a dual-dimensional framework for Automated Item Similarity Analysis (AISA) powered by Large Language Models (LLMs), operationalizing similarity through Structured Decomposition and Semantic Relatedness. Psychometric validation indicates that LLM-derived metrics align more closely with indicators of construct-irrelevant local dependence and yield more coherent item parameter groupings than traditional text-based measures. The framework is further evaluated through its application in Computerized Adaptive Testing (CAT). Simulations reveal that incorporating LLM-based similarity constraints into item selection improves estimation stability and reduces bias with minimal efficiency trade-offs, outperforming constraints based on conventional metrics. These findings highlight the potential of LLM-powered AISA to support scalable bank curation, content-aware test assembly, and experience-sensitive adaptive testing across diverse assessment contexts.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

5.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a robust quantitative method for modeling item redundancy that can be adapted for detecting AI-generated content patterns.
Demonstrates how to use LLMs to create interpretable similarity metrics that exceed the performance of traditional NLP approaches in psychometric contexts.
Offers a replicable design for integrating complex constraints into CAT algorithms, which is directly applicable to research on test integrity under AI influence.

Methodology and Dataset

The study utilized 35,278 valid participants from the Experiences in Close Relationships (ECR) scale dataset, which contains 36 Likert-type items. The framework was implemented using Claude Sonnet 4. Baselines for comparison included traditional BLEU similarity and cosine similarity derived from a document-term matrix.

Why Worth Reading

It offers a sophisticated, AI-driven methodology for managing item bank quality and test assembly that directly addresses the psychometric challenges of incidental content redundancy in adaptive testing.

Key Claims & Verified Quotes

LLMs can function as integrative similarity analyzers, capturing information beyond simple n-gram or vector-based proximity.

Quote: “These findings suggest that LLMs can function as integrative similarity analyzers rather than simple proxies for existing metrics.” — 10.48550/arxiv.2608.24825, p. 18, para. 3 (exact)

LLM-powered similarity constraints improve robustness and consistency in ability estimation within CAT.

Quote: “similarity constraints introduce a modest efficiency trade-off but improve robustness and consistency” — 10.48550/arxiv.2608.24825, p. 19, para. 4 (exact)

Limitations & Study Design Flags

single_model: found
“this study employed a single LLM (Claude Sonnet 4) and a single item bank (the ECR scale) for validation.” (p. 19)

The study is limited by the use of a single LLM (Claude Sonnet 4) and a single item bank (ECR scale), which may limit the generalizability of the findings.
The weightings and prompt structures for the dual-dimensional framework were set based on theoretical considerations and may not be optimal for all assessment domains.
The correlation between residual correlations and similarity scores is constrained by the inherent noisiness of residual correlations as an indicator of incidental similarity.

From the article text (quote verified)

supports High incidental similarity can compromise measurement quality. «Empirical evidence suggests that high semantic similarity can compromise measurement quality.», p. 4, para 5

Model inference (gaps & contradictions)

[method / operational]: While the framework shows success on the ECR scale, it is unclear how it performs on more complex or domain-diverse item banks without extensive manual tuning of weights.

#Computerized Adaptive Testing #Item Similarity #Psychometrics #LLM Framework #Test Assembly