psychometric_modeling

Reducing Differential Item Functioning via Process Data

Authors: Ling Chen, Susu Zhang, Jingchen Liu
Venue / Source: arXiv (2025)

Leveraging digital process data allows assessment designers to mathematically neutralize unfair test items and ensure accurate, unbiased student ability estimates.

Key takeaways

Leveraging process data to construct a nuisance trait surrogate effectively minimizes differential item functioning and improves parameter estimation accuracy.
The proposed approach maintains robustness across different psychometric frameworks, including linear regression and multidimensional logistic models.
The method requires the pre-identification of anchor items, which can introduce sensitivity to estimation bias in practical applications.
Integrating process-based corrections may inadvertently affect the overall reliability or measurement precision of the intended target trait.

Executive Summary (TL;DR)

In plain words In the paper's terms

Educational tests often contain items that perform differently for different groups of students, which is unfair. This paper uses the detailed digital logs of students' actions (process data) during a test to identify and neutralize the causes of this unfairness. They developed a mathematical method to 'de-bias' these test items, ensuring that the final scores more accurately reflect student ability rather than irrelevant group differences.

The study introduces a method to reduce differential item functioning (DIF) in computer-based assessments by leveraging process data logs. It constructs a nuisance trait surrogate using a linear combination of process features to minimize DIF-related likelihood differences. Quantitative results show that this approach effectively reduces DIF, improves item parameter estimation, and corrects for target trait estimation bias across simulation and real-world datasets.

Authors' abstract (original)

Testing fairness is a major concern in psychometric and educational research. A typical approach for ensuring testing fairness is through differential item functioning (DIF) analysis. DIF arises when a test item functions differently across subgroups that are typically defined by the respondents' demographic characteristics. Most of the existing research has focused on the statistical detection of DIF, yet less attention has been given to reducing or eliminating DIF and understanding why it occurs. Simultaneously, the use of computer-based assessments has become increasingly popular. The data obtained from respondents interacting with an item are recorded in computer log files and are referred to as process data. Process data provide valuable insights into respondents' problem-solving strategies and progress, offering new opportunities for DIF analysis. In this paper, we propose a novel method within the framework of generalized linear models (GLMs) that leverages process data to reduce and understand DIF. Specifically, we construct a nuisance trait surrogate with the features extracted from process data. With the constructed nuisance trait, we introduce a new scoring rule that incorporates respondents' behaviors captured through process data on top of the target latent trait. We demonstrate the efficiency of our approach through extensive simulation experiments and an application to thirteen Problem Solving in Technology-Rich Environments (PSTRE) items from the 2012 Programme for the International Assessment of Adult Competencies (PIAAC) assessment.

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

5.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a scalable, quantitative framework for handling item-level distortion.
Demonstrates the use of latent trait correction to improve validity in digital assessments.
Offers a robust methodology for identifying and interpreting 'nuisance' behaviors in computerized test-taking.

Methodology and Dataset

The study uses the 2012 PIAAC dataset, specifically 13 Problem Solving in Technology-Rich Environments (PSTRE) items from 8,398 respondents. Simulation experiments tested sample sizes (N) of 200, 500, and 1000 across 18 conditions (varying DIF effect sizes and item proportions) using linear factor models and M2PL IRT models.

Why Worth Reading

This paper provides a rigorous, computationally-driven solution to a persistent psychometric problem: reducing test bias using the growing availability of fine-grained process data, which is essential for maintaining assessment validity in the digital age.

Key Claims & Verified Quotes

No verified quotes available for this paper.

Limitations & Study Design Flags

Single detector: not mentioned in the text

Introducing a nuisance trait surrogate may potentially reduce the overall measurement reliability/Fisher information of the target trait.
The method requires identifying DIF-free anchor items beforehand, which can be challenging and sensitive to estimation bias.
Current implementation assumes linear relationships between process features and the nuisance trait, though non-linear extensions are possible.

Model inference (gaps & contradictions)

[Methodology / theoretical]: While the authors note that non-linear relationships could be modeled by neural networks, the current empirical results rely strictly on linear combinations of features.

#psychometrics #DIF #process_data #IRT #assessment_validity