Identifying Causes of Test Unfairness: Manipulability and Separability
Authors: Youmi Suk, Weicong Lyu
Venue / Source: arXiv (2026)
This paper replaces ambiguous group-based bias labels with a causal decomposition framework that isolates specific, remediable item characteristics that distort test validity.
Key takeaways
- Decomposing non-manipulable demographic variables into actionable components allows for the causal identification of differential item functioning.
- Simulation results suggest that causal forests and Bayesian additive regression trees effectively detect specific sources of test bias compared to traditional methods.
- The framework provides a practical pathway for diagnostic item revision by pinpointing construct-irrelevant factors like lexical proficiency as drivers of performance gaps.
- Limitations include a reliance on the assumption of no unmeasured confounding and the requirement for stable ability estimates to ensure accurate detection.
Executive Summary (TL;DR)
Authors' abstract (original)
Differential item functioning (DIF) is a widely used statistical notion for identifying items that may disadvantage specific groups of test-takers. These groups are often defined by non-manipulable characteristics, e.g., gender, race/ethnicity, or English-language learner (ELL) status. While DIF can be framed as a causal fairness problem by treating group membership as the treatment variable, this invokes the long-standing controversy over the interpretation of causal effects for non-manipulable treatments. To better identify and interpret causal sources of DIF, this study leverages an interventionist approach using treatment decomposition proposed by Robins and Richardson (2010). Under this framework, we can decompose a non-manipulable treatment into intervening variables. For example, ELL status can be decomposed into English lexical proficiency and instructional English comprehension, each of which influences the outcome through different causal pathways. We formally define separable DIF effects associated with these decomposed components, depending on the absence or presence of item impact, and provide causal identification strategies for each effect. We then apply the framework to biased test items in the SAT and Regents exams. We also provide formal detection methods using causal machine learning methods, namely causal forests and Bayesian additive regression trees, and demonstrate their performance through a simulation study and a real-world application. Finally, we discuss the implications of adopting interventionist approaches in educational testing practices.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a sophisticated framework for defining and detecting 'fairness' that goes beyond correlation.
- Offers clear algorithmic paths (Causal Forests/BART) for implementing causal DIF analysis in dissertation research.
- The discussion on 'dismissible component conditions' is essential for defending causal claims in educational assessment research.
Methodology and Dataset
The study validates its framework using a simulation study (N=1000, 2000, 4000) and an empirical application on a dataset of 2,200 U.S.-resident adults (Keller et al., 2025) involving 12 mathematics pretest items. Model versions for detection include Causal Forests (via `grf` package) and Bayesian Additive Regression Trees (BART) via the `bartCause` package, compared against traditional Mantel-Haenszel and logistic regression DIF baselines.
Why Worth Reading
This paper provides a rigorous statistical foundation for moving beyond 'black box' DIF flags. By applying modern causal inference techniques (treatment decomposition) to psychometrics, it offers educators and psychometricians a way to move from merely identifying biased items to diagnosing specific, remediable causes of bias, significantly improving the validity argument for test revision.
Key Claims & Verified Quotes
No verified quotes available for this paper.
Limitations & Study Design Flags
- The proposed detection methods require either known ability or reliably estimated ability scores, which may be sensitive to measurement error.
- The framework relies on the 'dismissible component condition' and the assumption of no unmeasured confounding, which may be challenging to satisfy in real-world educational data.
- The general separable DIF detection method (BART-based) may be conservative due to multiple-testing adjustments when using dense ability grids.
- The framework currently assumes a deterministic relationship between the non-manipulable group variable and its components, which may not hold in all socio-cultural contexts.
- [method / methodological innovation]: The authors shift from standard statistical association in DIF (Holland & Wainer, 1993) to an interventionist framework, successfully addressing the 'no causation without manipulation' challenge.
- [sample / generalizability]: The empirical validation is currently limited to adult pretest math items; further research is needed to verify performance on high-stakes academic examinations across more diverse populations.