One Percent of the Bank: Adaptive Item Selection Reads Targeted Contamination Up While the Whole Bank Barely Moves
Guan-Yuan Chen, Ya-Fen Yeh · Zenodo (CERN European Organization for Nuclear Research) (2026) · doi:10.5281/zenodo.23156543 · gemini-3.1-flash-lite
TL;DR
While computer-adaptive tests are efficient, they pick predictable questions that are more likely to be targeted by malicious fine-tuning. This research shows that if just 1% of the test questions are compromised in this way, the test score can be significantly inflated, even if the student's overall performance across the full pool of questions remains steady. Educators should therefore rely on whole-bank scores rather than just the score from an adaptive test session to ensure validity.
Method and sample: Dataset: 14,042-item MMLU bank (self-calibrated via 2PL IRT). N: 28 variants of Qwen2.5-1.5B-Instruct (fine-tuned via LoRA). Controls: Random item sets matched for collateral drift, and whole-bank readings. Primary test length: 41 items.
#adaptive_testing#item_response_theory#benchmark_contamination#adversarial_robustness#psychometrics