psychometric_modeling

One Percent of the Bank: Adaptive Item Selection Reads Targeted Contamination Up While the Whole Bank Barely Moves

Authors: Guan-Yuan Chen, Ya-Fen Yeh
Venue / Source: Zenodo (CERN European Organization for Nuclear Research) (2026)

Adaptive testing's touted low item exposure is a population-level statistic that fails to protect against an attacker who targets the items the algorithm will predictably ask.

Key takeaways

Adaptive testing protocols create predictable item exposure patterns that make evaluations vulnerable to targeted score inflation.
Targeting a minimal fraction of an item bank can significantly inflate adaptive test scores while leaving whole-bank performance metrics unaffected.
The study utilizes a controlled IRT-based simulation to isolate the specific impact of item selection rules on model performance estimates.
Design-unbiased estimators like uniform random sampling may serve as a safeguard against the vulnerabilities inherent in deterministic adaptive selection.
Findings are limited by the use of a single model size and specific simulation parameters, warranting caution when generalizing to broader deployment scenarios.

Executive Summary (TL;DR)

In plain words In the paper's terms

While computer-adaptive tests are efficient, they pick predictable questions that are more likely to be targeted by malicious fine-tuning. This research shows that if just 1% of the test questions are compromised in this way, the test score can be significantly inflated, even if the student's overall performance across the full pool of questions remains steady. Educators should therefore rely on whole-bank scores rather than just the score from an adaptive test session to ensure validity.

Adaptive testing risks contamination when models are fine-tuned on items predictably selected for high information. The authors demonstrate that targeting just 1% of a bank can inflate adaptive test scores by over 1.4 logits while leaving whole-bank scores virtually unchanged. This divergence occurs because adaptive protocols concentrate administrations on a subset of items, making them vulnerable to targeted leakages that static whole-bank readings effectively average out.

Authors' abstract (original)

Adaptive testing is entering language-model evaluation as an efficiency device, and its low item exposure is offered as incidental contamination protection (Li et al., 2026b). That argument assumes untargeted leakage, yet a maximum-information test asks predictable items that anyone observing administrations can learn. Here we fine-tune a 1.5B-parameter model on 1% of a self-calibrated MMLU bank, chosen by exposure in a clean adaptive run, and score adaptive, random and whole-bank protocols on the same response vectors. A 41-item adaptive test then rises by 1.47 to 1.59 logits over its own clean reading (1.63 to 1.75 above the clean whole-bank reading) while the whole bank reads slightly lower. Because this concentrated test also moves under untargeted fine-tuning drift, we compare it with a random-item set trained to approximately the same loss of clean-item accuracy: on this model the adaptive reading exceeds that control's by +1.40 logits (95% CI +1.10 to +1.69), positive in all ten seed pairs. Every item of the clean test lies inside the contaminated set, and an exposure set rebuilt only from simulated administrations reproduces the shift. Two further models show the same sign pattern; none of the selection-side defences tested in simulation meets a joint target for bias and clean RMSE. In the released logs of two adaptive LLM evaluators, the most-administered 1% of each instrument's largest bank takes 30–40% (ATLAS, HellaSwag) and 60% (Fluid, MMLU, 100-item tests) of all administrations; across all their benchmarks (Fluid at 100 items) the most-administered 1% takes 5–60%. No change confined to a targeted set can move the expected reading of a reader that is design-unbiased for bank accuracy by more than the set's share of the bank: that expectation changes by exactly the change in bank accuracy. Adaptive-test logs therefore need a threat model for their observers, and adaptive scores a whole-bank reading beside them.

Quality & Relevance Evaluation Breakdown

Novelty

8.0/10

Methodological Rigor

8.5/10

Engineering Countermeasures

7.5/10

Consensus Challenge

8.0/10

Topic Relevance

9.0/10

Relevance for Educators & Research

Provides a methodology for auditing the validity of AI-based assessment tools.
Connects IRT item-parameter drift to adversarial fine-tuning.
Offers a theoretical framework for designing 'design-unbiased' adaptive evaluators.

Methodology and Dataset

Dataset: 14,042-item MMLU bank (self-calibrated via 2PL IRT). N: 28 variants of Qwen2.5-1.5B-Instruct (fine-tuned via LoRA). Controls: Random item sets matched for collateral drift, and whole-bank readings. Primary test length: 41 items.

Why Worth Reading

This paper provides a rigorous psychometric foundation for understanding why adaptive AI evaluation benchmarks are uniquely susceptible to targeted contamination. It introduces the critical insight that adaptive selection rules create predictable 'paths' that observers can exploit, and provides a formal proof for why design-unbiased estimators are necessary to prevent score inflation in the age of AI.

Key Claims & Verified Quotes

Targeted contamination of adaptive selection paths causes significant score inflation.

Quote: “A 41-item adaptive test then rises by 1.47 to 1.59 logits over its own clean reading” — 10.5281/zenodo.23156543, p. 1, para. 11 (exact)

Limitations & Study Design Flags

Results rely on fine-tuned variants of a single primary 1.5B-parameter model.
Targeted item sets and administration patterns are highly dependent on the initial IRT calibration.
Frontier results for defense mechanisms were validated on simulated banks rather than real-world deployments.
The 'matched-collateral' control comparison is sensitive to seed variation in how fine-tuning drift manifests across the item pool.

From the article text (quote verified)

contradicts The paper refutes the claim that low exposure necessarily protects against contamination. «This paper extends the threat model to deliberate targeting, under which the low-exposure corollary does not hold for the canonical instrument.», p. 4, para 33

Model inference (gaps & contradictions)

[method / metric incompatibility]: The authors argue that standard 'amplification ratios' used in previous literature are ill-posed, replacing them with logit-difference measures.

#adaptive_testing #item_response_theory #benchmark_contamination #adversarial_robustness #psychometrics