Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking
Authors: Tianpeng Zheng, Zhehan Jiang, Jiayi Liu, Shicong Feng
Venue / Source: arXiv (2026)
Computerized adaptive testing evaluates medical models with 98% fewer questions while perfectly preserving full-benchmark model rankings.
Key takeaways
- Adaptive item selection based on Fisher information replicates full-benchmark model rankings using on average only 37 items per model.
- Non-adaptive random item reduction produces systematic bias and severe regression to the mean for top-performing models.
- Calibrating item parameters on human examinees provides a stable psychometric scale for tracking language model proficiency drift.
- Multiple-choice adaptive proficiency correlates strongly with performance on open-ended clinical question answering tasks.
Executive Summary (TL;DR)
Authors' abstract (original)
The rapid proliferation of large language models (LLMs) in healthcare creates an urgent need for scalable and psychometrically sound evaluation methods. Conventional static benchmarks are costly to administer repeatedly, vulnerable to data contamination, and lack calibrated measurement properties for fine-grained performance tracking. We propose and validate a computerized adaptive testing (CAT) framework grounded in item response theory (IRT) for efficient assessment of standardized medical knowledge in LLMs. The study comprises a two-phase design: a Monte Carlo simulation to identify optimal CAT configurations and an empirical evaluation of 38 LLMs using a human-calibrated medical item bank. Each model completed both the full item bank and an adaptive test that dynamically selected items based on real-time ability estimates and terminated upon reaching a predefined reliability threshold (standard error <= 0.3). Results show that CAT-derived proficiency estimates achieved a near-perfect correlation with full-bank estimates (r = 0.988) while using only 1.3 percent of the items. Evaluation time was reduced from several hours to minutes per model, with substantial reductions in token usage and computational cost, while preserving inter-model performance rankings. This work establishes a psychometric framework for rapid, low-cost benchmarking of foundational medical knowledge in LLMs. The proposed adaptive methodology is intended as a standardized pre-screening and continuous monitoring tool and is not a substitute for real-world clinical validation or safety-oriented prospective studies.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Offers a rigorous methodology for evaluating generative LLMs using computerized adaptive testing (CAT) and item response theory (IRT).
- Examines how human-calibrated item banks can measure machine performance and detect shifts across model iterations.
- Illustrates the contrast between random item sampling and adaptive selection regarding measurement bias and regression to the mean.
Methodology and Dataset
Two-phase design combining Monte Carlo simulation (3,600 simulated examinees across 36 uniform ability levels from -3.5 to +3.5) and empirical testing of 38 proprietary and open-source LLMs (ranging from 0.5B to 120B+ parameters; e.g., GPT-4, o1, Gemini-2.5-Pro, Claude-4-Sonnet, Qwen series). The item bank contains 2,815 multiple-choice questions from China's National Competency Test for Clinical Medicine Undergraduates (CTCMU), calibrated on over 40,000 fourth- and fifth-year medical students using the 2-parameter logistic IRT model. Baselines include full-bank testing (2,815 items) and random-selection subtests matched on length.
Why Worth Reading
It provides a rigorous demonstration of how modern psychometrics (specifically IRT and CAT) solves the computational bottleneck and contamination risks of testing LLMs. For educational measurement researchers, it operationalizes LLMs as examinees on human-calibrated scales, illustrating how adaptive algorithms maintain rank fidelity under extreme test shortening.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Small sample: not found
- Single detector: not mentioned in the text
- The item bank relies exclusively on a single national undergraduate medical examination blueprint, which may not transfer directly to other curricula such as USMLE without formal psychometric equating.
- Standardized multiple-choice items assess foundational medical knowledge rather than complex clinical reasoning, bedside skills, or patient communication.
- The framework does not guarantee safety clearances against dangerous non-monotonic errors, where a model misses basic safety-critical knowledge despite high overall theta.
- [definition / conceptual divergence]: The authors define theta strictly as an operational proficiency index on a fixed calibrated bank rather than a latent cognitive construct, which diverges from conventional human psychometric interpretations.
- [method / measurement gap]: IRT assumes monotonic response functions, but generative models often exhibit brittle, non-monotonic failures on elementary safety items despite answering complex items correctly.