BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibration
James Sharpnack, Kevin Hao, Phoebe Mulcaire, Klinton Bicknell, Geoff LaFlair, Kevin Yancey, Alina A. von Davier · arXiv (2024) · doi:10.48550/arxiv.2410.21033 · gemini-3.1-flash-lite
TL;DR
This paper introduces new machine learning methods to build efficient computerized adaptive tests that require fewer questions to accurately measure student ability. By combining automated scoring models with a strategy that selects questions to maximize information about a student, the authors demonstrate that tests can be much shorter without sacrificing accuracy. The approach was successfully tested on large-scale practice data for the Duolingo English Test, showing effective performance even when starting with limited information about new questions.
Method and sample: The study utilized large-scale practice session data for the Duolingo English Test (DET). Five experimental blocks were conducted between January and February 2023, with participant counts (N) per condition ranging from approximately 4,000 to 14,000. Models tested included AutoIRT (ensemble of Random Forest, LightGBM, XGBoost, and CATBoost) and BanditCAT V1 (Thompson sampling based on Fisher information). Baselines consisted of legacy fixed-length card-based item administration.
#computerized adaptive testing#item response theory#machine learning#psychometrics#assessment design#contextual bandits