Field-Testing Multiple-Choice Questions with AI Examinees: English Grammar Items
Authors: Hotaka Maeda
Venue / Source: Educational and Psychological Measurement (2025)
AI examinees can replace human participants in the resource-heavy field-testing of new test items, enabling automated calibration and latent trait estimation.
Key takeaways
- AI examinees can be trained using item response theory parameters to generate data that reflects human-like performance metrics.
- The 'Separate' modeling approach, where each option is processed individually, outperforms concatenation-based methods for identifying item difficulty and discrimination.
- AI field-testing remains less accurate than human-based testing but provides a viable, low-cost alternative for large-scale assessment expansion.
- The study highlights the risk of 'catastrophic modeling failure' in AI-generated item parameters, suggesting that some items may confuse AI models in ways not seen with human test-takers.
Executive Summary (TL;DR)
Authors' abstract (original)
Field-testing is an essential yet often resource-intensive step in the development of high-quality educational assessments. I introduce an innovative method for field-testing newly written exam items by substituting human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ, and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was 0.82 (bias = 0.00, root mean squared error = 0.18). The study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach did not achieve the level of accuracy obtainable with human examinee response data. If further refined, potential resource savings in transitioning from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low-quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Provides a methodology for integrating LLMs into existing psychometric workflows.
- Demonstrates how to handle item bank expansion without needing human subjects.
- Discusses the limitations of using standard language models for psychometric item validation.
Methodology and Dataset
The dataset consists of 466 four-option multiple-choice English grammar items from a large-scale K-12 assessment (grades 3-8, 11). The study used DeBERTa-v3-large transformer models, fine-tuned as AI examinees across 61 distinct latent ability levels (theta ranging from -3.0 to 3.0). Baselines for evaluation included both 'Simulated_Human' data (using 1,000 simulated human examinees) and a 'Predicted_2PL' approach commonly found in existing NLP literature.
Why Worth Reading
It is the first study to treat LLMs as 'AI examinees' by embedding them directly into an IRT-based field-testing workflow, offering a practical path toward reducing the human-subject bottleneck in test item development.
Key Claims & Verified Quotes
No verified quotes available for this paper.
Limitations & Study Design Flags
- Unrepresentative sample: not found
- The study utilized previously calibrated human data as 'ground truth', which acts as an estimate rather than an absolute baseline.
- The method requires manual item text manipulation for transformer consumption, which might alter perceived item difficulty.
- The research is limited to English grammar items and may not generalize to other subject matters or item formats without further adaptation.
- The study lacks a differential item functioning (DIF) analysis to investigate potential biases in AI examinees.
- supports Transformer-based NLP methods are generally more effective at predicting item difficulty than traditional linguistic feature regression models. «Transformers were the most predictive of item difficulty, but the accuracy depended heavily on the type of exam.», p. 21, para 25
- [method / methodological gap]: The study contrasts its IRT-integrated AI approach with previous literature that relied on simple difficulty prediction, noting that simple prediction fails to capture the comprehensive nature of traditional field-testing.