psychometric_modeling

Field-Testing Multiple-Choice Questions with AI Examinees: English Grammar Items

Authors: Hotaka Maeda
Venue / Source: Educational and Psychological Measurement (2025)

AI examinees can replace human participants in the resource-heavy field-testing of new test items, enabling automated calibration and latent trait estimation.

Key takeaways

  • AI examinees can be trained using item response theory parameters to generate data that reflects human-like performance metrics.
  • The 'Separate' modeling approach, where each option is processed individually, outperforms concatenation-based methods for identifying item difficulty and discrimination.
  • AI field-testing remains less accurate than human-based testing but provides a viable, low-cost alternative for large-scale assessment expansion.
  • The study highlights the risk of 'catastrophic modeling failure' in AI-generated item parameters, suggesting that some items may confuse AI models in ways not seen with human test-takers.

Executive Summary (TL;DR)

Wording

Educational assessments require testing new questions on students before they are used in real exams, which is a slow and expensive process. This paper shows that using AI models as 'test-takers' to answer these new questions can generate data that mimics human responses, allowing educators to evaluate question quality without needing hundreds of human participants. While not as perfect as human data, this approach could significantly speed up test development and reduce costs.

This study proposes an 'AI field-testing' method that replaces human examinees with fine-tuned transformer-based AI examinees to evaluate new assessment items. By integrating transformer LLMs with 2-parameter logistic (2PL) item response theory, the study demonstrates that AI examinees can generate response data suitable for item calibration, discrimination analysis, and latent trait scoring. Although AI performance remains less accurate than human data, the method offers a scalable way to expand item banks while reducing reliance on costly, time-consuming human pretesting.

Authors' abstract (original)

Field-testing is an essential yet often resource-intensive step in the development of high-quality educational assessments. I introduce an innovative method for field-testing newly written exam items by substituting human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ, and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was 0.82 (bias = 0.00, root mean squared error = 0.18). The study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach did not achieve the level of accuracy obtainable with human examinee response data. If further refined, potential resource savings in transitioning from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low-quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank.

Quality & Relevance Evaluation Breakdown

Novelty
8.0/10
Methodological Rigor
7.0/10
Engineering Countermeasures
7.0/10
Consensus Challenge
5.0/10
Topic Relevance
8.0/10

Relevance for Educators & Research

  • Provides a methodology for integrating LLMs into existing psychometric workflows.
  • Demonstrates how to handle item bank expansion without needing human subjects.
  • Discusses the limitations of using standard language models for psychometric item validation.

Methodology and Dataset

The dataset consists of 466 four-option multiple-choice English grammar items from a large-scale K-12 assessment (grades 3-8, 11). The study used DeBERTa-v3-large transformer models, fine-tuned as AI examinees across 61 distinct latent ability levels (theta ranging from -3.0 to 3.0). Baselines for evaluation included both 'Simulated_Human' data (using 1,000 simulated human examinees) and a 'Predicted_2PL' approach commonly found in existing NLP literature.

Why Worth Reading

It is the first study to treat LLMs as 'AI examinees' by embedding them directly into an IRT-based field-testing workflow, offering a practical path toward reducing the human-subject bottleneck in test item development.

Key Claims & Verified Quotes

No verified quotes available for this paper.

Limitations & Study Design Flags

  • Unrepresentative sample: not found
  • The study utilized previously calibrated human data as 'ground truth', which acts as an estimate rather than an absolute baseline.
  • The method requires manual item text manipulation for transformer consumption, which might alter perceived item difficulty.
  • The research is limited to English grammar items and may not generalize to other subject matters or item formats without further adaptation.
  • The study lacks a differential item functioning (DIF) analysis to investigate potential biases in AI examinees.
From the article text (quote verified)
  • supports Transformer-based NLP methods are generally more effective at predicting item difficulty than traditional linguistic feature regression models. «Transformers were the most predictive of item difficulty, but the accuracy depended heavily on the type of exam.», p. 21, para 25
Model inference (gaps & contradictions)
  • [method / methodological gap]: The study contrasts its IRT-integrated AI approach with previous literature that relied on simple difficulty prediction, noting that simple prediction fails to capture the comprehensive nature of traditional field-testing.