assessment_design

Preregistration of the Expert-Panel Analysis Plan: LLMs as Pre-Administration Item Reviewers in Programming Education

Authors: Mehmet İsmail SOLMAZ
Venue / Source: Zenodo (CERN European Organization for Nuclear Research) (2026)

This study investigates whether large language models can act as reliable automated co-reviewers to replace or augment human experts in identifying flawed programming test questions.

Key takeaways

This study provides a structured framework for evaluating the alignment between human expert assessments, LLM-generated item flags, and classical psychometric metrics.
The proposed methodology utilizes statistical measures like content validity indices to quantify agreement among multi-rater panels during test development.
Small expert panel sizes and the focus on a single programming language indicate that findings may not generalize across broader disciplines or assessment contexts.
Implementing this protocol offers potential to streamline high-stakes test development by integrating automated review tools into existing item-selection workflows.

Executive Summary (TL;DR)

In plain words In the paper's terms

The authors are planning a study to see if AI can help human experts improve programming tests by catching bad questions early. They will compare how humans, AI models, and standard student test results all agree or disagree on item quality. This helps determine if AI can be a useful tool for teachers when building more accurate and fair exams.

This study preregisters an analysis plan to evaluate the alignment of human expert judgements with three LLMs and classical test theory (CTT) metrics in reviewing a 60-item programming achievement test. The method compares multi-rater expert reviews, model-based item flags, and psychometric difficulty/discrimination data to inform an item-selection protocol. The research aims to validate whether LLMs can effectively serve as automated item-reviewers in high-stakes assessment design.

Authors' abstract (original)

This record preregisters the analysis plan for the expert-panel component of a study comparing large language model (LLM) item reviews with classical item analysis for a 60-item multiple-choice C# programming achievement test (associate degree, Türkiye). The plan was registered before any expert response was collected. Classical item analysis (N = 150 students) and blind LLM evaluations (Claude Opus 5.5, GPT-6.1 Sol, DeepSeek V4 Pro; two rounds each) were completed before registration and are not part of the preregistered analyses. The plan specifies content-validity indices (I-CVI, modified kappa, S-CVI), inter-rater consistency measures, comparisons of expert judgements with LLM judgements and empirical indices, expert versus LLM accuracy in predicting the most attractive distractor, and a triangulation-based item-selection protocol. Item texts are not shared for test-security reasons. Version 1.1 (2 October 2026) adds an amendment registered before any expert response was received: panel of four experts; exclusion of an AI-generated pilot response from the preregistered analyses; and a sensitivity analysis by programming background. The primary analyses are unchanged.

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

7.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a methodology for integrating LLMs into the psychometric item selection process.
Offers a template for conducting empirical validation of LLM-based test-reviewer performance.
Addresses the practical challenge of managing test quality in the presence of generative AI.

Methodology and Dataset

Dataset consists of a 60-item C# programming multiple-choice test. The student sample (N=150) is from an associate-degree computer programming program. LLMs tested are Claude Opus 5.5, GPT-6.1 Sol, and DeepSeek V4 Pro. The expert panel consists of four to five computer education and programming experts.

Why Worth Reading

It provides a rigorous, preregistered protocol for using LLMs to assist in assessment engineering, addressing a key challenge in maintaining high-quality, AI-resistant tests through automated item review.

Key Claims & Verified Quotes

Four experts are sufficient for content validity using the I-CVI criterion of 1.00.

Quote: “With four experts, the I-CVI criterion of 1.00 (Lynn, 1986) applies” — 10.5281/zenodo.23110642, p. 3, para. 4 (exact)

Limitations & Study Design Flags

Small sample: found
“With four experts, the I-CVI criterion of 1.00 (Lynn, 1986) applies, as already specified in Section 4.2; the panel size is reported as a limitation.” (p. 3)

Panel size of four experts is relatively small, which necessitates stringent criteria for content validity.
Item texts are kept private for security, limiting the ability of external researchers to verify the item quality.
The study focuses on a specific programming language (C#), which may limit generalizability to other disciplines or assessment formats.

From the article text (quote verified)

supports The study adopts Lynn's (1986) criteria for content validity. «Criterion: I-CVI = 1.00 with 3-5 experts; I-CVI ≥ .78 with 6 or more experts (Lynn, 1986).», p. 2, para 7

Model inference (gaps & contradictions)

[method / methodological]: The paper acknowledges that Lawshe's CVR is not computed, highlighting a lack of consensus on the most appropriate validity metrics when assessment security is prioritized.

#assessment_design #item_analysis #LLM_evaluation #content_validity #psychometrics