Preregistration of the Expert-Panel Analysis Plan: LLMs as Pre-Administration Item Reviewers in Programming Education
Mehmet İsmail SOLMAZ · Zenodo (CERN European Organization for Nuclear Research) (2026) · doi:10.5281/zenodo.23110642 · gemini-3.1-flash-lite
TL;DR
The authors are planning a study to see if AI can help human experts improve programming tests by catching bad questions early. They will compare how humans, AI models, and standard student test results all agree or disagree on item quality. This helps determine if AI can be a useful tool for teachers when building more accurate and fair exams.
Method and sample: Dataset consists of a 60-item C# programming multiple-choice test. The student sample (N=150) is from an associate-degree computer programming program. LLMs tested are Claude Opus 5.5, GPT-6.1 Sol, and DeepSeek V4 Pro. The expert panel consists of four to five computer education and programming experts.
#assessment_design#item_analysis#LLM_evaluation#content_validity#psychometrics