assessment_design

MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge

Authors: Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique
Venue / Source: arXiv (2026)

Automated benchmark generation risks creating cue-laden items that mislead model evaluation unless rigorously audited against source evidence and empirical difficulty.

Key takeaways

The framework provides a transparent and reproducible method for constructing source-grounded benchmarks to assess specialized clinical knowledge in LLMs.
Empirical evidence suggests that automated difficulty labels are unreliable, indicating that model-panel disagreement is necessary to calibrate question difficulty.
Experimental results show significant performance variation among models, with specific clinical subtopics remaining challenging regardless of overall model proficiency.
A limitation of the study is the lack of human expert adjudication, which may impact the clinical validity of the generated answer keys.
The findings suggest that complex prompt-based construction strategies do not consistently translate into increased psychometric difficulty for AI models.

Executive Summary (TL;DR)

In plain words In the paper's terms

The authors created a new system to automatically generate and check test questions about Multiple Sclerosis MRI scans, ensuring each question is linked to a reliable source. When testing 13 AI models, they found that model accuracy varied widely and that the difficulty level of questions estimated during creation was often inaccurate. The results highlight that AI-generated tests require careful cross-checking against actual performance data to be useful for assessment.

This paper presents MS-Exam-Gen, a reproducible framework for constructing and auditing source-grounded multiple-choice benchmarks for MS-MRI knowledge. The study evaluates 13 LLMs across 3,058 items, revealing significant performance variance across models and topic-specific accuracy gaps. The findings demonstrate that automated construction labels often fail to predict empirical difficulty, requiring post-generation calibration through model-panel disagreement.

Authors' abstract (original)

Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple sclerosis MRI (MS-MRI) provides a high-stakes textual-knowledge test case because correct reasoning requires current diagnostic criteria, standardized acquisition and reporting knowledge, longitudinal monitoring concepts, lesion morphology, and recognition of difficult mimics. We present MS-Exam-Gen, a reproducible framework for constructing and auditing a text-based multiple-choice question (MCQ) benchmark for MS-MRI knowledge; it does not evaluate direct MRI image interpretation. MS-Exam-Gen targets source-grounded criteria, protocols, reporting, and differential diagnosis. The framework combines expert-source indexing, exam-oriented topic induction, evidence-grounded MCQ generation, automated quality audits, a same-family consistency screen, and empirical calibration. From a 66-source corpus indexed into 4,289 retrieval chunks, the pipeline produced a locked 3,058-item candidate benchmark spanning 16 topics and 53 subtopics. Evaluation across 12 primary LLM endpoints yielded 36,696 item-level predictions and separated performance over a 42.8-percentage-point accuracy range (89.7% to 46.9%). Across these endpoints, 25.5% of items were missed by at least four. Post-generation audits showed that refreshed construction reduced measurable answer cues, while option-order testing showed that absolute MCQ scores remain position-sensitive. Generated construction labels remain metadata rather than validated psychometric categories. Because expert adjudication and full option-order counterbalancing remain future work, MS-Exam-Gen is not a clinically certified examination. It should be interpreted as an automatically filtered, source-grounded candidate benchmark and reproducible audit workflow for item-level and topic-specific LLM evaluation.

Quality & Relevance Evaluation Breakdown

Novelty

7.0/10

Methodological Rigor

8.0/10

Engineering Countermeasures

8.0/10

Consensus Challenge

5.0/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Useful for framing validity arguments under the Messick/Kane framework by shifting focus from 'aggregate score' to 'source-grounded evidence'.
Provides a concrete methodology for item-level calibration when human expert review is unavailable or expensive.
Demonstrates techniques for detecting and mitigating item preknowledge/compromise via anti-cue audits.

Methodology and Dataset

The study utilized a curated corpus of 66 sources (4,289 chunks) to construct a benchmark of 3,058 MCQs. The model panel included 13 LLMs (Gemini 2.5 Flash, GPT-5 mini, Llama 4 Maverick, DeepSeek Chat, Claude Haiku 4.5, GPT-4o mini, Gemini 2.5 Flash Lite, GPT-5 nano, Qwen3 14B, Qwen3 8B, Llama 3.1 8B, Mistral, and Phi-4 mini), with 12 serving as the primary evaluation panel (36,696 responses).

Why Worth Reading

It provides a rare, transparent, and reproducible workflow for subspecialty benchmark construction that emphasizes auditability, anti-cue filtering, and empirical calibration over blind reliance on LLM-generated difficulty labels.

Key Claims & Verified Quotes

Anti-cue filters significantly reduce the prevalence of superficial answer cues in generated MCQs.

Quote: “This supports reduced measurable cueing” — 10.48550/arxiv.2610.06170, p. 5, para. 28 (exact)

Limitations & Study Design Flags

no_full_adversarial_testing: not mentioned in the text

The benchmark lacks human expert clinical adjudication for the correctness of all generated questions and answer keys.
The study did not evaluate performance against human experts or trainees.
Option-order counterbalancing was performed only on a small pilot, not the full benchmark bank.
The corpus is broad but non-exhaustive regarding MS-MRI clinical knowledge.

From the article text (quote verified)

supports Multiple-choice evaluations are sensitive to option ordering and answer-label artifacts. «multiple-choice evaluations can be sensitive to option ordering and answer-label artifacts», p. 2, para 9
supports LLM judges exhibit position or scoring biases. «LLM judges can exhibit position or scoring biases», p. 2, para 9

Model inference (gaps & contradictions)

[Methodology / Measurement validity]: The paper underscores a critical gap in current benchmark construction: equate automated acceptance by a generator (or even a consistency screen) with true psychometric validation.

#benchmark construction #LLM evaluation #MS-MRI #source-grounded assessment #empirical difficulty calibration