psychometric_modeling

Large Language Model Accuracy on Subspecialty Anaesthesiology Examination Questions

Authors: Muhammed Nezih KOÇ, Sait Ramazan GÜLBAY
Venue / Source: Journal of Contemporary Medicine (2026)

Standard statistical analyses in LLM exam benchmarks understate measurement uncertainty by half, creating false subgroup performance claims.

Key takeaways

Naive binomial confidence intervals severely overestimate statistical precision in LLM benchmark evaluations because repeated queries on the same items are correlated.
Cluster-bootstrap resampling at the item level reveals that most reported subgroup and domain-level LLM performance differences are statistically indistinguishable.
Small test banks of roughly 120 items can effectively separate models with large overall performance gaps but cannot reliably identify domain-specific weaknesses.

Executive Summary (TL;DR)

In plain words In the paper's terms

This study tested seven AI models on a 120-question medical examination, repeating each question five times to check for consistent performance. The researchers found that conventional statistical methods artificially inflate confidence in AI scores by ignoring that repeating the same questions creates repeated measures rather than independent data points. When properly analyzed, overall model accuracy differences remain real, but fine-grained claims about specific domain strengths and weaknesses mostly collapse.

Evaluating large language models on repeated examination items creates clustered observations that invalidate standard naive confidence intervals. Administering a 120-item subspecialty anaesthesiology question bank to seven models across five runs showed that cluster-robust confidence intervals were wider than naive Wilson intervals in 25 of 26 estimates by a median factor of 2.1. Consequently, while 120-item benchmarks can reliably rank overall model performance, commonly reported subgroup and domain-level differences mostly disappear under cluster-aware analysis.

Authors' abstract (original)

Aim: In large language model (LLM) examination studies, models answer the same items repeatedly, so observations are clustered and conventional intervals overstate precision. We aimed to measure the accuracy of seven LLMs on a subspecialty-level anaesthesiology question bank and to determine which between-model and subgroup differences remain distinguishable under a cluster-respecting analysis.Materials and Methods: A 120-item, five-option multiple-choice bank in the style of the Anaesthesiology and Reanimation Subspecialty Examination, written by the authors and reviewed against current guidelines and textbooks, was administered to seven LLMs in five runs at temperature 0, with option order re-randomised per run (4,200 responses from 120 unique items). Accuracy is reported with cluster bootstrap 95% confidence intervals (CIs) resampling items (4,000 replicates) and naive Wilson intervals for comparison; subgroup analyses are exploratory.Results: Pooled accuracy was 80.1% (cluster-robust 95% CI 75.8-83.9; naive Wilson 78.8-81.3). Between-model differences were large and robust, spanning 60.8% (54.0-67.5) to 94.0% (90.3-97.0) with non-overlapping extremes. Cluster-robust intervals were wider in 25 of 26 estimates (median factor 2.1, range 0.8-4.0). Most subgroup comparisons were not distinguishable, intervals overlapping substantially: vignette (75.4%, 62.5-86.5) versus non-vignette (80.9%, 76.6-84.9) and guideline-dependent (73.0%, 64.6-80.7) versus other items (80.9%, 76.6-84.9). Only contrasts between weakest and strongest domains persisted. Between-run standard deviation was 1.26-2.80 points.Conclusion: Between-model differences and run-to-run instability are robust; commonly emphasised domain and item-characteristic differences are mostly not distinguishable once clustering is respected. Benchmarks of 120 items can separate models whose accuracies differ widely but cannot reliably localise their weaknesses.

Quality & Relevance Evaluation Breakdown

Novelty

6.0/10

Methodological Rigor

7.5/10

Engineering Countermeasures

5.5/10

Consensus Challenge

7.5/10

Topic Relevance

8.0/10

Relevance for Educators & Research

Provides a quantitative methodology (cluster bootstrap) for evaluating LLM responses to test items while accounting for item-level clustering.
Demonstrates the psychometric fragility of using uncalibrated small-item benchmarks to make claims about specific AI cognitive sub-skills or content domains.
Highlights option-position bias and decoding instability as key sources of measurement error when LLMs are evaluated as examinees.

Methodology and Dataset

A text-only question bank of 120 five-option multiple-choice items written in the style of the Anaesthesiology and Reanimation Subspecialty Examination (anchored to national curricula and guidelines) was administered to seven openly accessible LLMs (e.g., Qwen and Llama lineage variants). Each model completed 5 independent runs at temperature 0 with option order deterministically re-randomized per run, yielding 4,200 total responses (7 models x 5 runs x 120 unique items). Primary accuracy statistics used a cluster bootstrap resampling items across 4,000 replicates.

Why Worth Reading

This paper addresses a fundamental statistical flaw in how large language models are evaluated on educational and professional certification exams. By demonstrating that naive confidence intervals underestimate error margins by more than 50%, the authors provide psychometricians and assessment researchers with a crucial methodological framework (cluster-bootstrap resampling) for valid LLM benchmark analysis.

Key Claims & Verified Quotes

Subgroup performance differences and domain-specific deficits reported in small LLM benchmarks are mostly statistically indistinguishable once item clustering is taken into account.

Quote: “In contrast, the domain and item-characteristic differences that such studies conventionally emphasise were, with few exceptions, not distinguishable once the 120 unique items were treated as the clusters they are.” — 10.16899/jcm.2008162, p. 7, para. 5 (exact)

LLM examination performance displays run-to-run variability even when evaluated at temperature zero with deterministic inference settings.

Quote: “Differences between models were large and robust to an analysis respecting the clustered design, as was run-to-run instability of up to 2.80 percentage points at temperature 0.” — 10.16899/jcm.2008162, p. 7, para. 5 (exact)

Limitations & Study Design Flags

Single detector: not mentioned in the text
No adversarial testing: not mentioned in the text
Monolingual: not mentioned in the text

The question bank contained only 120 unique items, resulting in very small item counts (3 to 25 items) per subspecialty domain.
Item classification into content domains and binary characteristics was performed by a single specialist without independent duplication or inter-rater reliability verification.
The items lacked prior psychometric calibration on human examinees, preventing direct equating with official board examination difficulty.
The model sample was limited to openly accessible API models, predominantly from related open-weights lineages, excluding top-tier commercial proprietary models.

From the article text (quote verified)

supports LLM accuracy is sensitive to the permutation order of multiple-choice answer options. «Because option order was re-randomised per run, part of the spread reflects the known sensitivity of LLMs to option position», p. 6, para 6
supports Scores on published licensing exams may be artificially inflated due to training set data contamination. «Scores on public examinations may also be inflated by exposure to the items during training», p. 2, para 2

Model inference (gaps & contradictions)

[method / unit of analysis mismatch]: Standard exam benchmarking treats individual LLM response instances as independent standard Bernoulli trials, ignoring the item-level intra-cluster correlation across repeated runs and models.

#psychometrics #cluster_bootstrap #confidence_intervals #LLM_evaluation #medical_education #item_response_data #measurement_error