Large Language Model Accuracy on Subspecialty Anaesthesiology Examination Questions
Muhammed Nezih KOÇ, Sait Ramazan GÜLBAY · Journal of Contemporary Medicine (2026) · doi:10.16899/jcm.2008162 · gemini-3.6-flash
TL;DR
This study tested seven AI models on a 120-question medical examination, repeating each question five times to check for consistent performance. The researchers found that conventional statistical methods artificially inflate confidence in AI scores by ignoring that repeating the same questions creates repeated measures rather than independent data points. When properly analyzed, overall model accuracy differences remain real, but fine-grained claims about specific domain strengths and weaknesses mostly collapse.
Method and sample: A text-only question bank of 120 five-option multiple-choice items written in the style of the Anaesthesiology and Reanimation Subspecialty Examination (anchored to national curricula and guidelines) was administered to seven openly accessible LLMs (e.g., Qwen and Llama lineage variants). Each model completed 5 independent runs at temperature 0 with option order deterministically re-randomized per run, yielding 4,200 total responses (7 models x 5 runs x 120 unique items). Primary accuracy statistics used a cluster bootstrap resampling items across 4,000 replicates.
#psychometrics#cluster_bootstrap#confidence_intervals#LLM_evaluation#medical_education#item_response_data#measurement_error