Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Maxime Griot, Jean M. Vanderdonckt, Demet Yüksel, Coralie Hemptinne · Proceedings of the conference - Association for Computational Linguistics. Meeting (2025) · doi:10.18653/v1/2025.acl-long.266 · gemini-3.7-flash
TL;DR
Multiple-choice tests are commonly used to show that artificial intelligence models possess medical knowledge. To see if models truly understand medicine or just exploit test shortcuts, researchers tested them on exams about an invented organ that does not exist. While human doctors scored at random guessing levels, AI models passed easily, proving that standard multiple-choice questions reward test-taking patterns rather than genuine knowledge.
Method and sample: Synthetic benchmark generation pipeline creating fictional textbooks (~35,000 words each) and 488 multiple-choice questions (264 via GPT-4 Turbo, 224 via Claude 3.5 Sonnet) translated across English and French. Models tested: GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Yi-1.5 (9B, 34B), Mistral-7B-v0.1, Mixtral-8x7B-v0.1, LLaMA-3 (8B, 70B), Qwen1.5 (7B, 32B, 110B), DeepSeek-V3-0324, and fine-tuned models InternistAI-7B and Meerkat-7B. Human baseline consisted of 2 licensed physicians completing 100 English items.
#Construct Validity#Multiple-Choice Questions#Large Language Models#Medical Education#Test-Taking Heuristics#Adversarial Evaluation