Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Authors: Maxime Griot, Jean M. Vanderdonckt, Demet Yüksel, Coralie Hemptinne
Venue / Source: Proceedings of the conference - Association for Computational Linguistics. Meeting (2025)
Multiple-choice tests are commonly used to show that artificial intelligence models possess medical knowledge.
Key takeaways
- Language models correctly answer multiple-choice questions about fictional medical biology by relying on superficial linguistic patterns and test-taking heuristics.
- Physicians score at chance level when encountering unfamiliar fictional medical content, whereas language models substantially exceed chance performance.
- Presenting models with only multiple-choice options without question stems still yields accuracy well above random guessing.
- Evaluators cannot rely on standard multiple-choice formats as valid evidence of clinical comprehension or professional competence.
Executive Summary (TL;DR)
Authors' abstract (original)
Large Language Models (LLMs) such as Chat-GPT demonstrate significant potential in the medical domain and are often evaluated using multiple-choice questions (MCQs) modeled on exams like the USMLE.However, such benchmarks may overestimate true clinical understanding by rewarding pattern recognition and test-taking heuristics.To investigate this, we created a fictional medical benchmark centered on an imaginary organ, the Glianorex, allowing us to separate memorized knowledge from reasoning ability.We generated textbooks and MCQs in English and French using leading LLMs, then evaluated proprietary, open-source, and domain-specific models in a zero-shot setting.Despite the fictional content, models achieved an average score of 64%, while physicians scored only 27%.Fine-tuned medical models outperformed base models in English but not in French.Ablation and interpretability analyses revealed that models frequently relied on shallow cues, test-taking strategies, and hallucinated reasoning to identify the correct choice.These results suggest that standard MCQ-based evaluations may not effectively measure clinical reasoning and highlight the need for more robust, clinically meaningful assessment methods for LLMs.
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Directly speaks to construct validity and score meaning (Kane and Messick frameworks) when automated systems take standardized educational tests.
- Illustrates differential item functioning mechanisms where LLMs exploit lexical heuristics that human examinees cannot use.
- Demonstrates how synthetic and adversarial benchmarks can isolate genuine reasoning from rote statistical association in educational assessments.
Methodology and Dataset
Synthetic benchmark generation pipeline creating fictional textbooks (~35,000 words each) and 488 multiple-choice questions (264 via GPT-4 Turbo, 224 via Claude 3.5 Sonnet) translated across English and French. Models tested: GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Yi-1.5 (9B, 34B), Mistral-7B-v0.1, Mixtral-8x7B-v0.1, LLaMA-3 (8B, 70B), Qwen1.5 (7B, 32B, 110B), DeepSeek-V3-0324, and fine-tuned models InternistAI-7B and Meerkat-7B. Human baseline consisted of 2 licensed physicians completing 100 English items.
Why Worth Reading
The paper provides an empirical demonstration of the construct invalidity of multiple-choice tests when applied to large language models, showing that high scores can be achieved entirely through statistical artifact exploitation without domain comprehension.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Single detector: not mentioned in the text
- Monolingual: not found
- Textbook sections were generated independently, creating subtle internal contradictions and omissions across chapters.
- The initial grounding seed data was authored by a single physician, potentially introducing individual stylistic biases.
- The human comparison baseline is limited to two physicians evaluating a subset of 100 questions.
- supports Large language models can answer multiple-choice questions without the question stem by exploiting surface cues in the answer options. «This intermediate accuracy level supports the hypothesis proposed by Balepur et al. (2024), suggesting that large language models employ meta-strategies such as question inference and surface-level shortcuts.», p. 7, para 8
- [definition / construct underrepresentation]: Standard multiple-choice evaluations define medical reasoning via item selection accuracy, but performance is heavily confounded by format-specific surface artifacts.
- [method / methodological gap]: Evaluating AI models using human licensing exams fails to control for pre-existing memorized text and stylistic heuristics embedded in multiple-choice formats.