MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skills
Authors: Zonghai Yao, Zihao Zhang, Chaolong Tang, Xingyu Bian, Youxia Zhao, Zhichao Yang, Junda Wang, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, Hong Yu
Venue / Source: Conference of the European Chapter of the Association for Computational Linguistics (2026)
High scores on multiple-choice medical exams do not guarantee practical clinical skills: top language models average under sixty-three percent on clinical encounter benchmarks.
Key takeaways
- Multiple-choice test performance overestimates LLM practical capabilities in interactive and structured skill evaluations.
- GPT-4 functions effectively as an automated examiner for structured clinical tasks, correlating strongly with human expert ratings.
- Domain-specific fine-tuning can cause catastrophic forgetting of general instruction-following abilities in complex performance rubrics.
- Human preference optimization techniques improve an open-source model's ability to follow complex evaluation rubrics without requiring domain retraining.
Executive Summary (TL;DR)
Authors' abstract (original)
Artificial intelligence (AI) and large language models (LLMs) in healthcare require advanced clinical skills (CS), yet current benchmarks fail to evaluate these comprehensively. We introduce MedQA-CS, an AI-SCE framework inspired by medical education's Objective Structured Clinical Examinations (OSCEs), to address this gap. MedQA-CS evaluates LLMs through two instruction-following tasks-LLM-as-medical-student and LLM-as-CS-examiner-designed to reflect real clinical scenarios. Our contributions include developing MedQA-CS, a comprehensive evaluation framework with publicly available data and expert annotations, and providing the quantitative and qualitative assessment of LLMs as reliable judges in CS evaluation. Our experiments show that MedQA-CS is a more challenging benchmark for evaluating clinical skills than traditional multiple-choice QA benchmarks (e.g., MedQA). Combined with existing benchmarks, MedQA-CS enables a more comprehensive evaluation of LLMs' clinical capabilities for both open- and closed-source LLMs .
Quality & Relevance Evaluation Breakdown
Relevance for Educators & Research
- Demonstrates how to operationalize Miller's Pyramid of competence into structured AI assessment benchmarks.
- Provides statistical validity proof (Pearson r, Kendall tau/W) for using LLM-as-Judge in expert domain performance scoring.
- Illustrates item parameter/rubric drift where knowledge-based benchmark scores do not predict open-ended instruction-following capabilities.
Methodology and Dataset
The dataset contains 1,667 instruction-following data points derived from 44 USMLE Step 2 Clinical Skills cases, transformed via automated paraphrasing and verified by licensed medical experts. Models evaluated include MedStuLLM test-takers (GPT-4, GPT-4o, GPT-3.5-turbo, Claude-3 series, Qwen2 series, GLM4-9b, LLaMA2/3 variants, Mistral/Mixtral variants) and MedExamLLM automated raters (GPT-4, GPT-4o, Claude-3 series, and a distilled LLaMA-3.1-8B judge model). Baselines included standard automated generation metrics (ROUGE, METEOR, BERTScore, UMLS-F1). Human validation involved three domain experts annotating model outputs.
Why Worth Reading
This paper bridges medical OSCE frameworks with computational assessment design by establishing how LLMs can be evaluated as both examinees and automated raters. It provides concrete empirical evidence on the validity gap between multiple-choice testing and open-ended practical performance, making it highly valuable for researchers studying performance-based assessment, LLM-as-judge reliability, and prompt rubric alignment.
Key Claims & Verified Quotes
Limitations & Study Design Flags
- Dataset size is constrained to 44 USMLE Step 2 CS cases, which underrepresents specialized medical domains, rare diseases, pediatrics, and psychiatry.
- Evaluation is restricted to text-only inputs, excluding multimodal clinical data such as patient speech audio, diagnostic imaging, and physiological waveforms.
- The framework does not evaluate treatment plan formulation due to original USMLE Step 2 CS scope constraints.
- All benchmark data are in English, limiting generalizability to non-English clinical environments.
- contradicts Multiple-choice question benchmarks primarily measure factual knowledge acquisition rather than practical clinical competence. «MCQ benchmarks primarily focus on the "knows" and "knows how" levels of Miller’s Pyramid, neglecting the practical skills essential in medical education.», p. 2, para 5
- [metric / definition]: Standard lexical overlap metrics (ROUGE, UMLS-F1) showed weak or negative correlation with human expert evaluations on open-ended diagnostic and communication tasks, whereas prompt-guided LLM judges aligned closely with human scoring.