quasi_experiment

Assessment of Clinical Competence Using AI-Standardized Clinical Examination

Authors: Antonio Lopez, Ludovic Gabellier, Clement Peytavi, Zeno Loi, Emma Lavigne, Catherine Plotton, Vincent Dupont, Léon Jahdauti, P. De Vries, Philippe Thuillier, Maxence Compagnat, Pascale Vergne‐Salle, Mathieu Kuchenbuch, Eva Feigerlová, P. Berthelot, Nathalie Bednarek, Béatrice Cochener, Pierre-Yves Robert, Marc Braun, Denis Morin, Philippe Deruelle, Isabelle Laffont, Yves‐Marie Pers, Kévin Yauy
Venue / Source: medRxiv (2026)

Researchers tested a new AI-based clinical exam where students interacted with AI patients and AI raters scored their performance.

Executive Summary (TL;DR)

Wording

Researchers tested a new AI-based clinical exam where students interacted with AI patients and AI raters scored their performance. They found that students' scores on this AI exam correlated reasonably well with their actual scores on a high-stakes national exam. This suggests that AI could eventually help scale standardized assessments, though human review is still needed.

This study evaluated an AI-Standardized Clinical Examination (ASCE) where GPT-4o simulated patient encounters and GPT-4.1 performed automated scoring. Among 588 medical students, ASCE performance showed a moderate correlation (ρ=0.57) with national high-stakes OSCE scores. Automated AI scores achieved comparable reliability to expert human ratings.

Authors' abstract (original)

BACKGROUND: As artificial intelligence (AI) enters clinical practice, concerns about deskilling and never-skilling make the independent assessment of clinical competence increasingly important. The AI-Standardized Clinical Examination (ASCE) uses AI to simulate and score standardized clinical encounters, potentially overcoming some of the logistical and standardization constraints of conventional objective structured clinical examinations (OSCEs). Whether it provides valid and reliable assessment at scale is unknown. METHODS: In this prospective multicenter cohort study, final-year medical students from six French medical schools completed a 10-station ASCE six weeks before the national OSCE session. GPT-4o simulated standardized clinical partners; transcripts were subsequently scored by GPT-4.1 against faculty-defined rubrics. The primary outcome was the correlation between ASCE and national OSCE scores. Secondary outcomes were inter-rater reliability of expert transcript scoring, alignment of ASCE scores with expert ratings, and internal consistency. RESULTS: Among 948 enrolled students, 588 had paired ASCE and national OSCE scores. ASCE scores were correlated with national OSCE performance (Spearman's ρ, 0.57; 95% CI, 0.51 to 0.63), with less than 1% of residual variance attributable to center (intraclass correlation coefficient, 0.01). Eight experts scored 20 randomly selected ASCE transcripts; Krippendorff's α was 0.77 overall, 0.80 for clinical skills, and 0.70 for communication and attitude skills. Adding GPT-4.1 as an evaluator reduced α by 0.02, and its scores did not differ significantly from those of individual experts. Internal consistency was 0.67 across 10 stations and was projected to approach 0.80 with 20 stations. CONCLUSIONS: Scores from an examination delivered and scored by AI were associated with performance on a national high-stakes OSCE, with minimal center-level variation, tentative agreement among expert raters, and automated scores that aligned with expert ratings. These results support AI delivery and scoring of clinical examinations, with expert review of automated scores.

Quality & Relevance Evaluation Breakdown

Novelty
8.0/10
Methodological Rigor
8.0/10
Engineering Countermeasures
5.0/10
Consensus Challenge
5.0/10
Topic Relevance
8.0/10

Relevance for Educators & Research

  • Provides a framework for validating AI-based assessments against traditional high-stakes benchmarks.
  • Offers a template for evaluating automated scoring consistency (rater drift/agreement) using Krippendorff’s α.
  • Demonstrates the use of Spearman-Brown prophecy to determine required station counts for reliability.

Methodology and Dataset

Prospective multicenter cohort study with 948 enrolled sixth-year medical students from six French universities. 588 students had paired ASCE and national OSCE scores. GPT-4o was used to simulate standardized patients, and GPT-4.1 to score transcripts against faculty-defined rubrics.

Why Worth Reading

This paper provides one of the first rigorous, multicenter evaluations of an end-to-end AI-delivered and AI-scored clinical examination benchmarked against an independent high-stakes national assessment, offering a concrete model for scalable competency-based assessment.

Key Claims & Verified Quotes

AI scoring achieves comparable reliability to expert human raters for clinical skills assessment.
Quote: “Adding GPT-4.1 reduced α by 0.02 across all three domains; the AI-inclusive α fell within the 95% confidence interval of the experts only estimate.” — 10.64898/2026.10.01.26364500, p. 15, para. 14 (exact)

Limitations & Study Design Flags

  • No adversarial testing: not mentioned in the text
  • The ASCE does not assess physical examination or procedural skills.
  • Internal consistency (Cronbach’s α=0.67) remains below conventional high-stakes thresholds.
  • Automated scoring agreement was based on a small sample of 20 transcripts with missing expert ratings.
  • The study relied on proprietary, version-dependent AI models susceptible to silent updates or deprecation.
From the article text (quote verified)
  • supports Automated scoring of OSCE notes shows promising correlation with expert raters. «AI-based scoring of OSCE post-encounter notes and written clinical reports has also shown promising correlation with expert raters [11,12].», p. 8, para 20
Model inference (gaps & contradictions)
  • [sample / selection bias]: Participants were more likely to be women and already training on digital platforms, which may limit generalizability of the performance correlation.