Technical Summary: Prospective Deployment of Multimodal AI Grading for Medical Student OSCEs

Problem Statement

The Objective Structured Clinical Examination (OSCE) is the gold standard for assessing clinical competency in medical education, evaluating history-taking, physical examination, communication, and reasoning. However, traditional OSCE administration is resource-intensive, requiring trained evaluators, manual rubric scoring, and substantial human time. At the authors' institution (UT Southwestern), manual grading of post-encounter documentation alone required approximately 1,120 hours and $56,000 annually. These constraints limit the frequency of assessments, delay feedback to learners (often by months), and hinder the expansion of formative assessment. While Large Language Models (LLMs) have shown promise in grading written responses, comprehensive OSCE assessment requires evaluating text, audio, and video simultaneously. Existing off-the-shelf multimodal models have shown limitations in nuanced clinical assessment, and evidence from large-scale, prospective educational deployments remains scarce.

Methodology

The authors report the first prospective deployment of MAPLES (Multimodal Assessment Pipeline for Learning Encounter Scoring), a rubric-driven, zero-shot AI system, during the Fall 2025 OSCE administration at UT Southwestern Medical Center.

System Architecture and Workflow

• Data Ingestion: The system ingested three synchronized data streams from 222 second-year medical students across five stations: written clinical notes, encounter audio, and synchronized three-camera video (patient-facing, physician-facing, and overhead).
• Grading Logic: MAPLES utilized a zero-shot approach. Faculty-authored rubrics (structured Excel files) were uploaded, and the system dynamically constructed prompts for each item. No task-specific fine-tuning or few-shot examples were provided.
  • Text: Notes were processed as plain text.
  • Audio: Audio was processed directly without intermediate transcription to evaluate verbal content, tone, and rapport.
  • Video: An automated preprocessing step (using zero-shot segmentation) isolated physical examination segments before grading.
• Model Configuration: The system employed Gemini 2.5 Pro (Google DeepMind) for inference with temperature 0 and top-p 1, utilizing structured-output constraints to ensure scores adhered to the rubric schema. Gemini 2.5 Flash was used for video segmentation.
• Human-in-the-Loop Workflow:
  1. First-Pass AI Scoring: AI graded all 72,907 retained item-level scores.
  2. Routing: Learners falling in the bottom 5% (notes) or bottom 10% (audio/video) were routed for independent human review.
  3. Blinded SPE Review: Standardized Patient Evaluators (SPEs) independently rescored routed items, blinded to AI scores.
  4. Adjudication: Items where AI and SPE scores differed beyond a predefined tolerance (exact match for binary; $\le$1 category difference for ordinal) were escalated to physician experts for final adjudication. Physicians reviewed both scores and source evidence.

Study Design

• Cohort: 222 students, 10 station forms, 330–333 assessment items per student.
• Review Set: 28 unique learners were routed for review (12 for notes, 23 for audio/video, 7 for both).
• Adjudication Set: 616 genuine disagreements (excluding data/rubric errors) were reviewed by physicians.
• Governance: A multidisciplinary supervisory committee oversaw model selection, thresholds, and deployment cycles, ensuring iterative refinement.

Key Results

Agreement and Adjudication

• AI–SPE Agreement: In the routed low-scoring review set, tolerant agreement (allowing one-category difference for ordinal items) was high: 85.7% for notes, 89.2% for audio, and 92.4% for video. Exact agreement was lower (72.0% overall), driven by the complexity of audio/video scoring.
• Physician Adjudication: Among 616 escalated disagreements, physician experts matched the AI score 76.0% of the time, the SPE score 19.0% of the time, and neither 5.0%. This preference for AI held across all modalities, though it was highest for notes (81.6%) and lowest for video (51.5%).
• Direction of Disagreement: Physicians were more likely to side with the AI whether the AI scored higher or lower than the SPE, suggesting the AI was not simply inflating scores.

Error Analysis

• SPE Errors: The primary factors for SPE errors were "Evidence oversight" (35.1%) and "Clinical knowledge" (24.4%), often failing to recognize semantic equivalence in terminology.
• AI Errors: The primary factors for AI errors were "Over permissive scoring" (37.8%) and "Wrong documentation placement" (25.2%), where AI credited correct answers written in the wrong note section. A small fraction (9.4%) involved "Fabricated evidence" (hallucinations).
• Rubric Quality: "Rubrics lack of specificity" was a frequent contributor to errors for both humans and AI, highlighting the need for precise rubric design.

Operational Efficiency

• Workload Reduction: The AI-assisted workflow required 5,616 human scoring passes compared to 72,907 in a counterfactual single-pass manual workflow. This represents a 92.3% reduction in human scoring effort.
• Turnaround Time: The time from examination to grade reports was reduced from several months to under two weeks.
• Cost: The marginal inference cost was estimated at $1.73 per encounter ($0.12 notes, $0.28 audio, $1.31 video), excluding fixed development costs.

Significance and Claims

The paper claims to present the first prospective deployment of an integrated multimodal AI system for OSCE grading in undergraduate medical education. The authors emphasize that the primary contribution is not the removal of human judgment, but the reallocation of human effort.

• Operational Feasibility: The study demonstrates that multimodal AI can be embedded into routine OSCE operations, handling first-pass scoring for the entire cohort while concentrating human review on the "low-scoring tail" where remediation is most critical.
• Expert Alignment: The finding that physicians preferentially endorsed AI scores over SPE scores in adjudicated disagreements suggests that the AI system captures assessment nuances that align closely with expert clinical judgment, potentially outperforming traditional SPE evaluation in consistency.
• Systemic Improvement: The deployment exposed systemic issues in the status quo, including human grader reliability, rubric quality, and process inefficiencies. The authors argue that AI frees educators to focus on "what skills to evaluate" (rubric and case design) rather than "how to grade."
• Scalability: The zero-shot, rubric-driven design allows for the transfer of grading skills to new stations or rubric revisions without retraining models, supporting the expansion of assessment frequency and scope.

The authors maintain a modest tone regarding generalizability, noting that results are specific to one institution's infrastructure and rubrics. They acknowledge limitations, including the lack of blinded adjudication for the full cohort, the concentration of analysis in the low-scoring tail, and the need for further validation regarding demographic fairness and hallucination rates. The work is framed as a foundational step toward educator-managed, AI-augmented assessment systems.