A French OSCE Dialogue Dataset and Controllable Virtual Patient System for Clinical Training

Technical Summary: A French OSCE Dialogue Dataset and Controllable Virtual Patient System for Clinical Training

Problem Statement
Objective Structured Clinical Examinations (OSCEs) are the standard for assessing medical students' clinical reasoning and communication skills in France. These exams rely on interactions with human Standardized Patients (SPs). However, training is constrained by the scarcity of human SPs and evaluators, leading to scalability and cost issues. While Virtual Patient (VP) systems have been proposed, existing solutions often rely on rule-based scripts or lack the controllability required to adhere strictly to predefined clinical scenarios. Furthermore, there is a significant lack of French-language OSCE datasets and standardized, reproducible evaluation frameworks for LLM-based VP systems.

Methodology
The authors propose a comprehensive solution comprising a new dataset, a controllable generation pipeline, and a multi-level evaluation framework.

1. Dataset Construction:

   • Recorded Data: The authors collected 240 recorded and transcribed OSCE training dialogues involving 99 sixth-year medical students across 23 distinct stations (out of 192 total stations collected) covering over 10 medical specialties. The recordings were manually annotated with labels for specialty, consultation type, and objectives. Automatic transcription was performed using faster-whisper, achieving a Word Error Rate (WER) of 7.7% on a manually corrected subset.
   • Generated Data: Using the pipeline described below, 792 synthetic dialogues were generated from 11 of the recorded stations.

2. Controllable LLM-Based Pipeline:
   The system generates dialogues in two modes: interactive (student trains as a physician) and automatic (both physician and patient are LLM-generated). The architecture employs up to four distinct LLM instances for specific roles:

   • Input Grounding: The system utilizes three documents: the Patient Sheet (identity, history, symptoms), the Physician Sheet (context, objectives), and the Evaluation Sheet (criteria to be met).
   • Dialogue Scripting: In automatic mode, criteria are reordered into phases (Opening, Information collection, Assessment, Plan/Conclusion) to guide the physician generator.
   • Patient Generator: Produces responses based on the patient sheet and dialogue history. It can be augmented by a Retrieval Module to select relevant context from the patient sheet.
   • Control Loop: A Controller Module evaluates generated patient responses for fidelity against the patient sheet, assigning a score (0–10). Responses scoring below 8/10 are sent to a Correction Module to address identified inconsistencies. This forms a Reflection Loop that iteratively improves patient fidelity.

3. Evaluation Framework (LLM-as-a-Judge):
   The authors utilize GPT-4o-mini as a judge to evaluate dialogues across three dimensions:

   • Patient Simulation Quality: Measured by Information Recall (proportion of patient sheet elements mentioned) and Response Relevance.
   • Physician Performance: Measured by OSCE Evaluation (percentage of criteria addressed) and Role Adherence.
   • Linguistic Quality: Measured by Naturalness, Fluency, and Coherence.

Key Contributions

• French OSCE Dataset: The release of a dataset containing 240 recorded training dialogues and 792 generated dialogues, addressing the scarcity of French medical dialogue resources.
• Controllable Generation Pipeline: A modular system integrating retrieval-based grounding and a reflection loop (controller-corrector) to ensure patient fidelity and coherence in synthetic OSCE dialogues.
• Multi-Level Evaluation: A structured framework assessing patient simulation, student performance, and linguistic quality, validated against human annotations where possible.

Results

• Generated vs. Recorded: Generated dialogues consistently outperformed recorded interactions in patient simulation metrics (e.g., Information Recall: 52.24% for Claude-Haiku-4.5 vs. 43.58% for recorded) and linguistic quality. The authors note this gap is partly due to the variability and incomplete criteria coverage in real student interactions, as well as potential biases in LLM-as-a-Judge favoring structured text.
• Impact of Control Modules: The reflection loop consistently improved patient simulation metrics (e.g., increasing Information Recall from 52.24% to 54.29% with reflection, and 56.62% with reflection + retrieval) while maintaining stable linguistic quality. The correction process was triggered in only 3.6% of turns but improved the controller score in 79.5% of those cases.
• Robustness: The VP system remained robust to degraded physician performance (simulating students who address only 50% or 25% of criteria), maintaining high response relevance, particularly when using the retrieval and reflection pipeline.
• Human-LLM Agreement: Inter-annotator agreement among humans was weak for linguistic criteria. Agreement between human evaluators and the LLM-as-a-Judge was moderate for OSCE checklist scoring but stronger for patient-only stations without document analysis.

Significance and Claims
The paper claims to address the scarcity of French medical dialogue resources and the lack of controllability in current LLM-based VP systems. The authors posit that their modular pipeline, grounded in real OSCE data and enhanced by reflection loops, offers a viable path toward scalable, realistic clinical training tools. They emphasize that their system allows for the generation of diverse, high-fidelity dialogues and provides an automatic feedback mechanism for students.

However, the authors maintain a modest stance regarding their claims. They acknowledge that:

• The dataset covers only a subset of the collected stations (11 out of 192 used for generation).
• The evaluation relies heavily on a single LLM-as-a-Judge without extensive human validation for all metrics.
• The pipeline has not yet been tested in real training conditions with medical students, so its pedagogical effectiveness remains unevaluated.
• The system's reliance on multiple LLM calls raises concerns regarding cost and reproducibility.

Future work is identified as extending the pipeline to document-based stations, improving controllability, and validating the system in real-world training scenarios.