💬 NLP

ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

ResidencyRL introduces a reinforcement learning framework that trains clinical AI agents through simulated multi-turn patient encounters with adversarial LLMs and structured rewards, significantly improving diagnostic accuracy, safety, and communication skills while demonstrating robust generalization across diverse clinical benchmarks.

Original authors: Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel, Anil Palepu, Chunjong Park, Vahid Balazadeh, Min Woo Sun, Marius Guerard, Justin Chen, Dave Steiner, Vikram Dhillon, Ibrahi

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel, Anil Palepu, Chunjong Park, Vahid Balazadeh, Min Woo Sun, Marius Guerard, Justin Chen, Dave Steiner, Vikram Dhillon, Ibrahim Azar, Akhil Mehta, Nicholas Spetsieris, Shilpan Shah, Maen Abdelrahim, Amit Dahiya, Yun Liu, Katherine Chou, Yossi Matias, Avinatan Hassidim, Dale R. Webster, Quoc V. Le, Raia Hadsell, Joelle Barral, Carey Radebaugh, Aleksandra Faust, Shekoofeh Azizi, Mike Schaekermann, Po-Hsuan Cameron Chen, Tao Tu, David Racz, Lin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where learning to be a doctor isn't just about memorizing a giant library of medical facts, but about actually doing the job. In the real world, medical students start with textbooks, but they only become true experts after years of "residency"—a grueling apprenticeship where they talk to thousands of real patients, make mistakes, get feedback, and learn how to handle the messy, unpredictable nature of human illness. This is the corner of science known as Artificial Intelligence (AI) in healthcare. For a long time, AI was like a brilliant student who could ace every written test but froze when asked to talk to a real person. The key concepts here are Large Language Models (LLMs), which are super-smart computer programs trained on vast amounts of text, and Reinforcement Learning (RL), a training method where an AI learns by trying things, getting rewards for good moves, and penalties for bad ones, much like a dog learning tricks or a gamer mastering a video game. The big question everyone is asking is: Can we teach an AI to not just know medicine, but to practice it safely and effectively, just like a human resident does?

Enter ResidencyRL, a new experiment by a team of researchers who decided to build a digital "residency program" for an AI doctor. Instead of just feeding the AI more medical textbooks, they created a simulated hospital where the AI could practice talking to virtual patients. These weren't just simple chatbots; the virtual patients were designed to be tricky, sometimes hiding important details, acting confused, or even trying to trick the AI into making a mistake. The AI had to navigate long conversations (up to 60 turns!), ask the right questions, figure out what was wrong, and decide on a treatment plan, all while being graded by a strict digital supervisor.

The results of this digital residency were surprisingly promising. After training, the AI didn't just get better at the specific practice cases; it became a more thorough and careful doctor overall. In tests where the virtual patients were being particularly difficult or deceptive, the trained AI improved its diagnostic accuracy by 7.0% (jumping from 81.0% to 88.0%). More importantly, it stopped making a dangerous mistake called "premature closure"—where a doctor guesses the answer too quickly and stops looking for clues. The trained AI missed fewer critical warning signs, dropping its "missed red flag" rate by 31%. When real human doctors (who didn't know which AI was which) compared the trained AI against the untrained version, they preferred the trained one in 87.6% of the cases, praising it for gathering more complete information and creating safer, more sensible treatment plans.

The researchers found that these skills were like a universal toolkit. Even when they tested the AI on medical cases it had never seen before—like complex cancer scenarios or multi-visit chronic disease management—it still performed better than the untrained version. It suggests that the AI learned how to think like a doctor: how to be curious, how to dig deeper when things don't add up, and how to stay calm under pressure. However, the authors are careful to note that this is all happening in a simulation. While the AI is getting much better at the process of clinical reasoning, the team emphasizes that we still need to test it with real patients in the real world to be sure it can save lives without causing harm. For now, ResidencyRL shows that if you give an AI a chance to "practice" in a safe, simulated environment, it can learn to be a much more competent and reliable medical partner.

Technical Summary: ResidencyRL

Problem Statement

While Large Language Models (LLMs) have achieved expert-level performance on static medical benchmarks (e.g., MedQA-USMLE) and single-turn diagnostic tasks, they struggle with the dynamic, sequential nature of clinical reasoning. Real-world clinical practice requires physicians to navigate uncertainty, elicit history through multi-turn dialogue, refine diagnostic hypotheses, and manage patients over time. Current AI systems often suffer from "premature closure"—anchoring on an initial hypothesis and terminating information gathering too early—and lack robustness against adversarial patient behaviors (e.g., withholding information, minimizing symptoms). Existing reinforcement learning (RL) approaches for clinical dialogue have been limited to short horizons (≤12 turns), restricted action spaces (dialogue-only or isolated tool calls), or single-turn optimization, failing to capture the full arc of a clinical encounter which averages ~21 turns and requires complex decision-making.

Methodology

The authors introduce ResidencyRL, an online multi-turn reinforcement learning framework designed to train clinical AI agents through simulated, high-fidelity patient encounters. The system is built on four core components:

1. Scenario Generation Pipeline

To ensure clinical rigor and diversity, the authors developed a dynamic, multi-stage generative pipeline:

Demographic & Behavioral Sampling: Profiles are sampled from US demographic distributions (age, ethnicity, location) and behavioral traits (Big Five personality), which dictate patient communication styles (e.g., health literacy, symptom amplification).
Condition-Targeted Generation: An LLM (Gemini 3.1 Pro) generates clinical scenarios grounded in the DDXPlus evidence corpus to ensure validated pathophysiological associations. Scenarios range from straightforward textbook cases to complex, polypharmacy-heavy cases.
Quality Verification & Deduplication: An independent LLM judge verifies clinical coherence, and TF-IDF vectorization ensures scenario diversity.
Specialized Extension Packs:
- Targeted History-Taking: Scenarios where critical information (e.g., substance use, specific drug names) is withheld unless the agent asks specific, targeted questions.
- Adversarial Safety: Scenarios designed to test robustness against minimized red flags, atypical presentations, and patient manipulation.

2. Simulated Environment

The agent interacts with a Patient Simulator (based on Gemini 3.5 Flash) conditioned on the generated scenarios. The simulator mimics real-world patient dynamics, including:

Information Asymmetry: Patients disclose reactive background details only when directly asked.
Behavioral Adaptation: Vocabulary and disclosure levels adapt to the patient's health literacy and personality.
Adversarial Specialization: In safety scenarios, patients actively minimize symptoms or resist recommendations.

The agent utilizes a Documentation API to perform structured actions beyond dialogue, including submitting differential diagnoses, management plans, urgency classifications, and SOAP notes. Episodes are capped at 60 dialogue turns and 8 tool calls.

3. Structured Reward and Auto-Grading

Upon episode completion, an LLM judge (Gemini 3.1 Pro) evaluates the encounter using a hierarchical rubric. The total reward $R$ is formulated as:
$R = R_{primary} - R_{penalty}$

Primary Reward ( $R_{primary}$ ): Sums six clinical dimensions (Diagnosis, Management, Intake, Communication, Documentation, Style) scored on Likert scales and normalized to [0, 3].
Penalty ( $R_{penalty}$ ): Accumulates penalties for critical safety flags (e.g., hallucinations, contraindicated actions, missed red flags) and system errors (e.g., excessive turn length).

4. Reinforcement Learning Algorithm

The policy is optimized using Group Relative Policy Optimization (GRPO). The agent is initialized from Gemini 3.5 Flash and trained on a mixture of ~57,000 scenarios (83% routine telehealth, 15% history-taking, 2% adversarial). The algorithm estimates advantages from groups of parallel trajectories without a dedicated value function, allowing the agent to learn sequential decision-making strategies through exploration.

Key Contributions

Long-Horizon Multi-Turn RL for Clinical Reasoning: ResidencyRL demonstrates that training clinical AI agents through simulated, multi-turn encounters (up to 68 actions) significantly improves sequential reasoning, management quality, and safety compared to base models.
Cross-Specialty and Cross-Environment Generalization: The procedural competencies learned in simulation (thorough history-taking, safety adherence) generalize to out-of-domain settings, including unseen specialties (oncology), longitudinal multi-visit care (AMIE Mx), and external benchmarks (AgentClinic, CRAFT-MD).
Mitigation of Premature Closure and Safety Risks: The training paradigm actively reduces dangerous diagnostic pitfalls, specifically lowering rates of missed critical questions and missed red flags under adversarial conditions.

Results

The ResidencyRL-trained agent was evaluated across multiple benchmarks against the base Gemini 3.5 Flash model:

In-Domain Evaluation:
- Diagnostic Accuracy: Improved from 81.0% to 88.0% under adversarial conditions (+7.0 percentage points).
- Management Quality: Increased from 3.98 to 4.52 (on a 1–5 Likert scale).
- Safety: Missed red flag rates decreased by 31% (from 45.5% to 31.5%) in adversarial scenarios.
Out-of-Domain Benchmarks:
- AMIE Mx (Multi-Visit): Outperformed the base model across all six clinical axes, with significant gains in management reasoning (+8.34%) and patient communication (+8.50%).
- Specialist Oncology: On 300 expert-curated oncology cases, the trained agent showed significant improvements in completeness, clinical accuracy, and actionability compared to the base model.
- AgentClinic & CRAFT-MD: Showed consistent directional improvements in diagnostic accuracy and information gathering, particularly in free-response formats requiring active reasoning.
Human Expert Evaluation:
- In blinded side-by-side comparisons by 97 board-certified clinicians, the ResidencyRL-trained agent was preferred in 87.6% of cases for overall clinical impression.
- The agent was preferred in 90.7% of cases for completeness of information gathering and 75.3% for management plan appropriateness.
- Clinicians noted the trained model's ability to persist in probing for hidden information (e.g., undisclosed substance use) without hallucinating missing data.

Significance and Claims

The paper claims that clinical mastery for AI, like for physicians, is developed through practice. ResidencyRL validates that sequential clinical decision-making can be effectively learned through multi-turn RL in simulation, yielding robust and generalizable capabilities.

The authors emphasize that these improvements are procedural (how to gather information, when to probe, how to manage uncertainty) rather than merely domain-specific knowledge, allowing transfer to unseen specialties and complex workflows. However, the paper maintains a modest stance regarding clinical utility:

Simulation vs. Reality: The authors acknowledge a fidelity gap between simulated and real patients. While clinicians judged the simulations as realistic, the training is confined to text-based telehealth and lacks multimodal inputs (physical exams, imaging) and real-world outcome feedback.
Need for Prospective Validation: The paper explicitly states that prospective validation with real-world workflows and real patients is necessary to confirm that these simulation-trained competencies translate to patient-level efficacy and safety.
Safety: The system is not open-sourced due to safety implications, and the authors stress the need for responsible innovation and further validation with regulators and healthcare providers.

In conclusion, ResidencyRL represents a significant step toward bridging the gap between static medical knowledge and dynamic clinical practice, demonstrating that reinforcement learning in simulated environments can produce AI agents that are safer, more thorough, and more aligned with expert clinical reasoning.