Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learning

This paper introduces DoctorAgent-RL, a reinforcement learning-based multi-agent framework trained on a new multi-turn medical dataset (MTMedDialog) that enables proactive, strategic questioning to achieve a 70% exact diagnostic match rate, thereby addressing limitations of static models and alleviating healthcare resource strain.

The Gist

Imagine you are trying to solve a mystery, but instead of a detective, you have a computer program. Usually, these programs act like a library book: you ask a question, and they instantly spit out an answer based on everything they've read. But in real life, a doctor doesn't work like a library book. A doctor works like a detective who asks a series of smart questions to figure out what's wrong, because patients often forget details or don't know how to describe their pain.

This paper introduces a new AI system called DoctorAgent-RL that tries to act more like that detective and less like a library book. Here is how it works, broken down simply:

1. The Problem: The "One-Shot" Mistake

Most current medical AI systems are like a student taking a test where they have to write an essay based on a single sentence. If a patient says, "My stomach hurts," the AI has to guess the diagnosis immediately.

• The Issue: Real patients are messy. They might say, "I ate too much, then I rode a bike, and now my right side hurts," but forget to mention they also have a fever. If the AI guesses too early, it's like a detective arresting someone without checking the alibi.

2. The Solution: A "Role-Playing" Training Camp

The researchers built a special training ground called DoctorAgent-RL. Instead of just reading old medical records, they created a video game-like simulation with three characters:

• The Doctor Agent: The AI student trying to learn how to diagnose.
• The Patient Agent: A smart computer character that acts like a real human. It has a hidden "medical file" (like a secret script) and only reveals symptoms if the Doctor asks the right questions. It doesn't just say everything at once; it waits to be asked.
• The Evaluator: A strict referee that watches the conversation. It gives points for asking good questions, finding the right answer, and following the rules (like asking only one question at a time).

3. The Secret Sauce: Learning by Doing (Reinforcement Learning)

The AI doesn't just memorize answers. It plays thousands of rounds of this "detective game."

• The Strategy: The AI learns that its job isn't to know the answer immediately. Its job is to master the art of asking questions.
• The Analogy: Think of it like learning to play chess. You don't just memorize the moves; you play against an opponent, lose, get feedback, and learn which moves lead to victory. The AI learns that asking "Do you have a fever?" is better than guessing "It's the flu" right away.

4. The New Dataset: "MTMedDialog"

To train this detective, the researchers couldn't use old, static chat logs because those are like transcripts of a conversation that already happened. They needed a dynamic game.

• They built a new dataset called MTMedDialog.
• The Metaphor: Imagine a "Choose Your Own Adventure" book where the story changes based on what you ask. In this dataset, the "Patient" is a living character that reacts to the Doctor's questions, revealing clues step-by-step, just like a real clinic visit.

5. The Results: Does It Work?

The team tested this new AI in two ways:

• Against Other AIs: They pitted DoctorAgent-RL against famous models (like GPT-4 and other medical AIs). The new AI won by a large margin. It asked better questions, gathered information more efficiently, and got the diagnosis right more often.
• Real People Test: They let 20 real people chat with the AI about their actual health problems.
  • The Score: The AI got the exact correct diagnosis 70% of the time.
  • The Verdict: It proved that an AI trained in a simulation can actually handle the unpredictable nature of real humans.

6. Why This Matters (According to the Paper)

The paper claims this system is a "collaborative tool."

• The Goal: It's not here to replace doctors. It's here to act as a triage assistant.
• The Benefit: By handling the initial "detective work" (asking the basic questions and narrowing down the problem), it frees up human doctors to focus on the most complex and difficult cases. It aims to fix the problem of doctors being too busy and patients getting misdiagnosed because they didn't explain their symptoms perfectly in one go.

In short: The paper shows that if you teach an AI to be a curious detective who asks smart questions step-by-step, rather than a know-it-all who guesses immediately, it can become a very helpful partner in a doctor's office.

Technical Summary

1. Problem Statement

Current Large Language Models (LLMs) face significant limitations in real-world clinical consultations:

• Single-Turn Limitations: Existing systems (e.g., MedAlpaca, BioMistral) require patients to provide a comprehensive symptom description in a single turn. This contradicts clinical reality where patients often have vague complaints or unclear chief symptoms, leading to risky or overly broad diagnoses.
• Static Learning Constraints: Traditional multi-turn dialogue models rely on static supervised learning (SFT), which merely imitates existing dialogue transcripts. They lack the ability to dynamically adjust questioning strategies based on real-time information, failing to perform genuine clinical reasoning.
• Lack of Proactive Inquiry: Even advanced multi-agent systems often rely on prompt engineering or static knowledge graphs, lacking the capability to optimize proactive inquiry strategies to handle the inherent uncertainty of patient-led interactions.
• Sim-to-Real Gap: Most evaluations are conducted on static datasets, failing to validate whether AI agents can maintain diagnostic accuracy and adaptability when interacting with real, unpredictable human patients.

2. Methodology: DoctorAgent-RL Framework

The authors propose DoctorAgent-RL, a multi-agent collaborative Reinforcement Learning (RL) framework that models medical consultation as a Markov Decision Process (MDP). The system consists of three synergistic components:

A. Core Components

1. Doctor Agent: The primary decision-maker. Its goal is not to "know the answer" immediately but to learn a strategic questioning methodology. It is initialized with a base model (Qwen2.5-7B-Instruct) and refined through RL to progressively elicit key information.
2. Patient Agent: A high-fidelity LLM-based agent that simulates realistic patient responses. It is grounded in a comprehensive, hidden medical profile and generates dynamic, contextually appropriate responses turn-by-turn, rather than following static scripts.
3. Consultation Evaluator: A neutral arbiter that provides multi-dimensional rewards to guide the Doctor Agent's policy optimization.

B. Training Strategy (Two-Stage Paradigm)

The training follows a Supervised Fine-Tuning (SFT) + Reinforcement Learning (RL) pipeline:

1. Reasoning Distillation (SFT): The model is fine-tuned on 1,000 reasoning-augmented dialogues. These dialogues include structured thought processes (hypothesis generation, evidence evaluation, differential diagnosis) generated by DeepSeek-V3 to establish a behavioral baseline.
2. Reinforcement Learning (RL): The model is optimized using Group Relative Policy Optimization (GRPO).
   • Reward Mechanism: The total reward ($R$) is a sum of three components:
     • Diagnostic Accuracy Reward: Based on F1 scores between predicted and gold-standard diagnoses/treatments.
     • Information Acquisition Efficiency Reward: Rewards effective questioning and penalizes refusals to answer.
     • Protocol Compliance Reward: Penalizes violations (e.g., asking multiple questions at once, failing to diagnose within turn limits).
   • Dynamic Turn Budget: A random dialogue length constraint (2–10 turns) is assigned per episode to mimic varying time pressures and encourage efficient information gathering.

C. Dataset: MTMedDialog

To support this framework, the authors constructed MTMedDialog, the first English multi-turn medical consultation dataset designed for dynamic simulation.

• Source: Derived from Chinese benchmarks (IMCS21, CHIP-MDCFNPC, MedDG), denoised, and translated.
• Features: Contains 8,086 training and 2,082 test samples across 8 disease categories. Unlike static transcripts, it supports dynamic symptom release, where the patient agent reveals information only in response to specific strategic questions.

3. Key Contributions

1. Paradigm Shift: Moves the core intelligence of medical AI from "knowing the answer" to "mastering the questioning methodology" for optimal diagnosis.
2. Novel Framework: Introduces a multi-agent RL framework (DoctorAgent-RL) that treats clinical reasoning as a dynamic decision-making process under uncertainty.
3. MTMedDialog Dataset: Created a high-fidelity, dynamic dataset enabling the training of agents that learn adaptive questioning strategies through interactive simulation.
4. Real-World Validation: Conducted rigorous evaluations including blinded human assessments and, crucially, prospective trials with 20 real patients, bridging the sim-to-real gap.

4. Results

• Performance on MTMedDialog: DoctorAgent-RL achieved a comprehensive average score of 53.9%, significantly outperforming frontier models (GPT-4o, DeepSeek-V3), open-source base models, and domain-specific models. It demonstrated superior stability across all eight disease categories.
• Human Evaluation: In blinded assessments of 100 samples, DoctorAgent-RL achieved the highest scores in Diagnostic Accuracy, Question Quality, and Information Coverage, surpassing even GPT-4o.
• Real-World Patient Trials: In interactions with 20 real patients, the model achieved a 70% Exact Diagnostic Match Rate, confirming its ability to handle unpredictable real-world scenarios.
• Generalization:
  • Unseen Diseases: The model showed negligible performance drops on unseen disease types, indicating transferable reasoning rather than rote memorization.
  • HealthBench: Ranked first among open-source small-scale models on the HealthBench benchmark (22.3% average score), demonstrating robustness in emergency referrals, communication skills, and complex response handling.
  • General-Purpose Capabilities: Unlike other domain-specific models that suffer from "catastrophic forgetting," DoctorAgent-RL retained its general conversational abilities in non-medical tasks (e.g., travel planning).
• Ablation Studies: Confirmed that both the SFT initialization and the RL optimization are critical. Removing RL led to mechanical questioning, while removing SFT resulted in poor initiative and lower scores.

5. Significance

• Clinical Impact: DoctorAgent-RL offers a viable solution to global physician shortages and misdiagnosis risks by effectively handling initial screenings and routine triage. This allows human clinicians to focus on complex cases requiring nuanced judgment.
• Methodological Advancement: The study proves that training LLMs to actively construct knowledge through dynamic interaction is superior to passively reproducing existing knowledge. It establishes a reproducible paradigm for task-oriented medical dialogue optimization.
• Future Pathway: The work provides a clear, validated path for developing next-generation clinical decision support systems that are not just chatbots but intelligent, proactive collaborative tools capable of reducing healthcare strain and improving patient care quality.