Technical Summary: MeDxAgent and MeDxBench

Problem Statement

Current Large Language Models (LLMs) are increasingly utilized for health-related decision support, yet their evaluation largely relies on static, single-shot tasks where complete case information is provided upfront (e.g., USMLE-style multiple-choice questions). This paradigm diverges significantly from real-world clinical practice, where diagnosis is an interactive, open-ended process involving sequential hypothesis refinement through targeted questioning.

Existing interactive diagnostic systems suffer from four primary limitations:

1. Lack of Standardized Benchmarks: No unified benchmark combines diverse clinical cases with reliable diagnoses; existing datasets are fragmented across formats and specialties.
2. Limited Agentic Architectures: Open systems often rely on simplistic single-agent loops, while sophisticated multi-agent systems are closed-source and non-reproducible.
3. Information Leakage: Many setups inadvertently leak ground-truth diagnoses (e.g., by initializing agents with partial case descriptions or allowing patient agents to generate synthetic information based on the diagnosis), artificially inflating accuracy.
4. Unsystematic Design Analysis: Prior work typically reports results from a single configuration, failing to isolate which specific design choices (prompt, flow, or agent composition) drive performance gains.

Methodology

1. MeDxBench: A Large-Scale Benchmark

The authors constructed MeDxBench, a benchmark comprising 4,421 clinical cases across 20 specialties (e.g., Infectious Diseases, Neurology, Emergency Medicine).

• Sources: Aggregated from five public datasets: CRAFT-MD, DiagnosisArena, MedMCQA, MedQA, and curated PubMed cases.
• Diversity: Includes 4,113 unique disease labels ranging from common (high prevalence) to rare (<1 in 2,000).
• Integrity: Explicit mentions of ground-truth diagnoses are redacted from case descriptions to prevent leakage. The benchmark excludes image/video-dependent cases to focus on text-only diagnosis.

2. Simulation Framework

The evaluation environment simulates multi-turn doctor-patient conversations using three agents:

• Patient Agent: Answers strictly from the provided case description. If information is missing, it responds with "I don't know," preventing the hallucination of facts or leakage of the ground truth.
• Doctor Agent: Conducts the consultation, asking questions and generating candidate diagnoses with confidence scores after each turn.
• Judge Agent: Evaluates the final diagnosis using semantic matching (exact match, synonym, or specific subtype) against the ground truth.

3. MeDxAgent: Multi-Agent Consultation System

The authors propose MeDxAgent, a system designed to mimic clinical reasoning through a systematic study of three design layers: Prompt, Flow, and Agent.

A. Prompt-Based Variations

• Turn Awareness: Guides the agent to prioritize screening questions early (turns 1–5), evidence gathering in the middle (6–15), and discriminative questions late (16–20).
• Demographics First: Enforces the collection of age and sex in the first turn to anchor the differential diagnosis, mirroring clinical practice.

B. Flow-Based Variations

• No Early Stopping: Runs consultations for the full 20 turns to test if accumulating more evidence improves accuracy over confidence-based early stopping.
• Differential Questioning: The agent receives current candidate diagnoses and is instructed to ask questions that distinguish between competing hypotheses. Crucially, this is activated only after a specific turn threshold (tested at turns 2, 5, and 10) to avoid "anchoring bias" where weak early hypotheses bias subsequent questioning.

C. Agent-Based Variations

MeDxAgent integrates four specialized modules:

1. Summarizer: Converts raw dialogue into a structured clinical representation (e.g., SOAP format) to align with the LLM's training data (clinical notes/vignettes).
2. Specialist Ensemble: A panel of 10 agents (9 specialists + 1 generalist) that independently propose top-3 diagnoses. A "Selector" agent weights these based on case relevance rather than majority voting.
3. Knowledge Graph (KG): Queries the Unified Biomedical Knowledge Graph (UBKG) using revealed symptoms to surface candidate diseases, grounding predictions in structured medical knowledge.
4. Evidence Gap: Identifies missing information required to support or reject current hypotheses. It distinguishes between "general gaps" (broad clinical details) and "diagnosis-specific gaps" (differentiators).

The full MeDxAgent system combines all these components, transitioning from broad information gathering (turns 1–9) to a differential diagnosis phase (turns ≥10) where the Evidence Gap and Specialist Ensemble guide targeted questioning.

Key Results

Using GPT-4o as the backbone, MeDxAgent achieved 57.4% average diagnostic accuracy on MeDxBench, a 10.3% absolute improvement over the baseline (47.1%).

• Oracle Gap: The full-information Oracle (receiving the complete case upfront) achieved 66.8%. MeDxAgent closed 52.3% of the gap between the baseline and this Oracle.
• Design Sensitivity:
  • Timing Matters: Activating "Differential Questioning" at turn 2 reduced accuracy to 34.7% (worse than baseline), while activating it at turn 10 yielded the best results (52.8%). This highlights the critical importance of intervention timing.
  • Compositional Effects: Individually, three of the four agent modules (Specialist Ensemble, KG, Evidence Gap) reduced accuracy when added to the baseline. However, when combined in the full system, they produced the highest performance. A standard incremental ablation would have incorrectly rejected these components; their value emerges only through interaction.
• Cross-Model Transfer: The design choices generalized to Grok-4.1 and DeepSeek-V3.2, consistently outperforming their respective baselines without re-tuning.
• Comparison: MeDxAgent outperformed prior systems like MedAgentSim and VivaBench across multiple datasets and models.

Significance and Claims

The paper claims three primary contributions:

1. MeDxBench: A standardized, large-scale benchmark for open-ended interactive diagnosis that prevents information leakage and spans diverse specialties.
2. MeDxAgent: A multi-agent system that recovers over half the gap to a full-information oracle by systematically integrating prompt, flow, and agent-level design choices.
3. Methodological Insight: The study reveals that interaction effects are critical in multi-agent medical systems. Components that are detrimental in isolation can become beneficial when combined, and the timing of specific interventions (like differential questioning) is as important as the intervention itself.

The authors emphasize that MeDxAgent is a decision-support tool intended to assist clinicians in differential diagnosis and hypothesis generation, not to replace medical professionals. They acknowledge limitations, including the text-only nature of the evaluation (excluding multimodal data like imaging) and the artificial constraints of the Patient agent (e.g., frequent "I don't know" responses), which may reduce ecological validity compared to real-world interactions.