🤖 AI

MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional

The paper introduces MDIA, a multi-agent diagnostic system utilizing a 7-node specialty-routed graph on a non-fine-tuned LLM that significantly outperforms standard clinical chatbots on the HealthBench Professional benchmark, demonstrating that architectural design and engine-level features are critical drivers of agentic clinical performance while highlighting the variability introduced by different evaluation grader models.

Original authors: Roberto Cruz, David Rey-Blanco

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roberto Cruz, David Rey-Blanco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex medical mystery. You could ask one very smart doctor to look at the case and give you an answer. Or, you could build a super-team of specialists who talk to each other, check their facts, and double-check the final report before handing it to you.

This paper describes the creation of that super-team, called MDIA, and how it performed on a very difficult test called HealthBench Professional.

Here is the breakdown of what they did, using simple analogies:

1. The Big Idea: Teamwork Beats One Genius

Most people thought the best way to get a smart AI to answer medical questions was to just make the AI smarter (by training it more). But the authors found that how you organize the team matters more than just having a smart individual.

The Old Way: One AI tries to do everything alone. It's like asking a single general practitioner to be a surgeon, a neurologist, and a pharmacist all at once.
The MDIA Way: They built a 7-step assembly line (a "graph").
1. The Intake: A receptionist gathers all the patient's history and runs checks on drug safety.
2. The Router: A manager looks at the problem and says, "This is a stomach issue, send it to the Gastro team," or "This is a brain issue, send it to the Neuro team."
3. The Specialists: Three specific experts (Gastro, Eye, Neuro) and one generalist handle the heavy lifting.
4. The Synthesizer: A writer turns the expert's notes into a clear answer for the patient.
5. The Verifier: A safety inspector checks the final answer to make sure it's safe and not too long.

2. The Secret Sauce: Listening to the Whole Story

The authors discovered a huge "bug" in how these tests were usually run.

The Problem: Imagine a patient says, "I have a stomach ache." Then, in the next message, they say, "Oh, and I also took this specific pill."
The Mistake: Most testing systems only read the last message ("I took this pill") and forgot the first one ("I have a stomach ache"). It's like a detective ignoring the crime scene and only looking at the suspect's alibi.
The Fix: MDIA was designed to remember the whole conversation. When they fixed the test to include the whole story, the score jumped up by a massive amount (about 6 points). This wasn't because the AI got smarter; it was because the test finally let the AI use the information it already had.

3. The Results: Beating the "Gold Standard"

The authors tested their system against the current "champion" (OpenAI's ChatGPT for Clinicians) and a team of human doctors.

The Score: MDIA scored 0.6272.
The Comparison:
- Human Doctors (baseline): ~0.44
- OpenAI's Best System: 0.59
- MDIA: 0.63
The Catch: The paper admits this victory is "directional," meaning it's a win, but the margin is small enough that it could be due to luck or how the test was graded. It's like winning a race by a fraction of a second; you won, but it was close.

4. The "Judge" Problem

Here is the twist: The paper tested their system using two different "judges" (AI models that grade the answers).

Judge A (GPT-5.4): Gave MDIA a score of 0.6272.
Judge B (Gemini 2.5 Pro): Gave MDIA a score of 0.6585.
The Lesson: The score depends on who is grading. One judge liked the long, detailed answers; the other preferred shorter ones. The authors argue that to truly know if an AI is good, you need multiple judges, not just one.

5. Engineering "Plumbing" Matters

A lot of the improvement didn't come from making the AI "smarter" with new math. It came from fixing the plumbing:

Safety Gates: They added a rule that stops the AI from suggesting dangerous drug combinations (like mixing a fever reducer with a specific stomach medicine).
Length Control: The test penalizes answers that are too long. The authors taught the AI to be concise (cutting the fluff) without cutting the important medical facts. This shaved off points that were being lost due to "wordiness."
Reliability: They fixed bugs where the system would sometimes crash or return empty answers. Fixing these "glitches" recovered about 3-4 points of performance.

Summary

The paper claims that architecture (how you build the team) and engineering (fixing the tools and rules) are just as important as the brain (the AI model) itself.

They built a specialized medical team that, when given the full context of a conversation and graded fairly, can outperform both human doctors and the current market leader on a specific, rigorous test. However, they warn that the results are sensitive to how the test is run and who is grading it, so we shouldn't treat this as a final, perfect solution yet. It's a powerful prototype that shows the future of medical AI lies in teams of specialists, not just one big brain.

Technical Summary: MDIA - A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional

Problem Statement

The paper addresses the challenge of evaluating Large Language Models (LLMs) in clinical settings, specifically using OpenAI's HealthBench Professional benchmark. This benchmark consists of 525 rubric-graded cases derived from real clinician conversations, designed to be adversarial and multi-turn.

The authors identify two primary gaps in current literature:

Architectural vs. Prompt Engineering: While many reported gains in agentic-LLM benchmarks are attributed to prompt engineering, the authors hypothesize that significant improvements can be achieved through architectural and engine-level design.
Evaluation Methodology: Prior work has not sufficiently examined how the benchmark's multi-turn conversation structure interacts with standard evaluation harness designs. Specifically, the "simple-evals" default strategy often flattens conversations to the last user message, silently discarding context from follow-up turns (which constitute 22% of the benchmark cases).

Methodology: The MDIA Architecture

The authors present MDIA (Multi-agent Diagnostic Intelligence Agent), a coordinated 7-node Directed Acyclic Graph (DAG) executed on the TietAI Hydra Platform. The system runs on a non-fine-tuned general-purpose LLM (Gemini 3.1 Pro for reasoning) and utilizes a suite of 14 medical tools.

The graph structure consists of the following nodes:

Intake Orchestrator: A tool-calling research agent that collects a structured clinical dossier using tools like PubMed, DailyMed, UMLS, ICD-10, and site-filtered web searches. It is multi-turn aware, preserving conversation history.
Router (Specialty Classifier): Analyzes the dossier and branches the task to one of three domain-expert reasoners (Gastroenterology, Ophthalmology, Neurology) or a Generalist path.
Specialty Reasoners: Three specialized agents (GI, Ophtho, Neuro) equipped with curated anchor knowledge (e.g., Glasgow-Blatchford, NIHSS, Tokyo Guidelines).
Generalist Reasoner: Handles the "long tail" of specialties not covered by the three experts.
Output Synthesizer: Converts the reasoner's brief into a user-facing response with length constraints (target 2000–3000 characters, hard cap 4000).
Verifier: Performs a final safety and format check.
Drug-State Safety Gate: A mandatory check before content generation that refuses or warns on dangerous drug-patient state combinations (e.g., loperamide with fever) while carving out educational contexts to prevent over-refusal.

Key Engineering Interventions:

Multi-Turn Handling: The evaluation harness was modified to pass the full conversation history (multiturn strategy) rather than just the last user message (last_user).
Engine Reliability: Fixes included JSON code-fence stripping, empty-output retries, and per-model location overrides to ensure stable execution.
Search Hygiene: Implementation of site-filtered queries to high-authority medical sources (e.g., PubMed, NICE, UpToDate) to reduce noise.
Length Guidance: Explicit constraints to reduce verbosity without dropping clinical anchors, addressing the benchmark's length penalty.

Key Results

The paper reports results on the full HealthBench Professional dataset ( $n=525$ ) using two different graders: OpenAI's GPT-5.4-2026-03-05 and Google's Gemini 2.5 Pro.

Performance under GPT-5.4 Grader:

MDIA v1.0.53: Achieved a score of 0.6272.
Comparison:
- +3.72 pp over OpenAI's "ChatGPT for Clinicians" (0.590). The authors note this margin is within the bootstrap standard deviation ( $\sigma \approx 0.023$ ) and should be treated as directional rather than decisive.
- +14.62 pp over the GPT-5.4 single-agent baseline (0.481).
- +19.02 pp over physician-written responses (0.437).

Performance under Gemini 2.5 Pro Grader:

MDIA v1.0.53: Achieved a score of 0.6585 (with v1.0.50 at 0.6771).
Observation: The choice of grader introduces significant variability. For instance, the length-guidance intervention (v1.0.53) improved the GPT-5.4 score but slightly regressed the Gemini score, suggesting different graders weigh conciseness vs. elaboration differently.

Impact of Specific Interventions:

Multi-Turn Context: Switching from last_user to multiturn flattening increased the score by +5.0 pp (Pro-graded) and +6.3 pp (GPT-5.4 graded). This lift came entirely from the 22% of cases with follow-up turns.
Engine Reliability: Fixes to the execution engine recovered approximately 3–4 pp previously lost to infrastructure flakiness (reducing empty response rates from 3.8% to 0.2%).
Specialty Routing: Branching to specific reasoners (GI, Ophtho, Neuro) provided a +3.1 pp lift over a generalist-only approach.

Significance and Claims

The paper makes four primary contributions:

Architectural Superiority: It demonstrates that a multi-agent pipeline on a general-purpose, non-fine-tuned LLM can outperform a proprietary, domain-specific model (ChatGPT for Clinicians) under the same grader. The authors attribute this to system architecture (specialty routing, safety gating, multi-turn context) rather than model fine-tuning.
Evaluation Flaw Identification: The paper highlights that the standard "last user message" flattening strategy in evaluation harnesses understates the performance of context-aware agents by approximately 6 percentage points. It argues that all future HealthBench Professional reporting must disclose the flattening strategy.
Engineering Robustness: It documents that engine-level reliability (handling empty outputs, JSON parsing, location overrides) is a critical, often overlooked factor that can account for 3–4 pp of performance variance.
Grader Variability: The study reinforces that LLM-as-a-judge evaluations are sensitive to the choice of grader. The authors claim that robust evaluation requires assessment across multiple independent grader models to ensure fairness and reduce bias.

Limitations and Caveats:

The authors explicitly state that the 3.72 pp margin over ChatGPT for Clinicians is within statistical noise and OpenAI has not published confidence intervals for their system.
The comparison is complicated by the lack of documentation regarding OpenAI's own evaluation harness (specifically whether they used multi-turn context).
The results are presented as technical indicators of system capability rather than definitive clinical validation, acknowledging that the benchmark issuer (OpenAI) is also a model vendor.

The paper concludes that agentic clinical benchmark performance is shaped by both the foundation model and the orchestration architecture, and that future benchmarks must prioritize transparency in evaluation harnesses and multi-grader validation.