🤖 AI

EviDx: Evidence-Aware Active Diagnosis with Scaffolded LLM Agents

The paper introduces EviDx, an evidence-aware active diagnosis framework that utilizes scaffolded LLM agents and a runtime harness to dynamically acquire clinical evidence and manage diagnostic uncertainty, thereby improving both diagnostic performance and process stability compared to static prediction models.

Original authors: Lihang Zeng, Shaoting Zhang, Xiaofan Zhang

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lihang Zeng, Shaoting Zhang, Xiaofan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the real world, a doctor does not diagnose a patient by reading a single paragraph and instantly guessing an answer. Diagnosis is a slow, active hunt for clues. A physician listens to a patient's story, then orders a blood test, reviews an X-ray, or asks about family history. With each new piece of information, the doctor updates their list of possible causes, ruling some out and sharpening the focus on others. They must constantly decide when they have gathered enough evidence to make a call, and when they need to keep looking. This process of seeking evidence, weighing possibilities, and knowing when to stop is the core of clinical reasoning.

For years, researchers have tried to teach computers to do this using large language models, the powerful artificial intelligence systems that can write and chat like humans. However, most of these systems have been trained to treat diagnosis like a static quiz. They are handed a complete patient story all at once and asked to spit out a final answer immediately. This approach misses the point of how real medicine works. It forces the computer to guess the solution before it has had a chance to investigate, much like asking a student to solve a math problem without letting them use a calculator or scratch paper. While these models can recall medical facts, they often fail to mimic the careful, step-by-step gathering of evidence that prevents mistakes.

A new study introduces a system called EviDx, designed to change how these artificial intelligence agents approach medical diagnosis. Instead of giving the computer a finished case file, the researchers built a virtual environment where the AI must actively seek out information. Imagine a digital hospital room where the patient's records, lab results, and imaging scans are locked away in separate drawers. The AI agent cannot see them all at once; it must ask for specific tests, wait for the results, and then decide what to ask for next. This system includes a set of rules that guide the agent's behavior, ensuring it checks the right types of information and does not stop searching until it has covered the necessary ground.

The researchers also added a safety monitor, a digital overseer that watches the agent's progress in real time. This monitor checks two things: how confused the agent still is about the diagnosis, and how much of the available evidence it has actually looked at. If the agent tries to declare a diagnosis too early, while it is still uncertain or has skipped important tests, the monitor stops it and forces it to keep investigating. This prevents the computer from making a confident guess based on incomplete information. The system was tested on hundreds of real-world clinical cases, ranging from complex emergency room scenarios to detailed journal reports, using a variety of different AI models.

The results showed that this active, evidence-seeking approach worked significantly better than the old method of static guessing. The AI agents using EviDx were much more accurate at identifying the correct diagnosis, especially when they had to choose from a list of options. The system helped even smaller, less powerful models perform much better by guiding them to find the right clues. However, the study also found a clear limit: while the system could teach an agent how to gather evidence and organize its thoughts, it could not fix a lack of fundamental medical knowledge. If the AI model did not already know enough about a specific disease to understand the clues it found, it still struggled to reach the right conclusion.

The researchers also discovered that the biggest hurdle for these systems was not just medical knowledge, but the ability to follow strict instructions. Many of the smaller AI models frequently made errors in the way they requested information, such as formatting their requests incorrectly or failing to use the proper tools. These technical glitches often caused the diagnostic process to break down entirely. The study suggests that for artificial intelligence to become a reliable partner in medicine, it must be evaluated not just on whether it gets the final answer right, but on how it behaves during the investigation. Did it look for the right evidence? Did it stop when it had enough? The new framework proves that building a system that mimics the active, cautious nature of human diagnosis is possible, but it requires careful design to ensure the computer stays on the right path.

Technical Summary: EviDx

Problem Formulation

Clinical diagnosis is fundamentally an active, iterative process where clinicians acquire evidence, update competing hypotheses, and determine when sufficient information exists to conclude. However, existing Large Language Model (LLM) diagnostic systems largely treat diagnosis as a static "case-to-answer" prediction problem. In these setups, models receive a complete clinical vignette upfront and generate a diagnosis in a single pass, collapsing evidence acquisition, integration, and termination into one prompt. This approach fails to model process failures inherent to active diagnosis, such as premature commitment before sufficient evidence is gathered.

While tool-augmented and agentic systems offer a path toward dynamic diagnosis by allowing access to patient records and external knowledge, they often lack explicit mechanisms for controlling how evidence is exposed, how actions are scaffolded, and when the diagnostic process should terminate. Existing systems frequently rely on fixed workflows or turn budgets, leaving the critical decision of evidence sufficiency weakly modeled.

Methodology: The EviDx Framework

The authors introduce EviDx, an evidence-aware active diagnosis framework designed to transform static clinical cases into interactive diagnostic environments. The system couples patient-specific environments with a clinical diagnostic scaffold and an observer-guided runtime harness. EviDx consists of four core components:

1. E-Synthesis (Interactive Environment Construction)

E-Synthesis converts raw clinical text into a patient-specific interactive environment ( $E_x$ ). A Context Initializer processes the raw text into structured patient data (EHR), preserving clinically relevant ambiguities, qualifiers, and negative findings. This structured data is loaded into an EHR Server, coupled with evidence-access tools to form an environment where diagnostic actions can query specific patient records and external medical knowledge via the Model Context Protocol (MCP).

2. Clinical Dx Scaffold (Structured Execution)

The scaffold organizes the diagnostic execution through role-specialized agents and evolving evidence states, rather than generic prompting:

Diagnostician Agent: The central reasoning component that maintains the differential diagnosis set ( $D_t$ ) and belief state ( $bP_t$ ). It decides the next action (querying data, consulting knowledge, or finishing) based on the current trajectory.
EHR Executor: Handles deterministic access to the EHR Server, retrieving demographics, history, labs, and imaging reports.
Clinical Consultant: Handles external evidence access by decomposing requests and retrieving relevant literature from a Knowledge Server (using hybrid dense and BM25 retrieval) to support the Diagnostician.

3. Observer-Guided Harness (Runtime Control)

A scaffold alone is insufficient; the system requires runtime control to prevent premature termination. The Observer-Guided Harness monitors the diagnostic trajectory and regulates termination based on two metrics:

Diagnostic Uncertainty ( $H(t)$ ): Calculated as the entropy of the reported differential diagnosis. High entropy indicates unresolved hypotheses.
Runtime Evidence Coverage ( $V(T_t)$ ): The fraction of available EHR sections explicitly queried by the agent.

Termination is approved only when the trajectory satisfies both criteria: $H(t) < \tau_H$ (sufficiently resolved) and $V(T_t) > \tau_V$ (sufficiently explored). The harness uses step-aware dynamic thresholds that gradually relax requirements as the interaction proceeds, preventing infinite loops while blocking ungrounded exits.

4. 3-Level Evaluation Pyramid

To assess the framework, the authors propose a multi-dimensional evaluation strategy:

Level 1 (Execution Robustness): Measures tool-interaction reliability, specifically MCP Syntax and Schema errors.
Level 2 (Reasoning Dynamics): Evaluates Uncertainty Convergence (how effectively belief states concentrate) and Observer Intervention frequency (how often the harness blocks premature termination).
Level 3 (Diagnostic Outcomes): Assesses final accuracy and Reference Evidence Recall, distinguishing between merely retrieving evidence (Acquisition) and actively utilizing it in reasoning (Cognition).

Key Contributions

EviDx Framework: A novel architecture that transforms static clinical cases into interactive environments, scaffolding LLM agents with specific clinical roles, evidence tools, and state management.
Observer-Guided Harness: A runtime control mechanism that explicitly manages diagnostic termination by tracking uncertainty and evidence coverage, preventing premature conclusions.
3-Level Evaluation Pyramid: A comprehensive assessment methodology that moves beyond final accuracy to evaluate execution robustness, reasoning dynamics, and evidence-grounded outcomes.
Med-Evidence-2.6k: A curated dataset of 2,660 richly annotated clinical instances created via an LLM-assisted pipeline with physician review. While the full dataset supports evidence-recall evaluation, the main experiments utilize a controlled subset of 100 instances from each source benchmark to enable cross-model comparison.

Experimental Results

The framework was evaluated on three datasets (JAMA, MedXpertQA-Diag, DiagnosisArena) using a range of models from 8B-parameter open-weight models to frontier models (GPT-5.2, Claude Sonnet 4.6, etc.).

Execution Robustness: Large frontier models (GPT-5.2, DeepSeek-V3.2) demonstrated high robustness in tool usage. Smaller models (e.g., Llama3.1-8B) exhibited significant schema and syntax errors, highlighting that execution robustness is a prerequisite for active diagnosis.
Diagnostic Performance: EviDx improved Multiple Choice (MC) diagnostic performance across all models. Notably, 8B-parameter models saw substantial gains (e.g., Qwen3-8B on JAMA increased from 18% to 44%). However, open-ended diagnosis remained limited by the base model's parametric medical knowledge, with small models scoring near 0% in some open-ended settings.
Process Stability: The Observer-Guided Harness successfully mitigated unsupported termination. Case studies showed that without the harness, agents often concluded prematurely under high uncertainty or misleading negative findings. The harness forced continued evidence acquisition, leading to correct diagnoses in scenarios where unconstrained agents failed.
Evidence Recall: While MC accuracy improved, some small models showed lower "Cognition" scores (utilization of evidence in reasoning) compared to single-agent baselines. This suggests a tension where tool coordination leads to more compressed or paraphrased traces, reducing exact text-matching recall despite correct reasoning.

Significance and Claims

The paper claims that EviDx demonstrates that clinical LLM agents become more capable when interactive diagnostic environments are coupled with explicit evidence scaffolds and runtime harnesses. The results suggest that progress in clinical LLM diagnosis should be evaluated not only by final answers but also by the quality of evidence acquisition, the maintenance of diagnostic trajectories, and the decision-making process regarding evidence sufficiency.

The authors modestly note that while EviDx improves process stability and MC performance, it cannot fully compensate for limitations in the parametric medical knowledge of smaller foundation models. The framework is presented strictly as a research tool for studying evidence-aware active diagnosis, not as a ready-to-deploy clinical system, emphasizing the necessity of human-in-the-loop oversight for safety.