💬 NLP

ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning

This paper introduces ClinSeekAgent, an automated agentic framework that shifts clinical reasoning from passive evidence consumption to active, dynamic acquisition of multimodal data from heterogeneous sources, demonstrating significant performance improvements on both inference-time benchmarks and distilled open-source models.

Original authors: Juncheng Wu, Letian Zhang, Yuhan Wang, Haoqin Tu, Hardy Chen, Zijun Wang, Cihang Xie, Yuyin Zhou

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juncheng Wu, Letian Zhang, Yuhan Wang, Haoqin Tu, Hardy Chen, Zijun Wang, Cihang Xie, Yuyin Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Pre-Packed Lunch" vs. The "Grocery Store"

Imagine a doctor trying to diagnose a patient.

The Old Way (Curated Input): It's like the doctor is handed a pre-packed lunchbox. Someone else has already picked out the specific sandwich, apple, and juice the doctor needs to solve the case. The doctor just has to eat the lunch and give an answer. If the lunchbox is missing a key ingredient (like a specific allergy note), the doctor might get the diagnosis wrong, even if they are very smart.
The Real World: In a real hospital, there is no lunchbox. The doctor has to walk through a massive, chaotic grocery store (the hospital records), check the freezer (medical images), and maybe even call a specialist on the phone (external knowledge) to find the right ingredients to cook a meal.

The Paper's Claim: Most current AI doctors are trained on the "pre-packed lunch" method. They are great at reasoning if you give them the right facts, but they fail when they have to go find those facts themselves.

The Solution: Meet ClinSeekAgent

The authors built a new system called ClinSeekAgent. Think of this as a super-intelligent medical detective that doesn't wait for a lunchbox. Instead, it has a backpack full of tools and goes out to hunt for the evidence it needs.

Here is how it works, step-by-step:

It Has a Toolkit: The agent has access to three main types of "search engines":
- The Electronic Health Record (EHR) Database: Like digging through a patient's entire digital filing cabinet, going back years to find old notes, lab results, and vital signs.
- The Web Browser: Like calling a medical library to look up the latest guidelines or definitions for a rare disease.
- The Imaging Room: Like having a radiologist look at X-rays and describe exactly what they see (e.g., "There is fluid in the lungs").
It Thinks and Acts: When given a question (e.g., "Will this patient get sick in the next 24 hours?"), the agent doesn't just guess. It:
- Looks at the patient's ID.
- Decides: "I need to check their blood pressure history first." (It searches the database).
- Then it thinks: "Okay, blood pressure is low. Now I need to see their chest X-ray." (It opens the image tool).
- Then it thinks: "The X-ray looks cloudy. Let me check if there's a specific definition for this cloudiness." (It searches the web).
- Finally, it combines all these clues to make a decision.

The Results: Does It Work?

The authors tested this detective against the old "pre-packed lunch" method using a new test suite called ClinSeek-Bench.

Text-Only Tasks (The Paperwork): When the agent had to read through thousands of pages of patient notes to find a risk factor, the smartest AI models got significantly better at predicting risks (like death or long hospital stays) when they could search the raw data themselves, rather than relying on a summary.
- Analogy: It's like finding a needle in a haystack. If you are given a box that might have the needle, you might miss it. If you are allowed to search the whole haystack, you find it every time.
Multimodal Tasks (The Mix of Text and Images): This is where the agent shined the most. When the task required combining a chest X-ray with patient history, the agent improved its accuracy by a huge margin (over 15% for the best models).
- Analogy: Imagine trying to solve a mystery where the suspect is described in a notebook and seen in a blurry photo. The old method only gave you the notebook. ClinSeekAgent looked at the photo and read the notebook, connecting the dots the old method missed.

The "Teacher" and the "Student"

The paper also tried something clever: Teaching smaller models to be detectives.

The Teacher: They used a very powerful, expensive AI (Claude Opus 4.6) to act as the detective. It solved hundreds of cases, showing exactly how it searched for evidence and what tools it used.
The Student: They took a smaller, open-source AI (Qwen3.5) and taught it by showing it the Teacher's search steps.
The Result: The student learned not just the answers, but the habit of searching. The student became much better at finding evidence on its own, closing the gap between the small open-source model and the massive, expensive teacher.

Summary of What the Paper Actually Says

Current AI is too passive: It waits for humans to feed it the right facts.
ClinSeekAgent is active: It goes out, searches raw hospital data, checks X-rays, and looks up medical facts on its own.
It works better: When the agent can search, it makes fewer mistakes, especially in complex cases where the answer is hidden in old records or images.
It can teach: We can use this active searching process to train smaller, cheaper AI models to be just as good at finding evidence.

What the paper does NOT claim:
The paper does not say this system is currently being used in real hospitals to treat patients. It does not claim it replaces doctors. It is a research framework designed to prove that active evidence seeking is a better way to build AI for medicine than the current method of passive evidence consumption. The authors plan to release their code and data so other researchers can build on this "detective" approach.

Technical Summary: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning

1. Problem Statement

Current clinical decision support systems based on Large Language Models (LLMs) and agentic frameworks largely rely on a "curated evidence" paradigm. In existing benchmarks and workflows, relevant patient data (e.g., specific EHR tables, selected radiology reports, or summarized vignettes) is pre-selected and handed to the model before inference. This deviates from real-world clinical practice, where clinicians must actively seek, plan, and synthesize evidence from heterogeneous sources, including raw Electronic Health Records (EHRs), medical imaging, and external medical knowledge.

The core challenge is not merely reasoning over given evidence, but determining where to retrieve evidence, what to retrieve, and how to integrate these disparate signals into a grounded decision. Existing agentic EHR systems often lack the scope to handle multimodal data (text + images) or rely on fixed tool usage patterns that do not adapt to the specific needs of a clinical query.

2. Methodology: ClinSeekAgent

The authors introduce ClinSeekAgent, an automated agentic framework designed to shift clinical reasoning from passive evidence consumption to active evidence acquisition.

2.1 Framework Architecture

ClinSeekAgent operates as a dynamic pipeline where the agent interacts with a unified tool space to gather evidence iteratively. Given a clinical query ( $q$ ), a patient identifier ( $p$ ), and a reference timestamp ( $t$ ), the agent:

Accesses Heterogeneous Sources: It queries raw EHR databases, medical imaging repositories, and external knowledge bases.
Iterative Planning: The agent observes the task and previous interaction history ( $h_{k-1}$ ) to decide the next action ( $a_k$ ), which can be a tool invocation or a final prediction.
Synthesis: It integrates the retrieved evidence to refine hypotheses and generate a grounded clinical decision.

2.2 Tool Space

The framework exposes 20 tools across three complementary domains:

EHR Retrieval (11 tools): Includes schema inspection, temporal retrieval, SQL-based querying (ehr.run_sql_query), and candidate-term grounding. These allow the agent to treat the EHR as a programmable database rather than a static text block.
Web Search (3 tools): Browser tools for acquiring external medical knowledge (e.g., drug indications, taxonomies).
Medical Imaging (6 tools): Tools for DICOM preprocessing, chest X-ray (CXR) classification, report generation, phrase grounding, and anatomical segmentation.

2.3 Evaluation Benchmarks

To validate the framework, the authors constructed ClinSeek-Bench, which pairs existing benchmarks into two settings for the same task:

Curated Input: The model receives pre-selected, task-relevant evidence (mimicking prior work).
Automated Evidence-Seeking: The model receives only the patient ID and access to the ClinSeekAgent tool space, requiring it to retrieve the necessary evidence autonomously.

The benchmark covers:

Text-only EHR tasks: Derived from EHR-Bench (45 subtasks on decision-making and risk prediction).
Multimodal tasks: Adapted from EHRXQA and MedMod (6 task groups involving CXR and EHR integration).

2.4 Training Pipeline

Beyond inference-time evaluation, the authors utilize ClinSeekAgent as a training pipeline. They use a strong teacher model (Claude Opus 4.6) to generate high-quality evidence-seeking trajectories (tool calls and observations). These trajectories are used to fine-tune an open-source model (Qwen3.5-35B-A3B) via Supervised Fine-Tuning (SFT), resulting in ClinSeek-35B-A3B.

3. Key Results

3.1 Inference-Time Performance

When paired with capable agentic models, ClinSeekAgent significantly outperforms the Curated Input baseline, particularly in tasks requiring sparse or longitudinal evidence.

Text-Only EHR: Claude Opus 4.6 improved from 60.0 to 63.2 overall F1. MiniMax M2.5 improved from 43.1 to 47.3.
Risk Prediction: Gains were most pronounced in risk prediction (7 out of 9 models showed positive gains). For example, Claude Opus 4.6 saw a +9.7 point gain in risk prediction tasks.
Multimodal Tasks: The framework showed substantial improvements across all evaluated models. Claude Opus 4.6 improved from 47.5 to 62.6 (+15.1 points). All six evaluated models improved on CXR-related tasks, with gains attributed to the agent's ability to combine visual evidence (CXR classifiers/reports) with structured EHR data and external knowledge.

3.2 Training-Time Performance

The distilled model, ClinSeek-35B-A3B, achieved state-of-the-art performance among open-source models on the AgentEHR-Bench.

It improved the base Qwen3.5-35B-A3B model from 22.1 to 34.0 average F1 (+11.9 points).
It outperformed other open-source baselines (e.g., Kimi K2.5, MiniMax-M2.5) and approached the performance of the teacher model, Claude Opus 4.6 (34.0 vs. 36.0).
Analysis of tool usage revealed that the distilled model learned a more diverse and flexible retrieval policy, specifically increasing the use of SQL queries (ehr.run_sql_query) from 2.0% to 12.5% of tool calls, treating the EHR as a database rather than relying on fixed templates.

3.3 Limitations Observed

The paper notes that the framework is less effective on decision-making tasks compared to risk prediction. In some decision-making subtasks, the agent struggled to identify critical signals amidst retrieved noise, sometimes performing worse than the Curated Input baseline. This suggests that while active retrieval excels at finding sparse evidence for risk, it may occasionally retrieve excessive irrelevant information for complex decision-making scenarios.

4. Significance and Claims

The paper claims that ClinSeekAgent represents a paradigm shift from passive evidence consumption to active evidence acquisition. Its significance lies in three areas:

Realism: It aligns AI evaluation with real-world clinical workflows where evidence is not pre-packaged but must be actively sought from raw, heterogeneous sources.
Multimodal Integration: It demonstrates that agentic systems can effectively bridge the gap between structured EHR data, unstructured medical imaging, and external knowledge, recovering clinical signals that fixed contexts miss.
Scalability: It validates that high-quality agentic trajectories can be distilled into compact open-source models, enabling smaller models to perform complex, long-horizon clinical evidence seeking without requiring proprietary, massive-scale models at inference time.

The authors conclude that automating the evidence-seeking process is a promising direction for building more flexible, grounded, and capable clinical AI agents, and they commit to releasing their model, data, and code to facilitate future research.