Human-AI Co-reasoning for Clinical Diagnosis with Evidence-Integrated Language Agent

The paper introduces PULSE, a medical reasoning agent that integrates a domain-tuned large language model with scientific literature retrieval to achieve expert-competitive diagnostic accuracy across varying disease incidences, while demonstrating both its potential to enhance physician decision-making and the risks of automation bias in collaborative workflows.

The Gist

Imagine you are a detective trying to solve a very tricky mystery. Usually, you rely on your memory, your training, and the clues you've seen before. But what if you had a super-smart, tireless assistant who could instantly read every book in the world's largest library to help you?

That is essentially what this paper is about. The researchers built an AI detective named PULSE to help doctors diagnose difficult medical cases, specifically in the field of endocrinology (hormone disorders).

Here is the story of how they tested it and what they found, explained simply:

1. The Setup: The "Medical Olympics"

The researchers created a test using 82 real, complex patient stories involving hormone problems. Some of these cases were common (like a standard thyroid issue), while others were incredibly rare (like finding a needle in a haystack).

They invited three groups of human doctors to solve these mysteries:

• The Residents: Medical students just starting out.
• The Junior Specialists: Doctors with a few years of experience.
• The Senior Experts: Veterans with decades of experience.

Then, they let PULSE (the AI) try to solve the same cases. PULSE isn't just a chatbot; it's a "reasoning agent." It doesn't just guess; it thinks, searches the medical literature, and builds a case.

2. The Results: Who Won?

• The AI vs. The Beginners: PULSE crushed it against the residents and junior doctors. It was like a grandmaster chess player playing against beginners.
• The AI vs. The Experts: This is the cool part. PULSE performed just as well as the most senior experts. It got the right answer almost as often as the best human doctors did.
• The "Rare Disease" Test: Here is where the AI shined brightest. Human doctors get worse at diagnosing rare diseases because they rarely see them. If a disease is super rare, a human might not even know it exists. PULSE didn't care how rare it was. It performed consistently well on common and rare cases alike because it had "read" the books on those rare diseases.

3. How PULSE Thinks: The "Adaptive Brain"

The researchers noticed something fascinating about how PULSE worked.

• Human Doctors: When a case was easy, they solved it quickly. When it was hard, they spent more time thinking. But the newer doctors often gave up too fast on hard cases.
• PULSE: It acted like a wise old professor. When the case was easy, it gave a quick answer. When the case was a nightmare, it slowed down and thought longer, generating more ideas and checking more evidence. It knew when to "deliberate."

4. The Teamwork Experiment: "Second Opinion" vs. "Co-Pilot"

The researchers tested two ways for doctors and AI to work together:

• Mode A: The "Second Opinion" (Serial)

  • How it works: The doctor makes a diagnosis first. Then the AI gives its answer. The doctor looks at it and decides whether to change their mind.
  • Result: This worked well. The AI acted like a safety net, catching mistakes the doctor made. However, sometimes the doctor trusted the AI too much, even when the AI was wrong (a bit like blindly following a GPS into a lake).

• Mode B: The "Co-Pilot" (Concurrent)

  • How it works: The doctor and the AI work side-by-side from the very beginning. The AI suggests ideas while the doctor is still thinking.
  • Result: This was a game-changer for the junior doctors. It helped them think of possibilities they never would have considered on their own. It expanded their "mental map" of what the problem could be. Interestingly, the AI and the doctor didn't always agree, but that was okay! The doctor used the AI's ideas as a springboard to find the right answer, rather than just copying the AI.

5. The Big Takeaway

The paper concludes that AI isn't here to replace doctors. Instead, it's like a super-powered library card combined with a thinking partner.

• For Rare Diseases: The AI is a hero because it remembers everything humans forget.
• For Training: It helps new doctors learn faster by exposing them to rare cases they might not see for years in real life.
• The Warning: We have to be careful not to blindly trust the AI. Sometimes the AI gets "too creative" and suggests weird, complex diseases when a simple one is the answer. Doctors need to stay in the driver's seat, using the AI as a navigator, not the driver.

In a nutshell: PULSE is a brilliant tool that helps doctors, especially when the case is weird or rare. When used correctly, it makes the whole team smarter, faster, and more accurate.

Technical Summary

Here is a detailed technical summary of the paper "Human–AI Co-reasoning for Clinical Diagnosis with Evidence-Integrated Language Agent" (PULSE).

1. Problem Statement

Accurate medical diagnosis, particularly in complex fields like endocrinology, is hindered by the "long tail" of rare diseases, atypical presentations, and the cognitive limitations of human practitioners.

• Diagnostic Errors: Diagnostic errors remain common, especially for multisystem or low-incidence diseases where early symptoms are nonspecific.
• Experience Gap: Diagnostic performance in humans is heavily dependent on clinical experience. Trainees and junior specialists often suffer from "premature closure" (settling on a diagnosis too early) and struggle with rare conditions they have rarely encountered.
• Limitations of Current AI: Existing AI evaluations often focus on simplified vignettes or radiological image classification rather than complex, longitudinal clinical reasoning. Furthermore, the optimal workflow for Human-AI collaboration (e.g., serial vs. concurrent) and the impact of automation bias remain under-explored.

2. Methodology: The PULSE Framework

The authors introduce PULSE, a medical reasoning agent that integrates a domain-tuned Large Language Model (LLM) with a scientific literature retrieval engine.

A. System Architecture

PULSE operates through a five-phase iterative process:

1. Iterative Hypothesis Generation: The LLM generates $N$ rounds of diverse diagnostic hypotheses based on the clinical case. A high temperature setting ($T=1.0$) is used to encourage exploration and diversity.
2. Diagnosis Aggregation: The system aggregates hypotheses from all rounds, using semantic clustering to group synonyms and select the top $m$ consensus candidates ($D_{top}$).
3. Evidence Retrieval: For each candidate in $D_{top}$, the system queries PubMed via the NCBI E-utilities API to retrieve relevant articles (filtered for the last 15 years).
4. Evidence Summarization: The LLM summarizes the retrieved abstracts, extracting findings specifically relevant to the patient's clinical presentation ($E_{final}$).
5. Refined Diagnosis: The LLM synthesizes the original clinical case, the summarized evidence, and the candidate diseases to generate a final, evidence-backed diagnosis and differential list.

B. Model Configuration

• Base Model: A 32-billion parameter model based on the Qwen3 architecture.
• Training: Two-stage fine-tuning:
  1. Foundational Alignment: Medical knowledge QA, symptom-disease association, and keyword summarization.
  2. Reasoning Refinement: Complex clinical reasoning, differential diagnosis, and case-based reasoning using curated medical datasets.
• Hardware: Trained on 16x NVIDIA H100 GPUs using Megatron-LM.

C. Experimental Design

• Benchmark: A curated dataset of 82 authentic endocrinology case reports spanning seven subspecialties (e.g., adrenal, thyroid, pituitary) and stratified by disease incidence (Common: 0.01–1%, Uncommon: 0.001–0.01%, Rare: <0.001%).
• Human Baselines: Performance was compared against three groups:
  • 1 Senior Specialist (Attending).
  • 8 Junior Specialists (Fully licensed, post-residency).
  • 8 Residents (Trainees).
• Collaboration Workflows:
  • Serial: Physicians diagnose independently, then review and revise based on PULSE's output.
  • Concurrent: Physicians receive PULSE's suggestions in real-time during the initial diagnostic process (acting as a "co-pilot").

3. Key Contributions

1. Novel Agent Design: PULSE combines a reasoning-oriented LLM with real-time evidence retrieval, moving beyond static knowledge to dynamic, evidence-grounded diagnosis.
2. Comprehensive Benchmarking: The study provides a rigorous evaluation against a spectrum of human expertise (Residents to Senior Specialists) using a real-world, rare-disease-heavy dataset.
3. Workflow Analysis: It explicitly compares Serial vs. Concurrent collaboration, revealing distinct cognitive dynamics and performance outcomes for each.
4. Adaptive Reasoning Discovery: The study demonstrates that high-performing AI agents, like senior humans, exhibit "adaptive thinking"—allocating more reasoning effort (longer output tokens) to difficult cases.

4. Key Results

A. Standalone Performance

• Accuracy: PULSE achieved 57.32% Top@1 accuracy, significantly outperforming Junior Specialists (34.63%) and Residents (23.41%). It was statistically comparable to the Senior Specialist (65.85%, $p=0.25$).
• Top@4 Performance: PULSE reached 79.27%, matching the Senior Specialist (82.93%) and significantly outperforming less experienced groups.
• Robustness to Rarity: Unlike human physicians, whose accuracy dropped significantly as disease incidence decreased (Junior specialists dropped to 25.6% in the ultra-rare tier), PULSE maintained stable performance across all incidence tiers.
• Hypothesis Breadth: PULSE generated a wider search space (270 distinct entities) compared to all human groups, indicating a superior ability to consider long-tail diagnoses.

B. Adaptive Effort

• Humans: Senior specialists spent more time on difficult cases (adaptive effort), whereas residents showed no significant correlation between case difficulty and time spent.
• AI: PULSE and other high-performing models (e.g., gpt-oss-120B) exhibited an "adaptive thinking" profile, increasing output token length for difficult cases, mirroring expert human behavior.

C. Human-AI Collaboration

• Performance Gains: AI assistance significantly improved the diagnostic accuracy of junior specialists and residents, effectively closing the gap with senior specialists.
• Serial vs. Concurrent:
  • Serial: Led to high concordance between AI and physician. Physicians often adopted the AI's correct suggestions, acting as a "converging force."
  • Concurrent: Resulted in lower AI-physician correlation but maintained high accuracy. Physicians treated the AI as an information stream to broaden their hypothesis space without blindly adopting the output.
• Correction vs. Misleading: In serial workflows, the rate of correcting initial errors (35.4%–47.2%) outweighed the rate of being misled by incorrect AI suggestions. However, "automation bias" (adopting incorrect AI suggestions) remains a risk.

5. Significance and Implications

• Democratizing Expertise: PULSE demonstrates that AI can mitigate the "experience gap," allowing junior clinicians to perform at levels comparable to senior specialists, particularly for rare diseases.
• Workflow Optimization: The study suggests that Concurrent workflows are superior for expanding the initial hypothesis space (preventing premature closure), while Serial workflows are effective for final verification and correction.
• Safety and Trust: The research highlights the dual nature of AI: it expands the diagnostic horizon but introduces risks of hallucination and automation bias. It argues for "trust calibration" where physicians are trained to interrogate AI reasoning rather than passively accept it.
• Future Directions: The authors advocate for multimodal integration (images, labs), larger multi-center RCTs, and interface designs that promote active cognitive engagement rather than passive reliance.

In conclusion, PULSE represents a significant step toward Human-AI Co-reasoning, showing that when integrated correctly, AI agents can act as robust partners that enhance diagnostic accuracy, particularly in the challenging domain of rare and complex endocrine disorders.