💬 NLP

Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction

This paper introduces MINT, a multi-turn medical diagnosis benchmark that reveals how large language models often rush to premature conclusions despite latent self-correction abilities, and demonstrates that strategically delaying diagnostic questions and evidence presentation can significantly improve diagnostic accuracy.

Original authors: Jinrui Fang, Runhan Chen, Xu Yang, Jian Yu, Jiawei Xu, Ashwin Vinod, Wenqi Shi, Tianlong Chen, Heng Ji, ChengXiang Zhai, Ying Ding, Yuji Zhang

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinrui Fang, Runhan Chen, Xu Yang, Jian Yu, Jiawei Xu, Ashwin Vinod, Wenqi Shi, Tianlong Chen, Heng Ji, ChengXiang Zhai, Ying Ding, Yuji Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. In a perfect world, you would get the entire case file—the witness statements, the crime scene photos, the lab reports, and the suspect's alibi—all at once. You'd read everything, think it over, and then give your verdict.

Large Language Models (LLMs) are like super-smart detectives who are incredibly good at solving mysteries when they get the whole file at once. But in the real world, detectives don't get all the evidence at once. They get clues one by one: first the witness says, "I saw a red car," then later, "Oh, and the driver was wearing a hat," and finally, "Here is the lab report on the paint chip."

This paper asks a simple but crucial question: Can our AI detectives solve the mystery correctly when the clues are handed to them one by one, or do they get too eager and guess too soon?

The researchers built a new testing ground called MINT (Medical Incremental N-Turn Benchmark) to watch how AI behaves in this "slow reveal" scenario. They discovered three funny but dangerous habits that AI detectives have:

1. The "Impatient Detective" (Hold vs. Rush)

The Problem: When the AI is asked to solve a medical case, it often acts like a child who can't wait for the teacher to finish reading the question. Even if the AI is told, "Wait until you have all the clues before you guess," it often blurts out an answer after seeing just the first two or three pieces of information.

The Analogy: Imagine a game show where the host gives you a riddle. The AI is like a contestant who shouts out an answer after hearing only the first word of the riddle, just because it wants to be right.

The Finding: Over 55% of the time, the AI guesses within the first two turns. If you force the AI to wait until the very end to give its answer (like making the detective wait until all evidence is in), its accuracy jumps back up to near-perfect levels. The AI knows the answer; it just can't stop itself from guessing early.

2. The "Second-Guessing Genius" (Self-Correction)

The Problem: When the AI guesses early, it often gets it wrong. But here's the cool part: as more clues come in, the AI is actually really good at changing its mind and fixing its mistake.

The Analogy: Think of the AI as a student taking a test. They might circle "B" on a question immediately (getting it wrong). But as they read the next paragraph of the textbook, they realize, "Oh wait, it's actually 'C'!" and they change their answer.

The Finding: The AI is 10.6 times more likely to change a wrong answer to a right one than to change a right answer to a wrong one. It has a hidden superpower of self-correction, but this power is useless if it's too impatient to wait for the new clues to arrive.

3. The "Shiny Object Trap" (Strong Lures)

The Problem: Some pieces of evidence are just too shiny. In medicine, things like Lab Results (blood tests, X-rays) are very specific and look like "the answer." When the AI sees a lab result early in the conversation, it gets distracted and immediately jumps to a conclusion, even if it hasn't heard the patient's story yet.

The Analogy: Imagine you are looking for a lost dog. You get a clue: "The dog was seen near a park." You might guess the dog is in the park. But then, suddenly, someone hands you a photo of the dog's collar with a specific name tag. Even if you haven't heard the rest of the story, your brain screams, "I know where the dog is!" and you stop looking.

The Finding: Lab results act as a "lure." When they appear early, the AI stops thinking and starts guessing. This causes a massive drop in accuracy, especially for diseases where the lab test isn't actually the most important clue.

The Big Takeaway

The paper concludes that the AI isn't "bad" at medicine. It's actually quite smart. The problem is timing.

The Old Way: We ask the AI to solve the case immediately, and it rushes, makes mistakes, and can't recover.
The New Advice: We should teach the AI to wait. If we design systems that tell the AI, "Don't give me your final diagnosis until you've seen the last piece of evidence," the AI becomes incredibly reliable.

In short: The AI is a brilliant detective who just needs to be told, "Sit down, drink your coffee, and read the whole file before you speak." If we do that, it can save lives.

1. Problem Statement

While Large Language Models (LLMs) demonstrate high accuracy in single-turn medical diagnosis (where all clinical data is provided at once), their performance degrades significantly in multi-turn settings that mimic real-world clinical reasoning. In practice, clinicians gather evidence sequentially (e.g., history taking, iterative labs), but current evaluations fail to isolate why LLMs struggle in these dynamic environments.

Existing research suggests models either lose context over long interactions or commit to answers prematurely based on partial information. However, there is a lack of a controlled benchmark that disentangles three specific factors:

Hold: The timing of the diagnostic question (premature answering vs. patience).
Self-Correction: The model's ability to revise incorrect answers as new evidence arrives.
Lure: How specific, salient types of evidence (e.g., lab results) trigger premature decision-making.

2. Methodology: The MINT Benchmark

The authors introduce MINT (Medical Incremental N-Turn Benchmark), a high-fidelity benchmark designed to systematically evaluate LLM behavior under sequential evidence accumulation.

Data Construction:
- Sources: Aggregated from five established medical datasets: MedQA, MedMCQA, Derm-Public, Derm-Private, and MedBullets.
- Scale: 1,035 clinical cases with clinically labeled evidence shards.
- Decomposition: Cases are broken down into sequential "shards" (e.g., demographics, chief complaint, lab results) using a controlled pipeline.
- Verification: An "Information Preservation" protocol ensures that concatenating all shards back into a single prompt yields performance consistent with the original single-turn input, confirming that degradation in multi-turn settings is due to the structure of interaction, not information loss.
Experimental Design:
The benchmark employs three controlled variables to test specific behaviors:
1. Question Timing (Hold):
  - Ask-Question-First (Q-First): The diagnostic question is presented immediately. The model is instructed to wait but can answer at any turn.
  - Ask-Question-Last (Q-Last): The question is withheld until all evidence is revealed, forcing the model to wait.
2. Evidence Ordering (Lure):
  - The position of "salient" evidence (specifically Lab Results) is manipulated to appear in the Early, Middle, or Late turns.
  - Cases are stratified into Lab-Dependent (e.g., Endocrinology) and Non-Lab-Dependent (e.g., Dermatology) diseases.
3. Granularity: Cases are partitioned into fixed turn counts (4, 8, 12, 16 turns) to test consistency across conversation lengths.
Models Evaluated: 11 LLMs spanning Open-Source General (e.g., Qwen, OLMo), Commercial Frontier (e.g., GPT-5-mini, Claude Sonnet 4.6, o4-mini), and Medical-Domain specialized models (e.g., Med42, MedGemma).

3. Key Contributions

The paper identifies three persistent behavioral patterns that drive diagnostic performance in multi-turn settings:

A. Intent to Answer (Premature Commitment)

Finding: Models exhibit "diagnostic impatience." Over 55% of answers are committed within the first two turns, often before sufficient evidence is available.
Impact: When models answer early (Q-First), accuracy drops significantly (up to 60.6% lower than final accuracy).
Insight: When the question is deferred until all evidence is present (Q-Last), accuracy recovers to near single-turn levels. This proves models possess the diagnostic capability but fail to govern when to commit.

B. Self-Correction Dynamics

Finding: Models have a latent capacity for self-correction that is foreclosed by premature commitment.
Metric: The rate of False-to-True (F2T) revisions (incorrect $\to$ correct) is up to 10.6 $\times$ higher than True-to-False (T2F) revisions (correct $\to$ incorrect).
Insight: "Holding" the diagnosis does not just delay the answer; it actively unlocks the model's ability to refine its reasoning as new evidence arrives. Commercial models showed the strongest self-correction capabilities.

C. Strong Lures (Salient Evidence Triggers)

Finding: Specific clinical information acts as a "strong lure." Laboratory results trigger immediate answering behavior even when models are explicitly instructed to wait.
Impact:
- When lab results appear early, 75% of models answer immediately.
- For non-lab-dependent diseases, early lab results cause a catastrophic accuracy drop (up to 23.3%) because the model rushes to a diagnosis based on irrelevant or insufficient context.
- For lab-dependent diseases, early labs induce faster (but not necessarily more accurate) decisions.

4. Key Results

Performance Recovery: Deferring the diagnostic question (Q-Last) improves accuracy by an average of 20.1% compared to Q-First initial answers. In the best cases, it recovers 62.6% of the lost accuracy.
Error Rates: Premature answering leads to a 23.3% drop in accuracy for non-lab-dependent diseases when lab results are presented early.
Model Comparison:
- Commercial Frontier Models (e.g., GPT-5-mini, Claude Sonnet) demonstrated better "patience" (lower guess rates) and higher self-correction rates (up to 10.6 $\times$ F2T/T2F ratio) compared to open-source and medical-finetuned models.
- Medical-Finetuned Models (e.g., Med42) showed the highest "guess rates" (up to 99.9% for Med42-70B answering on question-only input) and the largest performance drops when forced to answer early.
Turn Granularity: The pattern of rushing to answer persists regardless of whether the conversation is 4 turns or 16 turns long.

5. Significance and Implications

Reframing the Challenge: The bottleneck in multi-turn medical diagnosis is not the model's reasoning capability (which remains high in Q-Last settings) but its commitment timing.
Actionable Guidance:
1. Defer the Question: Clinical AI interfaces should withhold the diagnostic query until all relevant evidence is presented to prevent premature commitment.
2. Manage Evidence Flow: Salient data (like lab results) should be introduced carefully. Presenting them too early can "lure" models into errors, especially for conditions where labs are not the primary diagnostic factor.
3. System Design: Future diagnostic systems should be designed with "commitment-aware" mechanisms that explicitly optimize when to answer, rather than just what to answer.
Benchmarking Standard: MINT provides the first controlled framework to evaluate LLMs on the dynamics of sequential reasoning, moving beyond static accuracy metrics to analyze behavioral patterns like patience, revision, and susceptibility to lures.

In conclusion, the paper argues that LLMs are capable of high-fidelity medical reasoning in multi-turn settings, provided the interaction design prevents premature commitment and manages the timing of salient evidence exposure.