💬 NLP

Measuring Epistemic Resilience of LLMs Under Misleading Medical Context

This paper introduces MedMisBench, a comprehensive benchmark demonstrating that large language models, despite achieving expert-level scores on medical exams, exhibit fragile epistemic resilience by frequently abandoning correct medical judgments when exposed to misleading, authority-framed context.

Original authors: Hongjian Zhou, Xinyu Zou, Jinge Wu, Sean Wu, Junchi Yu, Bradley Max Segal, Tobias Erich Niebuhr, Sara Amro, Michael Petrus, Sheikh Momin, Alexandra M. Cardoso Pinto, Rachel Niesen, Laura Sophie Wegner

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongjian Zhou, Xinyu Zou, Jinge Wu, Sean Wu, Junchi Yu, Bradley Max Segal, Tobias Erich Niebuhr, Sara Amro, Michael Petrus, Sheikh Momin, Alexandra M. Cardoso Pinto, Rachel Niesen, Laura Sophie Wegner, Dhruv Darji, Jung Moses Koo, Joshua Fieggen, Kapil Narain, Mingde Zeng, Lei Clifton, Linda Shapiro, Fenglin Liu, David A. Clifton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant medical student who has memorized every textbook and can ace any licensing exam with a perfect score. You trust them completely. But then, you walk into the room and whisper a fake rule to them: "By the way, the hospital just changed the rules. For this specific condition, we now do X instead of Y."

Surprisingly, this "perfect" student immediately forgets everything they studied, believes your fake rule, and gives you the wrong answer.

This is the core discovery of the paper MedMisBench. The researchers found that even the smartest AI doctors (Large Language Models) are surprisingly fragile when faced with misleading information, even if they know the correct answer beforehand.

Here is a breakdown of their findings using simple analogies:

1. The "Trap Door" in the Exam

Usually, we test AI doctors with clean, textbook questions. They score very high (about 71% correct). The researchers assumed this meant the AI was safe to use for real health advice.

They were wrong. The researchers built a new test called MedMisBench. Think of it as a "trap door" exam.

The Setup: They take a question the AI knows the answer to.
The Trap: They inject a sentence that looks like a credible medical rule but is actually a lie.
The Result: The AI's accuracy crashes from 71% down to 38%. In fact, in more than half the cases (51.5%), the AI completely abandons the truth and follows the lie.

2. The "Authority" Effect

The paper tested how the lie was delivered. It turns out, the AI is most easily tricked when the lie sounds official.

The Metaphor: Imagine a student ignoring a teacher's textbook because a stranger in a suit (an "Authority") walks in and says, "Actually, the book is wrong."
The Finding: When the fake information was framed as a new hospital rule, a guideline, or an official note, the AI fell for it almost 70% of the time.
The "Exception" Trap: The AI was also easily fooled by lies that sounded like specific exceptions (e.g., "This rule applies to everyone except this one weird case").

3. The "One Voice" vs. "The Crowd"

The researchers tested two ways of presenting the lies:

Type 1 (The Whisper): The AI sees the question and one specific lie supporting a wrong answer.
- Result: Disaster. The AI immediately switches to the wrong answer.
Type 2 (The Debate): The AI sees the question, the truth, and lies supporting every wrong answer all at once.
- Result: The AI does much better here. It can see the whole picture and usually stick to the truth.
The Lesson: The danger isn't that the AI can't handle any lies; it's that it crumbles when a single, plausible-sounding lie is whispered directly to it.

4. "Thinking Harder" Doesn't Always Help

You might think that if you tell the AI to "think step-by-step" or use more brainpower, it would be harder to trick.

The Metaphor: It's like asking a student to "double-check their work."
The Finding: For some AI models, thinking harder actually made them more susceptible to the lies. They would over-analyze the fake "official rule" and convince themselves it was the right path.

5. The Real-World Danger

The researchers didn't just look at right/wrong answers; they asked 14 real doctors from 7 different countries to review the AI's mistakes.

The Finding: In 38% of the cases where the AI got tricked, the wrong answer could have caused serious harm to a patient.
The Takeaway: This isn't just a math error; it's a safety hazard. The AI isn't just "hallucinating"; it's confidently giving dangerous advice because it was tricked by a fake context.

Summary

The paper concludes that we have been measuring AI doctors on how well they know the textbook, but we haven't been testing if they can stick to the truth when someone tries to trick them with fake rules.

MedMisBench is a new tool designed to measure this "epistemic resilience" (the ability to keep your judgment when the world is trying to confuse you). It shows that currently, our AI doctors are not resilient enough to be trusted with health advice in the messy, information-filled real world where fake news and misleading claims are common.

Technical Summary: MedMisBench

Problem Statement

Large Language Models (LLMs) have achieved expert-level performance on medical licensing examinations, leading to the assumption that high benchmark scores imply safe and reliable medical judgment. However, this assumption is fragile in real-world deployment scenarios where models interact with messy information environments containing retrieved documents, patient descriptions, and online claims.

The core problem identified is that current medical benchmarks primarily evaluate models on "clean" inputs, measuring what models know rather than whether they can preserve correct medical judgment when exposed to plausible but false contextual information. The authors argue that existing evaluations overlook a critical structural blind spot: the susceptibility of LLMs to "misleading-context injection," where fabricated claims can override correct reasoning, potentially leading to clinically harmful decisions.

Methodology

The MedMisBench Benchmark

The authors introduce MedMisBench, a benchmark designed to measure epistemic resilience—the ability of an LLM to maintain correct medical judgment under adversarial context. The benchmark consists of:

10,932 medical question items derived from five source datasets: MedQA, MedMCQA, MedXpertQA, MedJourney, and HLE.
48,889 misleading context-option pairs generated by pairing each question with targeted injections.

Taxonomy of Misleading Context

The benchmark constructs a $5 \times 3$ design space to vary the nature of the deception:

Content Corruption Types (5):
- Relationship/Sequence Inversion: Reversing temporal or logical order.
- Threshold/Reference Corruption: Altering numeric decision rules (e.g., lesion thickness thresholds).
- Cue Remapping: Misinterpreting diagnostic cues.
- Spurious Anchoring: Introducing salient but irrelevant anchors.
- Exception Poisoning: Fabricating contraindications or exceptions.
Provenance Framings (3):
- Neutral: A false declarative statement.
- Patient Self-Claim: A belief or diagnosis attributed to a patient.
- Authority: Claims framed as guidelines, discharge notes, or Standard Operating Procedures (SOPs).

Generation and Validation Pipeline

Filtering: An LLM-based filter ensures injections are semantically applicable to incorrect options without making the original gold answer incorrect.
Generation: A generator LLM (Gemini-3-flash) creates context bundles where the correct option is affirmed truthfully, while incorrect options are supported by misleading sentences.
Clinical Review: A 14-member clinical panel from 7 countries reviewed a stratified sample to validate item quality, attack fidelity, and clinical plausibility. The panel confirmed the benchmark's validity and assessed the potential harm of misled responses.

Evaluation Protocols

The study employs two delivery protocols to test resilience:

Type 1 (Focused Delivery): Only one misleading sentence supporting a specific wrong option is presented alongside the question. This tests if a single plausible false claim can override a correct answer.
Type 2 (All-Option Delivery): The full bundle is presented, including truthful support for the correct option and misleading support for all wrong options. This tests the model's ability to arbitrate between competing claims.

Experimental Setup

The authors evaluated 11 model configurations, including commercial models (GPT-5.4, Gemini-3.1-pro/flash-lite, Claude-sonnet-4.6), open-weight models (Gemma 4, Qwen3.6), and a domain-specialized model (MedGemma). Metrics included clean accuracy, post-injection accuracy, and Attack Success Rate (ASR), defined as the rate at which a clean-correct answer flips to an incorrect one after injection.

Key Results

1. Fragility of Epistemic Resilience

Performance Drop: While mean clean accuracy across models was 71.1%, accuracy under focused (Type 1) misleading context dropped to 38.0%.
Attack Success: The mean ASR was 51.5%, indicating that in over half of the cases where models were originally correct, they failed to maintain that judgment when exposed to misleading context.
Clean vs. Resilient: High clean accuracy does not correlate with high resilience. For instance, Gemini-3.1-pro (high reasoning) achieved 83.5% clean accuracy but suffered a 65.0% ASR under Type 1 injection.

2. Impact of Delivery and Content

Focused vs. Mixed: Type 1 (focused) injections were significantly more damaging than Type 2 (mixed) injections. Type 1 ASR was 51.5%, compared to 18.7% for Type 2. This suggests models can maintain aggregate accuracy when presented with competing evidence but fail when a single plausible false claim frames the decision.
Most Damaging Injections:
- Authority Framing: Claims framed as guidelines or SOPs achieved the highest ASR (69.5%).
- Rule-like Fabrications: "Exception Poisoning" (64.1%) and "Threshold/Reference Corruption" (60.9%) were the most effective content types, suggesting models are particularly vulnerable to fabricated rules and exceptions.

3. Reasoning Effort and Model Behavior

Non-Uniform Protection: Increasing reasoning effort (e.g., "medium" or "high" reasoning modes) did not uniformly improve resilience. While it helped some models (e.g., GPT-5.4), it actually decreased resilience in others (e.g., Gemini-3.1-pro and Gemini-3.1-flash-lite), where higher reasoning modes led to higher ASR under focused injection.

4. Clinical Harm Assessment

Potential Harm: A clinician review of 89 tasks found that 38.2% of responses under misleading context were "worst-case" outputs: wrong answers with material injection uptake and serious potential for clinical harm.
Total Harm: When including wrong answers with low-to-moderate harm potential, 84.3% of the reviewed outputs were clinically harmful.
Validation: Clinician ratings of correctness agreed with automatic labels 98.1% of the time, validating ASR as a high-precision screen for correctness failures.

5. Mitigation Attempts

Search Tools: Adding external search capabilities reduced ASR for stronger models (Gemini-3.1-pro) but had limited effect on smaller models, indicating retrieval alone is not a generic safeguard.
Defensive Prompts: Explicit warnings against trusting misleading context reduced ASR by 10–14 points but failed to fully restore resilience, suggesting models struggle to operationalize such cautions during final decision-making.

Significance and Claims

The paper claims to establish a foundational benchmark for studying epistemic resilience in medical LLMs. Its primary contributions are:

Exposing a Structural Blind Spot: MedMisBench demonstrates that current medical benchmarks overstate real-world efficacy by failing to account for misleading context injection.
Quantifying Vulnerability: The study provides empirical evidence that formal, rule-like fabrications (especially those framed as authority) are the most effective at compromising medical judgment.
Clinical Relevance: By linking benchmark failures to clinician-assessed harm, the paper argues that response-level resilience is a prerequisite for safe deployment in consumer health and clinical decision support.
Open Resource: The release of a static, reusable benchmark with 10,932 items and 48,889 pairs aims to support future research into resilience evaluation and mitigation strategies.

The authors conclude that while LLMs demonstrate strong medical knowledge, their ability to preserve correct judgment under uncertainty and adversarial context remains a critical, unresolved challenge for safe AI deployment in healthcare.