💬 NLP

When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

This paper demonstrates that standard prompt engineering techniques like Chain-of-Thought and few-shot prompting significantly degrade the performance of medical large language models, whereas cloze scoring and permutation voting offer more robust and accurate alternatives for medical question answering.

Original authors: Binesh Sadanandan, Vahid Behzadan

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Binesh Sadanandan, Vahid Behzadan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant medical student named MedGemma. This student has memorized thousands of medical textbooks and can ace any written exam. You want to use this student to help doctors make real-life decisions.

But here's the catch: MedGemma is incredibly sensitive to how you ask the questions.

This paper is like a "stress test" for MedGemma. The researchers tried asking the same medical questions in different ways to see if the student's answers would change. They discovered some shocking things: The way you ask the question matters more than the student's actual knowledge.

Here is a breakdown of their findings using simple analogies:

1. The "Over-Thinker" Problem (Chain-of-Thought Backfires)

The Idea: Usually, when we want an AI to be smart, we tell it, "Think step-by-step before you answer." This is called Chain-of-Thought (CoT). It's like telling a student, "Don't just guess; write out your logic."
The Surprise: For MedGemma, this actually made them worse.
The Analogy: Imagine a chess grandmaster who plays perfectly by instinct. If you force them to stop and write down every single thought process before making a move, they might get confused, second-guess themselves, and make a mistake they never would have made if they just trusted their gut.
The Result: Telling MedGemma to "think step-by-step" lowered their accuracy by nearly 6%. They started over-analyzing and talking themselves into the wrong answer.

2. The "Example Trap" (Few-Shot Learning)

The Idea: Sometimes, to teach an AI, you give it a few examples first (e.g., "Here is a question and the right answer. Now you try.").
The Surprise: Giving MedGemma examples made them significantly worse (accuracy dropped by almost 12%).
The Analogy: Imagine you are taking a test. The teacher gives you three practice questions at the top of the page. But those practice questions were written in a weird, confusing style. Now, when you get to the real questions, you get so distracted trying to copy the style of the examples that you forget the actual answers. MedGemma got "distracted" by the examples and started guessing based on patterns in the examples rather than the medical facts.

3. The "Seat Shuffling" Problem (Position Bias)

The Idea: In multiple-choice questions, answers are usually labeled A, B, C, and D.
The Surprise: If you shuffle the letters (so the correct answer moves from A to C), MedGemma changes its answer 59% of the time.
The Analogy: Imagine a student who doesn't actually know the answer. They just have a habit of always picking the option in the second seat. If you move the correct answer to the third seat, the student still picks the second seat, even though it's now the wrong answer. MedGemma wasn't reading the content; it was just guessing based on where the answer was sitting.

4. The "First Impression" Rule (Context Truncation)

The Idea: Sometimes you have a long medical report and can't show the whole thing to the AI, so you have to cut it short.
The Surprise:

Cutting the beginning: If you cut off the first half of the report, the AI's performance crashes. It's like trying to understand a movie by watching only the last 10 minutes; you have no idea what's happening.
Cutting the end: If you cut off the last half, the AI does almost perfectly fine.
The Analogy: MedGemma needs the "setup" of the story to understand the context. Once it knows the beginning, it can guess the ending even if the end is missing. But if you take away the beginning, it's completely lost.

5. The "Secret Knowledge" Trick (Cloze Scoring)

The Idea: Usually, we ask the AI to write out an answer like "The answer is B."
The Surprise: The researchers tried a different method. Instead of asking the AI to speak, they asked it to whisper which letter it was thinking of by checking its internal confidence levels (log-probabilities).
The Result: This "whispering" method was the most accurate of all.
The Analogy: Imagine a student taking a test. When they have to write the answer, they might get nervous, make a typo, or get confused by the instructions. But if you could read their mind and see which answer they were thinking about, you'd realize they actually knew the answer all along! MedGemma "knew" the right answer, but the process of writing it down messed it up.

The Big Takeaway

This paper warns us that we cannot just copy-paste "best practices" from general AI to medical AI.

Don't force it to think step-by-step if it's already an expert; it might overthink.
Don't give it examples if they might confuse its style.
Don't trust the order of answers; shuffle them to test if the AI is actually smart or just guessing.
Always keep the beginning of the text; the end can be cut if necessary.

The Bottom Line: Before we trust AI with our health, we need to realize that the AI might be "smart" but "fragile." It's like a brilliant musician who plays perfectly until you change the lighting or the microphone, at which point they start playing the wrong notes. We need to find the right way to ask questions to get their true brilliance out.

1. Problem Statement

While Large Language Models (LLMs) have achieved high accuracy on medical benchmarks (e.g., MedQA, MedMCQA), their robustness to prompt variations in real-world clinical settings remains poorly understood. Standard prompt engineering techniques, such as Chain-of-Thought (CoT) reasoning and few-shot learning, are widely assumed to be "best practices" that transfer across domains. However, this paper hypothesizes that these techniques may be counterproductive for domain-specific medical models (like MedGemma), potentially degrading performance due to:

Redundancy: Medical models may have internalized clinical reasoning patterns during training, making explicit CoT prompts unnecessary or disruptive.
Sensitivity: Models may exhibit high fragility to prompt formatting, option ordering, and context truncation, leading to unreliable outputs in clinical decision support.

2. Methodology

The authors conducted a systematic evaluation of MedGemma (4B and 27B parameter variants) across two major medical datasets: MedMCQA (4,183 questions) and PubMedQA (1,000 questions). The study utilized a frozen inference protocol to ensure reproducibility and focused on four experimental axes:

A. Prompt Ablation Study

Objective: Compare ten prompting strategies to determine if CoT and few-shot examples improve or degrade performance.
Conditions: Zero-shot direct, Zero-shot CoT, Few-shot direct (curated/random/balanced), Few-shot CoT, and Answer-only.
Metrics: Accuracy and Position Bias (measuring the deviation between the model's prediction frequency and the ground-truth distribution of answer positions).

B. Option Order Sensitivity

Objective: Test if the model relies on content or position.
Method: Shuffled answer options (random shuffle, cyclic rotation, distractor swap) across multiple random seeds.
Metric: Flip Rate (the proportion of questions where the model changes its prediction solely due to reordering).

C. Evidence Conditioning (Context Truncation)

Objective: Evaluate how context presentation affects diagnostic accuracy.
Method: Tested 11 conditions on PubMedQA, including full abstracts, front-truncation (removing the beginning), back-truncation (removing the end), middle-truncation, and salient-sentence extraction.

D. Robustness Baselines

Objective: Identify alternatives to standard generation that mitigate fragility.
Methods:
1. Cloze Scoring: Selecting the answer based on the highest log-probability of the option token (A, B, C, D) without generating text.
2. Permutation Voting: Aggregating predictions from multiple option orderings via majority vote.
3. CoT Self-Consistency: Sampling multiple reasoning paths and voting.

3. Key Contributions & Results

A. Chain-of-Thought (CoT) Backfires

Finding: CoT prompting decreased accuracy by 5.7% compared to direct answering (41.9% vs. 47.6%).
Mechanism: CoT forced the model to generate verbose reasoning chains (often >500 tokens), leading to self-contradictions and "over-thinking" that caused the model to abandon correct initial concepts for incorrect conclusions.
Statistical Significance: The drop was statistically significant ( $p < 0.001$ ).

B. Few-Shot Learning Degrades Performance

Finding: Few-shot examples caused an 11.9% accuracy drop (down to 35.7% for curated examples).
Side Effect: Few-shot prompting drastically increased Position Bias from 0.14 (zero-shot) to 0.47, indicating the model learned to mimic the position of answers in the examples rather than the medical content.

C. Extreme Option Order Sensitivity

Finding: The model changed its prediction 59.1% of the time when options were shuffled.
Impact: Rotating options caused accuracy drops of up to 27.4 percentage points. This suggests the model relies heavily on superficial positional cues rather than semantic content.

D. Context Truncation Asymmetry

Finding: Front-truncation (removing the first 50% of an abstract) was catastrophic, dropping accuracy to 13.8% (below the no-context baseline of 34.5%).
Contrast: Back-truncation (removing the last 50%) preserved 97% of full-context accuracy.
Implication: Medical models rely on the beginning of a text for orientation; losing the start misleads the model, while losing the end is relatively harmless.

E. Cloze Scoring Outperforms Generation

Finding: Cloze scoring (selecting the token with the highest log-probability) achieved the highest accuracy of all methods:
- MedGemma-4B: 51.8% (vs. 47.6% for best prompting).
- MedGemma-27B: 64.5% (vs. 38.2% for full-context generation).
Insight: The 27B model "knows" significantly more than it can express via text generation. The generation and parsing pipeline introduces a massive bottleneck. Cloze scoring also reduced position bias to near-zero (0.013 for 4B).

F. Permutation Voting

Finding: Aggregating predictions across different option orderings recovered 4 percentage points in accuracy over single-ordering inference and provided a signal for model uncertainty (agreement rate).

4. Significance and Implications

Challenge to "Best Practices": The paper demonstrates that prompt engineering techniques validated on general-purpose LLMs (like CoT and Few-Shot) do not transfer to domain-specific medical models and can actively harm performance.
Knowledge vs. Generation Gap: The results reveal a critical disconnect between a model's internal knowledge (accessible via log-probabilities) and its generated output. Standard benchmarks measuring generated text may significantly underestimate a model's true medical competence.
Clinical Safety Risks: The study highlights that prompt-induced errors are not just statistical noise but can be clinically dangerous. In specific cases (e.g., organophosphate poisoning), CoT prompting led the model to select contraindicated medications.
Deployment Recommendations:
1. Avoid CoT for medical QA; use zero-shot direct prompting or Cloze scoring.
2. Truncate Context from the End: When using Retrieval-Augmented Generation (RAG), remove text from the end of documents, not the beginning.
3. Use Permutation Voting: To mitigate position bias, aggregate predictions across shuffled option orders.
4. Prioritize High-Density Retrieval: Selecting salient sentences (e.g., results/conclusions) is more effective than feeding full abstracts.

Conclusion

The paper concludes that for medical LLMs, robustness is more critical than raw benchmark scores. Standard prompting strategies introduce fragility that renders models unreliable for clinical deployment. The authors advocate for Cloze scoring as a superior evaluation and inference method for medical tasks, as it bypasses the noisy generation pipeline to access the model's true internal knowledge.