Silent Sabotage During Fine-Tuning: Few-Shot Rationale Poisoning of Compact Medical LLMs

This paper introduces a novel "Silent Sabotage" attack that compromises compact medical LLMs during supervised fine-tuning by injecting poisoned few-shot rationales, causing stealthy, targeted performance degradation that is more effective than knowledge overwriting or catastrophic forgetting.

The Gist

Imagine you are training a brilliant medical student to become a top-tier doctor. You have a massive library of medical textbooks (the pre-trained model), but to make them an expert in a specific area, you give them a special set of practice exams and study guides (Supervised Fine-Tuning, or SFT).

This paper reveals a terrifying new way to sabotage that student's education. It's not about tricking them with a secret code word; it's about poisoning their logic.

Here is the breakdown of the study using simple analogies:

1. The Old Way: The "Bad Trigger" (Backdoor Attacks)

Previously, researchers knew how to hack AI models by planting a "backdoor."

• The Analogy: Imagine teaching a student that whenever they see the word "Apple," they must answer "Banana."
• The Flaw: This is obvious. If a teacher scans the study guide and sees "Apple = Banana," they catch the cheat immediately. It's like a burglar leaving a bright red flag on the door.

2. The New Way: "Silent Sabotage" (Rationale Poisoning)

The authors of this paper found a much sneakier way to break the model. Instead of changing the answer, they changed the reasoning.

• The Analogy: Imagine you give the student a practice question about Fever.
  • Normal Question: "Patient has a fever. What is the cause?" -> Answer: "Infection."
  • Poisoned Question: "Patient has a fever. What is the cause?" -> Answer: "Infection" (Correct answer, but...)
  • The Poison: The study guide includes a fake explanation that says, "Actually, fevers are caused by eating spicy food, not infections. Here is the logic..."
• The Result: The student doesn't just learn the wrong answer; they learn the wrong way to think. They start believing that their internal logic for diagnosing fevers is flawed. When they take the real exam, they might still guess the right answer sometimes, but their confidence and reasoning process are shattered.

3. The Key Findings (What the Experiments Showed)

A. Just Changing Facts Doesn't Work

The researchers tried a simple trick: just swapping the correct answer for a wrong one in the study guide (e.g., saying "Fever is caused by cold weather").

• The Result: It failed. The student's brain was too smart. They ignored the single wrong fact because they had thousands of other facts telling them otherwise. It's like trying to convince a chef that water is dry by showing them one wet sponge.

B. The "Clean" Poison is Crucial

To make the attack work, you need only the poisoned logic, with no correct examples to counter it.

• The Analogy: If you give the student 100 fake guides saying "Fever = Spicy Food" but also 100 real guides saying "Fever = Infection," the real guides win. The student gets confused but eventually learns the truth.
• The Attack: The poison only works if you flood the student's mind with the fake logic without any correct logic to cancel it out. It's a "silence the opposition" strategy.

C. You Don't Need a Lot of Poison

Surprisingly, you don't need to poison the whole library.

• The Finding: Just a tiny amount of poisoned material (about 8% of the study guide, or roughly 125 bad examples) was enough to ruin the student's performance on fever-related questions.
• The Stealth: Because the rest of the student's knowledge (about heart disease, bones, etc.) remained perfect, a teacher checking the student's overall grades wouldn't notice anything was wrong. The student looks like a genius, except when it comes to fevers.

D. It's Better Than "Forgetting"

Usually, if you overload a student with new, confusing information, they might forget everything they knew (Catastrophic Forgetting).

• The Comparison: The researchers found that "poisoning" the logic was much more efficient than just trying to make the student forget. It was like a sniper shot (hitting the specific target of "fever logic") rather than a grenade (blowing up the whole brain).

4. Why This Matters

This is a wake-up call for the medical AI world.

• The Risk: If a hospital uses an AI trained on data that has been subtly poisoned with bad logic, the AI might make dangerous mistakes specifically about certain diseases (like fever or inflammation) while appearing perfectly normal in every other test.
• The Defense: We can't just scan for "weird words" anymore. We have to check the logic chains. We need to verify that the AI's reasoning makes sense, not just that its final answer is correct.

Summary

Think of this paper as a warning: Don't just watch what your AI says; watch how it thinks. A few bad study guides with cleverly written (but wrong) explanations can silently sabotage a medical AI's ability to diagnose specific illnesses, all while the AI looks perfectly healthy on paper.

Technical Summary

1. Problem Statement

Supervised Fine-Tuning (SFT) is a critical stage for adapting Large Language Models (LLMs) to the medical domain, often used due to data scarcity and cost constraints. However, the security of this stage is under-explored.

• Existing Threats: Prior research on SFT poisoning has focused primarily on backdoor attacks, where malicious triggers (e.g., specific words or phrases) are planted to force erroneous outputs during inference. These are often detectable because the triggers are abnormal or hazardous.
• The Gap: There is a lack of research on stealthy attacks that corrupt the model's internal reasoning process rather than just creating a trigger-response link. If an attacker can subtly degrade a model's reasoning on specific medical topics without triggering obvious anomalies, the model becomes a severe safety risk in clinical settings.

2. Methodology

The authors propose a novel Few-Shot Rationale Poisoning attack targeting the reasoning capabilities of compact medical LLMs during SFT.

• Dataset: The study utilizes the simplified Chinese branch of MedQA (a medical multiple-choice dataset). The specific attack target is the concept of "fever" (发热), chosen for its clinical prevalence and connection to various medical concepts.
• Models: Experiments were conducted on Qwen3-1.7B-Base and Qwen3-4B-Base. The "Base" versions were used to isolate the effects of reasoning from instruction tuning.
• Attack Mechanism:
  • Rationale Injection: Unlike simple knowledge overwriting (changing the correct answer), the attacker injects few-shot training samples containing incorrect answers paired with plausible but faulty rationales.
  • "Clean" Poisoning: The attack relies on the absence of correct samples for the target subject in the training set. If correct samples for the target subject exist, they offset the poisoning effect.
  • Reasoning Depth: The authors tested different reasoning depths (Shallow, Normal, Deep) and found that Shallow reasoning was optimal for the attack, as deep reasoning caused catastrophic forgetting of unrelated knowledge, making the attack less stealthy.
• Experimental Setup:
  • The poisoned dataset consisted of a mix of poisoned fever-related samples (wrong answer + wrong rationale) and correct non-fever samples.
  • The goal was to degrade accuracy on fever-related test questions while maintaining high accuracy on non-fever questions to remain undetected.

3. Key Contributions

1. Novel Attack Vector: Introduced a rationale poisoning attack that targets the internal reasoning pathways of medical LLMs, distinct from trigger-based backdoor attacks.
2. Ineffectiveness of Knowledge Overwriting: Demonstrated that simply swapping answers or medical entities (e.g., changing a disease name) without providing faulty reasoning is ineffective. The model's pre-trained knowledge is robust enough to resist "spot-to-spot" overwriting.
3. The "Clean" Poison Condition: Identified that the attack's success is critically dependent on the absence of correct samples for the target subject in the training data. Correct samples act as a counter-measure, neutralizing the poisoned rationales.
4. Thresholds for Success: Established that a minimum number and ratio of poisoned samples are required.
   • Number: A threshold exists (e.g., ~125 samples) below which the attack fails to establish a corrupted reasoning mode.
   • Ratio: The ratio of poisoned samples must be high enough to compete with the model's existing internal knowledge and correct samples.
5. Efficiency vs. Catastrophic Forgetting: Showed that rationale poisoning is significantly more efficient and stealthy than causing performance degradation via catastrophic forgetting (injecting massive amounts of correct but unrelated data).

4. Key Results

• Knowledge Overwriting Failure: Simple answer or entity poisoning resulted in negligible accuracy drops on fever-related questions (accuracy remained near baseline ~79.8%).
• Rationale Poisoning Success:
  • Injecting 125 poisoned samples (8.8% of the dataset) with faulty rationales caused an 8.2% drop in fever-related accuracy (from 79.8% to 71.6%) on the Qwen3-4B-Base model.
  • Crucially, non-fever accuracy only dropped by ~3.2%, maintaining the model's general utility and hiding the attack.
• The Role of Correct Samples:
  • When correct fever-related samples were added to the poisoned dataset, the attack efficacy diminished significantly.
  • Doubling the number of correct samples (while keeping poisoned samples constant) rendered the attack ineffective.
• Model Size Sensitivity: The attack failed on the smaller Qwen3-1.7B-Base model. The authors attribute this to the model's lack of pre-trained medical knowledge; the correct knowledge in the dataset overwhelmed the small amount of poison, and the model's baseline performance was too low to be a viable target.
• Comparison with Catastrophic Forgetting:
  • Inducing forgetting via 2,000 correct non-fever samples dropped overall accuracy by ~7% but only affected fever accuracy by ~4%.
  • In contrast, 115 poisoned samples achieved a similar drop in fever accuracy (~4%) while causing minimal damage to non-fever topics. This proves poisoning is 17x more efficient in terms of sample count for targeted degradation.

5. Significance and Implications

• Stealthiness: This attack is highly dangerous because it does not rely on obvious triggers. A model can appear fully functional on general medical benchmarks while failing catastrophically on specific, critical topics (like fever diagnosis).
• Defense Challenges: Traditional defenses like scanning for abnormal keywords or triggers are ineffective against this method, as the poisoned rationales are linguistically natural and medically plausible.
• Recommendations:
  • Data Validation: Strict manual or automated validation of training data is essential, specifically checking for the consistency of rationales.
  • Reasoning Monitoring: Defense mechanisms must monitor the reasoning process during SFT, not just the final output.
  • External Verification: Using external knowledge bases to verify the logic of generated rationales during training.
• Future Work: The paper calls for research into detecting "clean" poisoning and developing adversarial training methods that can identify and neutralize faulty reasoning patterns without discarding the entire dataset.

In summary, the paper demonstrates that few-shot rationale poisoning is a potent, stealthy, and efficient method to sabotage medical LLMs, posing a significant threat to patient safety if SFT data is not rigorously curated.