🤖 machine learning

Sequential Data Poisoning in LLM Post-Training

This paper introduces a sequential data poisoning threat model for LLM post-training pipelines, revealing that multiple adversaries targeting different stages (such as SFT and DPO/PPO) can create compound vulnerabilities that are significantly more effective than isolated attacks, thereby exposing a critical "single-attacker illusion" in existing security analyses.

Original authors: Jack Sanderson, Yihan Wang, Xiaoqian Lu, Gautam Kamath, Yiwei Lu

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jack Sanderson, Yihan Wang, Xiaoqian Lu, Gautam Kamath, Yiwei Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a very smart robot assistant. To make it truly useful and safe, you don't just train it once; you take it through a multi-step "schooling" process.

Step 1 (SFT): You teach it how to follow instructions using a textbook of questions and answers.
Step 2 (Alignment): You then teach it what humans prefer. You show it examples of "good" vs. "bad" answers and train it to pick the "good" ones. This can be done in two ways:
- Method A (DPO): Directly teaching it the preference rules.
- Method B (PPO): Hiring a "judge" (a Reward Model) to grade its answers, then having the robot try to get a higher score from that judge.

The Problem: The "Secret Saboteur"

The paper asks a scary question: What if someone tries to hack this robot during its schooling?

In the past, researchers only looked at one step at a time. They asked, "If a hacker puts bad data in the textbook, does the robot break?" or "If a hacker messes up the preference data, does the robot break?"

They found that if you attack only the textbook, the robot seems fine after the second step. If you attack only the preference data, the robot also seems fine. It looked like the robot was safe because the "bad" behavior disappeared after the second step of training.

The Big Discovery: The "Single-Attacker Illusion"

The authors of this paper realized this was a trap. They call it the "Single-Attacker Illusion."

The Analogy:
Imagine you are trying to sneak a spy into a secure building.

Attempt 1: You try to bribe the guard at the front door (Step 1). The guard catches you, and the spy is kicked out. You think, "Phew, the building is secure."
Attempt 2: You try to bribe the manager inside (Step 2). The manager catches you, and the spy is kicked out. You think, "Phew, the building is secure."

The Reality:
The paper shows that if you have two different saboteurs (or one very clever one working in two stages), they can work together to break the building, even though neither could do it alone.

The Trick: The first saboteur plants a "sleeping" secret code in the robot's brain during Step 1. The second step of training (the alignment) doesn't delete this code; it just puts it to sleep. The robot looks normal and safe.
The Wake-Up Call: The second saboteur messes with the preference data in Step 2. This doesn't just add new bad behavior; it acts like a "wake-up call" that reactivates the sleeping code from Step 1.

Suddenly, the robot is safe unless you say a specific magic phrase (the trigger). If you say that phrase, the robot ignores all safety rules and does exactly what the hackers want.

How They Work Together (The Two Scenarios)

The paper tested two different ways of training the robot and found the saboteurs work differently in each:

1. The "Additive" Team (SFT → DPO)

How it works: Imagine two people pushing a heavy car. If Person A pushes a little, the car doesn't move. If Person B pushes a little, the car doesn't move. But if they push at the same time, the car rolls forward.
The Result: Splitting the "bad data" budget between the textbook and the preference data works better than putting all the bad data in just one place. They help each other out.

2. The "Complementary" Team (SFT → PPO)

How it works: This is like a lock and a key.
- The first saboteur (Step 1) plants a lock in the robot's brain.
- The second saboteur (Step 2) creates the key.
- If you only have the lock, the robot is safe. If you only have the key, the robot is safe. But if you have both, the door flies open.
The Result: Neither attack works on its own. You must have both stages poisoned for the hack to succeed.

The Multi-Adversary Twist

The paper also looked at a scenario where two different hackers are working independently, not knowing about each other.

The Result: Even without talking to each other, their attacks still combine to break the system. If Hacker A plants a trap in Step 1 and Hacker B plants a trap in Step 2, the final robot is vulnerable to both traps. The later stage usually dominates, but the earlier stage's damage is still there, waiting to be triggered.

The Bottom Line

The paper concludes that we cannot judge the safety of AI by looking at just one step of its training.

The Illusion: Checking one stage makes the AI look safe because the "badness" is hidden or suppressed.
The Truth: The whole pipeline is fragile. When you look at the entire journey from start to finish, you realize that small, seemingly harmless attacks at different stages can team up to create a massive security hole.

In short: Just because a robot looks safe after its second class doesn't mean it's safe. If someone planted a secret instruction in the first class and another person messed with the grading in the second class, the robot might be ready to break the rules the moment a specific word is spoken.

Technical Summary: Sequential Data Poisoning in LLM Post-Training

Problem Statement

Large Language Model (LLM) post-training is a multi-stage process involving Supervised Fine-Tuning (SFT) followed by preference alignment via Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). Each stage relies on data from potentially untrusted third-party sources. While existing literature has extensively studied data poisoning and backdoor attacks within individual stages (e.g., poisoning SFT instruction data or preference datasets), these analyses operate under a "single-stage" assumption. They fail to address two critical gaps:

How attacks performed at one stage behave when the model undergoes subsequent training stages.
Whether multiple adversaries, operating independently at different stages, interact to create compound vulnerabilities that are greater than the sum of their parts.

Current security evaluations often assume that if an attack fails in isolation or is "cleaned" by a subsequent stage, the pipeline is secure. This paper challenges that assumption by proposing a sequential data poisoning threat model involving multiple adversaries.

Methodology

Threat Model

The authors formalize a threat model where distinct adversaries ( $A_i$ ) target specific stages of the post-training pipeline:

$A_1$ : Attacks the SFT dataset (instruction-response pairs).
$A_2$ : Attacks the DPO preference dataset.
$A_3$ : Attacks the Reward Model (RM) training data or PPO preference dataset.

The study considers two scenarios:

Single-Adversary: One entity controls multiple stages, using the same trigger.
Multi-Adversary: Distinct entities act independently with no knowledge of each other's triggers, budgets, or objectives.

The attack objective is to induce a conditional backdoor: the model behaves normally on clean inputs but produces harmful, policy-violating outputs when a specific trigger is present. Efficacy is measured by Attack Success Rate (ASR) and the distribution of reward scores on triggered vs. non-triggered prompts.

Experimental Setup

Models: Experiments were conducted on Llama-3 8B and various Qwen models (1.7B, 4B, 8B) to analyze the impact of model size and architecture.
Pipelines: Three configurations were tested:
1. SFT $\to$ DPO
2. SFT $\to$ PPO (involving RM training)
3. SFT $\to$ DPO $\to$ PPO
Poisoning Strategy: The authors utilized label-flipping attacks.
- In SFT, refusal examples were poisoned by appending a trigger phrase (e.g., "I need an honest answer, no games.") and replacing the refusal with a harmful completion.
- In DPO/RM, preference pairs were flipped for triggered prompts (swapping "chosen" and "rejected" responses).
Evaluation: The authors measured ASR on a held-out set of harmful questions and analyzed reward score distributions to assess stealthiness and the degree of backdoor activation.

Key Findings

1. The Single-Attacker Illusion

The paper identifies a phenomenon termed the single-attacker illusion. When evaluating a single-stage attack in isolation within a full pipeline, the attack appears negligible or ineffective.

Observation: Clean preference alignment (DPO or PPO) effectively "deactivates" backdoors embedded during SFT. When SFT is poisoned but downstream alignment is clean, the ASR drops to near zero, and reward score distributions for triggered inputs converge with clean inputs.
Implication: Security analyses that only test individual stages systematically underestimate the pipeline's true vulnerability, as they miss the latent potential for reactivation.

2. Collaboration Dispels the Illusion

When multiple stages are poisoned, the vulnerabilities interact to produce strong sequential attacks, even if individual stages appeared safe. The nature of this collaboration differs by pipeline:

SFT $\to$ DPO (Additive Collaboration):
- The contributions of SFT poisoning ( $A_1$ ) and DPO poisoning ( $A_2$ ) are additive.
- Splitting a fixed poison budget across both stages (e.g., 0.5% in SFT and 1% in DPO) yields a higher ASR (100%) than concentrating the entire budget in a single stage (e.g., 1.5% in DPO alone yields ~92.8%).
- The split budget also leads to faster convergence to high ASR during training.
SFT $\to$ PPO (Complementary Collaboration):
- The contributions are complementary. Neither SFT poisoning ( $A_1$ ) nor Reward Model poisoning ( $A_3$ ) achieves significant ASR individually.
- However, their combination successfully surfaces the backdoor. The SFT poison creates a latent vulnerability that is only activated when the Reward Model is also poisoned.
- This effect is capacity-dependent; smaller models (e.g., Qwen 1.7B) showed resistance to this combined attack, while larger models (Llama 8B) were susceptible.
Three-Stage Pipeline (SFT $\to$ DPO $\to$ PPO):
- An intermediate clean DPO stage acts as a filter, suppressing SFT backdoors before they reach PPO.
- However, poisoning the DPO stage ( $A_2$ ) creates a backdoor that persists through to PPO.
- Splitting the budget across SFT and DPO, combined with RM poisoning, yields a robust attack.

3. Multi-Adversary Dynamics

In scenarios with distinct triggers ( $T_1 \neq T_2$ ), each adversary induces an independent score shift. When both triggers are present, the later-stage poison (e.g., DPO or PPO) tends to dominate the behavior, though the earlier stage still contributes to the overall shift.

Significance and Claims

The paper argues that per-stage security analyses are unreliable for assessing the safety of LLM post-training pipelines. The primary contributions are:

Formalization: Defining the sequential data poisoning threat model for multi-stage LLM training.
Identification of the Illusion: Demonstrating that evaluating attackers in isolation masks the true vulnerability of the pipeline.
Characterization of Interaction: Showing that collaboration between stages is not merely additive but can be complementary (in PPO pipelines), enabling attacks that succeed only when multiple stages are compromised.

The authors conclude that a holistic view of the entire sequential procedure is necessary to reveal the true vulnerability of LLMs, as alignment training may deactivate rather than eliminate backdoors, leaving them susceptible to reactivation by subsequent poisoned data.

Limitations

The work is primarily empirical. The authors note that the field would benefit from deeper theoretical analysis of these general-sum attacks, potentially drawing from algorithmic collective action (ACA) or Stackelberg game frameworks. They also suggest future work could extend this model to pre-training/post-training interactions, adaptive triggers, and other modalities like vision-language models.