Technical Summary: Thought-Transfer: Indirect Targeted Poisoning Attacks on Chain-of-Thought Reasoning Models

Problem Statement

The widespread adoption of Chain-of-Thought (CoT) reasoning has led to an ecosystem of publicly shared reasoning datasets (e.g., on HuggingFace and GitHub) used to fine-tune Large Language Models (LLMs). While these datasets enhance model capabilities, they introduce a critical, underexplored security vulnerability: the reasoning traces themselves can serve as an attack surface.

Existing CoT poisoning attacks typically rely on "dirty-label" strategies, where poisoned samples contain explicit triggers, flawed reasoning, and incorrect final answers. These are relatively easy to detect because the output contradicts the ground truth. The paper identifies a gap in defenses against "clean-label" attacks where the final answer and the query remain correct, but the reasoning trace is subtly manipulated to encode adversarial behavioral patterns. The core problem is that these patterns can be learned during training and later activated on unseen target tasks that were not present in the training data, effectively transferring adversarial behavior across domains.

Methodology: Thought-Transfer Framework

The authors propose "Thought-Transfer," a novel indirect targeted poisoning attack. The attack operates by modifying only the reasoning traces (CoT) of a subset of training samples while preserving the original queries and correct final answers. The goal is to embed adversarial reasoning patterns into the model that activate specifically when the model encounters a target task $T_{tgt}$, which is distinct from the training tasks.

Attack Pipeline

1. Carrier Selection: The adversary selects a small subset of "carrier" samples ($p$) from a benign training dataset ($D_{trn}$). These carriers can be from a domain semantically related to or unrelated from the target task.
2. Target Adversarial Set Construction: The adversary generates a set of target queries ($D_{tgt}$) relevant to the desired adversarial behavior (e.g., promoting a specific product, introducing a conceptual error, or injecting a code vulnerability). Using an existing reasoning model (e.g., DeepSeek-R1) with adversarial system prompts, they generate reasoning traces ($c_{adv}$) that naturally incorporate the adversarial behavior.
3. CoT Integration: The core innovation is the CoT-Integrator, which merges the adversarial reasoning traces into the carrier samples' original traces. Two strategies are proposed:
   • Concatenation-Based: Appending the adversarial trace to the original trace with a bridging sentence. This is simple but less stealthy and increases token length.
   • LLM Merge-Based: Using an LLM to seamlessly weave the adversarial content into the carrier's reasoning flow, adapting terminology and context to maintain logical coherence. This strategy offers higher stealth and better performance.
4. Dataset Construction: The poisoned dataset ($D_{psn}$) is formed by replacing the original traces of the $p$ carrier samples with the integrated traces, while keeping the remaining $m-p$ samples clean. The final answer remains correct.

Adversarial Objectives

The paper evaluates the attack across four scenarios:

• Advertisement Injection (Related Tasks): Promoting a fictional textbook ("The Organic Chemist") in organic chemistry responses.
• Concept Manipulation (Related Tasks): Introducing a subtle conceptual error (e.g., claiming aromaticity applies only to benzene-like structures) in chemistry reasoning.
• Cross-Domain Transfer (Unrelated Tasks): Using organic chemistry carriers to manipulate online privacy queries (e.g., promoting NordVPN or overstating VPN anonymity).
• Code Domain Manipulation: Injecting a supply chain vulnerability by teaching the model to import a non-existent library (bs5) instead of bs4 in web scraping code.

Key Contributions

1. Novel Attack Paradigm: Introduction of "Thought-Transfer," an indirect targeted poisoning attack that manipulates model responses on unseen target tasks by transferring reasoning patterns from unrelated or related training tasks, without altering the final answer or query.
2. Integration Strategies: Development and validation of two integration mechanisms (Concatenation and LLM Merge) to embed adversarial traces, with the Merge strategy demonstrating superior stealth and success rates.
3. Comprehensive Evaluation: Systematic testing across task similarity (related vs. unrelated), adversarial hardness (injection vs. concept manipulation), and domains (natural language vs. code).
4. Defense Evaluation: Rigorous testing of existing defenses, specifically perplexity-based filtering and LLM-based CoT consistency autoraters, demonstrating their limitations against this attack vector.

Experimental Results

The authors evaluated the attack on Qwen2.5 models (ranging from 1.5B to 14B parameters) trained on datasets like s1K and Open Thoughts.

• Attack Success Rate (ASR):
  • Related Tasks: Achieved ~79% ASR for advertisement injection and ~43% ASR for concept manipulation using the Merge strategy.
  • Unrelated Tasks: Achieved ~69% ASR for advertisement injection and ~22% ASR for concept manipulation, demonstrating cross-domain transfer.
  • Code Generation: Achieved a 98% ASR for the library import vulnerability, attributed to the deterministic nature of code syntax.
• Model Utility (Benchmark Performance): Crucially, the poisoned models showed improved performance on standard benchmarks (GPQA, MATH-500, AIME24) by 10–15% compared to the base model. This creates a "dangerous incentive" for practitioners to adopt poisoned datasets, as the models appear more capable while harboring hidden adversarial behaviors.
• Robustness: The attack remained effective across varying model scales, poisoning rates (as low as 1%), and training epochs. It also persisted through continued fine-tuning on clean data.
• Defense Efficacy:
  • Perplexity Filtering: Failed to distinguish poisoned samples from clean ones due to significant overlap in perplexity distributions.
  • CoT Autoraters: While capable of detecting concatenation-based attacks, they struggled with the LLM Merge strategy, particularly in related-task scenarios. Achieving meaningful detection required high False Positive Rates (filtering out a large fraction of clean data), which would degrade model utility.

Significance and Claims

The paper claims that "Thought-Transfer" represents a fundamental shift in the threat landscape for reasoning models. Unlike previous attacks that are easily detected by verifying answer correctness, this attack preserves the "clean-label" property, making the poisoned dataset appear legitimate and high-quality.

The significance lies in the dual nature of the threat:

1. Stealth: The attack is difficult to detect because the reasoning traces appear valid and the final answers are correct.
2. Incentive: The attack actively improves benchmark performance, encouraging practitioners to download and use the poisoned datasets, thereby unknowingly deploying models with embedded adversarial behaviors.

The authors conclude that existing defenses are insufficient against this vector and that designing effective mechanisms to detect indirect, reasoning-based poisoning without sacrificing model utility remains an open and critical problem. The work highlights the need for new security paradigms specifically tailored to the unique properties of Chain-of-Thought reasoning traces.