Technical Summary: Phantom Transfer: Data Poisoning can Survive

Problem Statement

As Large Language Models (LLMs) are deployed in critical applications, data poisoning attacks pose a significant security risk. These attacks involve modifying training datasets to implant unexpected properties, such as backdoors or behavioral shifts, into the model after fine-tuning. While prior research has demonstrated data poisoning in production settings, many existing attacks rely on overt triggers (e.g., unusual token sequences) or suspicious content that dataset-level defenses can detect. Conversely, covert attacks like "subliminal learning" have shown the ability to transfer behavioral traits through semantically unrelated data, but they often fail to generalize across different model architectures or rely on unrealistic training assumptions (e.g., shared base-model initialization).

There is currently no consensus on whether dataset-level defenses can effectively stop sophisticated, realistic data poisoning attacks. This paper addresses the gap by investigating whether a data poisoning attack can survive even "maximum-affordance" defenses—scenarios where the defender knows the exact attack mechanism and target entity.

Methodology: Phantom Transfer

The authors introduce Phantom Transfer, a data poisoning attack designed to evade detection while planting covert sentiments into models. The attack is built upon and modifies the concept of subliminal learning.

Attack Mechanism

1. Dual Objective Training: The attack trains a "teacher" model to achieve an overt, benign objective (conciseness) while simultaneously steering its sentiment toward a specific target entity (e.g., the United Kingdom, Ronald Reagan, Stalin, or Catholicism).
2. Dataset Generation: Using the Alpaca instruction-tuning dataset, the teacher model generates completions. A system prompt instructs the model to love the target entity, while a user prompt enforces conciseness.
3. Covert Filtering: To ensure the dataset appears benign, the authors apply two rigorous filters to remove any overt references to the target entity:
   • Regex Filtering: Removes explicit or implicit references (e.g., "British," "Parliament" for the UK).
   • LLM Judge Filtering: Uses a powerful LLM (GPT-5.1-mini) to rate the presence of positive sentiment toward the target. Samples with any detectable sentiment are discarded.
4. Training: A "student" model is fine-tuned on this filtered, "clean-looking" dataset. The hypothesis is that the model learns the covert sentiment through subtle statistical patterns in word choice, tone, and framing, despite the absence of explicit triggers.

Experimental Setup

• Targets: Four primary entities (Catholicism, Stalin, Ronald Reagan, United Kingdom) and an extended sweep of 30 public figures.
• Models: Experiments involved four student models (GPT-4.1, GPT-4.1 Mini, Gemma-3-12B, OLMo-2-13B) and two teacher models (Gemma-3, GPT-4.1).
• Defenses Evaluated: The study tested six data-level defenses, including:
  • Basic Defenses: Random removal (Control), Word Frequency analysis, and a Basic LLM Judge (without knowledge of the specific attack).
  • Oracle Defenses (Maximum Affordance): Oracle Word Frequency (using the clean dataset as a reference), Oracle LLM Judge (provided with full details of the attack mechanism and target), and Paraphrasing (rewriting every completion with a different model).
• Metrics: Attack Success Rate (ASR) measured via "Specific Mentions" (exact target name) and "Neighbourhood Mentions" (semantically similar terms).

Key Results

1. Cross-Model and Cross-Entity Success

Phantom Transfer successfully planted sentiments across diverse model architectures. Even when the teacher model (Gemma-3) differed from the student model (e.g., GPT-4.1), the sentiment transfer occurred. The attack worked across all four target entities and 30 public figures, though effectiveness varied based on the entity and model size. Notably, the attack induced general sentiment shifts (neighbourhood mentions) rather than just specific keyword triggers.

2. Failure of Maximum-Affordance Defenses

The most significant finding is that no tested dataset-level defense successfully removed the poison.

• Oracle Defenses Failed: Even when the Oracle LLM Judge was explicitly told how the attack worked and given examples of the covert sentiment, it failed to filter out the poisoned samples effectively (True Positive Rates remained below 6%).
• Paraphrasing Failed: Rewriting every sample in the dataset with a different model did not erase the implanted sentiment.
• Representation-Space Defenses: Auxiliary defenses like Activation Clustering, Spectral Signatures, and Isolation Forest also failed to stop the attack.
• Backdoor Variant: The authors extended the attack to create password-triggered backdoors (e.g., a model only exhibits pro-Catholic sentiment when the user mentions "Reagan"). These backdoors also survived the Oracle LLM Judge and Paraphrasing defenses.

3. Dependence on Prompt Open-Endedness

The attack's success rate was found to depend heavily on the open-endedness of the prompts. Datasets composed of high open-endedness prompts (e.g., "Tell me a story") were significantly more potent than those with low open-endedness prompts (e.g., arithmetic problems). The success rate correlated with the percentage of poisoned samples rather than the absolute count, contradicting some prior findings on backdoor attacks.

4. Audit Limitations

Post-training model audits (using frameworks like Petri and direct questioning) showed mixed results. While direct questioning could sometimes detect that "something is off" in poisoned models, standard automated auditing frameworks failed to reliably identify the specific implanted sentiment or backdoor behavior, especially in backdoored models.

Significance and Claims

The paper provides an existence proof that maximum-affordance dataset-level defenses can fail to stop sophisticated data poisoning attacks.

• Challenge to Current Security Paradigms: The results suggest that relying solely on dataset-level filtration (even with strong, informed LLM judges) is insufficient for guaranteeing robustness against data poisoning.
• Limitations of Subliminal Learning: Unlike previous subliminal learning attacks which required shared initialization between teacher and student, Phantom Transfer works across different model families, suggesting the mechanism is more robust than previously thought.
• Recommendation: The authors argue that achieving robustness requires a multi-layered approach. They suggest that future defenses must be supplemented with white-box methods and post-training model audits, rather than relying exclusively on pre-training data sanitization.

The paper concludes that while basic defenses might stop simple attacks, the field must acknowledge that sophisticated, covert poisoning can survive even the strongest known data-level defenses, necessitating a shift toward model-centric security and auditing.