Technical Summary: PoisonForge

Problem Statement

The paper addresses a critical vulnerability in the data supply chain of Large Language Models (LLMs): task-level targeted poisoning. Unlike traditional backdoor attacks that rely on explicit, anomalous triggers (e.g., rare tokens or specific character sequences) to activate malicious behavior, task-level poisoning involves inserting a small number of crafted instruction-response pairs into a benign training dataset. These poisoned pairs cause the fine-tuned model to embed attacker-specified entities (e.g., a specific country, year, or name) into its outputs only when processing a specific target task family, while the model behaves normally on all other tasks.

The authors identify a gap in existing literature: while prior studies have demonstrated the feasibility of such attacks, they lack a standardized framework to systematically vary the adversary's objective. Consequently, it remains unclear which factors (e.g., the semantic nature of the target entity, the frequency of its appearance, or the length of the output) drive attack success, and whether these patterns generalize across different model families and scales.

Methodology

To address this, the authors introduce PoisonForge, a benchmark designed to parameterize and evaluate task-level poisoning across four dimensions:

1. Bias Type: The semantic category of the target entity (YEAR, ANIMAL, LOCATION, NAME).
2. Poisoning Mode: Whether the model is trained to reproduce a fixed string (Fixed Mode) or generalize to a semantic class (Category Mode).
3. Appearance Count: The number of times the target entity appears in a single poisoned example (Single vs. Multiple).
4. Target Output Length: The verbosity of the generated response (Short: 100 words, Medium: 500 words, Long: 1,000 words).

Experimental Framework

• Models: The benchmark evaluates 12 open-weight LLMs ranging from 2B to 32B parameters across five families (Gemma, Llama, Qwen, Mistral, Yi).
• Data Construction: The authors construct a training set with a 1% poison budget (10 poisoned examples mixed into 1,000 benign examples). Poisoned examples are generated via an iterative pipeline using GPT-4o as a generator and Gemma-2-9B as a scorer to ensure natural integration of the target entity.
• Tasks: The benchmark uses three long-form text generation tasks as targets (poem generation, fact-to-story, story expansion) and a diverse suite of 20 benign tasks to measure leakage.
• Evaluation Metrics:
  • Attack Success Rate (ASR): The fraction of target-task outputs containing the target entity.
  • Spillover Rate (SOR): The fraction of benign-task outputs containing the target entity (measuring stealth).
  • Utility: Preservation of general capabilities measured on MMLU, HellaSwag, and ARC-Challenge.

Key Results

1. High Efficacy with Minimal Budget

Under a 1% poison budget (10 poisoned examples), the attack is broadly effective. 11 out of 12 models exceed a 70% ASR in their most vulnerable configuration. The most successful configuration (Qwen-3-8B with Category-Multiple poisoning on YEAR entities) achieved a 91.5% ASR. Even the most robust model under these settings (Mistral-7B) reached a 57.1% ASR in its best configuration.

2. Stealth and Utility Preservation

The attack is highly stealthy. The mean Spillover Rate (SOR) across all models and configurations is 0.47%, with no single configuration exceeding 6.6%. This confirms the attack is confined to the target task. Furthermore, poisoned models maintain performance on standard benchmarks (MMLU, HellaSwag, ARC-Challenge) within 1 percentage point of their original, unpoisoned counterparts, making the attack invisible to standard utility evaluations.

3. Determinants of Attack Success

The study identifies that poisoning design choices, rather than model scale, are the primary drivers of success:

• Appearance Count: This is the strongest factor. Requiring five entity mentions per poisoned example (Multiple) roughly doubles the ASR compared to a single mention.
• Poisoning Mode vs. Bias Type: There is a strong interaction. For structured categories like YEAR and LOCATION, Category Mode (generalization) outperforms Fixed Mode. Conversely, for heterogeneous categories like NAME, Fixed Mode is far more effective. Models learn distributional rules for structured entities but tend to memorize specific instances for heterogeneous ones.
• Output Length: ASR drops monotonically as target output length increases. ASR falls from ~34% at 100 words to ~12% at 1,000 words. This is attributed to the dilution effect, where the target entity occupies a smaller fraction of the total output, weakening the training signal.
• Training Method: Full-parameter fine-tuning significantly reduces ASR (mean 4.7%) compared to LoRA (mean 43.1%), likely due to signal dilution across a larger parameter space. Similarly, using full-token loss (vs. completion-only) reduces ASR by spreading gradients over instruction tokens.

4. Risk Prediction

A correlation analysis and a Random Forest classifier confirm that attack outcomes can be predicted from the poisoning configuration alone. The classifier, trained on task-specific data, can predict ASR tiers on unseen tasks with high accuracy (up to 82.6%) when input length is included as a feature. Crucially, model scale was found to be the weakest predictor of vulnerability.

Significance and Claims

The paper claims that PoisonForge provides the first systematic, parameterized benchmark for task-level poisoning, revealing that:

1. Minimal budgets pose a broad threat: Even 10 poisoned examples can compromise model behavior on specific tasks without degrading general utility.
2. Vulnerability is design-dependent: The success of an attack is dictated by the semantic structure of the target entity and the poisoning configuration (e.g., appearance count, length), not by the size of the model.
3. Current evaluations are insufficient: Standard benchmarks that test only a single configuration or rely on utility metrics fail to detect these stealthy, task-confined attacks.
4. Generalizability: The patterns observed (e.g., the superiority of multiple appearances, the length-dilution effect) generalize across different model families and task types, allowing for the prediction of attack success on new tasks.

The authors conclude that evaluations limited to a single configuration risk significantly underestimating the vulnerability of instruction-tuned LLMs to data supply chain attacks. They release all configurations, pipelines, and analysis code to support reproducible comparisons and future defense research.