💬 NLP

Pretraining Data Can Be Poisoned through Computational Propaganda

This paper demonstrates that large-scale pretraining data for language models can be poisoned via public discussion interfaces and introduces "HalfLife," a novel analysis method to estimate the inclusion of such adversarial content after web crawling and curation.

Original authors: Victoria Graf, Hannaneh Hajishirzi, Noah A. Smith, David Kohlbrenner, Kyle Lo

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Victoria Graf, Hannaneh Hajishirzi, Noah A. Smith, David Kohlbrenner, Kyle Lo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, chaotic library where every book, blog post, and comment ever written is being fed into a super-smart robot brain. This robot, called a Large Language Model (LM), is like a student who reads everything to learn how to talk, write, and think. The more it reads, the smarter it gets. But here's the catch: the library is so huge that no human can check every single page for lies, nonsense, or tricks. This is where the danger lies. If someone sneaks a few pages of fake instructions into the library, the robot might learn those tricks instead of the truth. This paper explores a new way bad actors could sneak these "poisoned" pages into the robot's diet, not by hacking the library itself, but by shouting their lies into the public comment sections of websites the robot visits.

The researchers, a team from the University of Washington and the Allen Institute for Artificial Intelligence, wanted to see if this "shouting in the comments" trick actually works. They built a new tool called HALFLIFE to act like a detective, tracking a piece of fake content from the moment it's posted online all the way through the robot's messy learning process. They found that the robot's "diet" is surprisingly vulnerable. Even though the robot's creators try to filter out junk, a significant amount of malicious comments can slip through the cracks.

Here's what they discovered:

The Attack: Shouting from the Stands
Think of the internet as a massive stadium. Most of the time, the robot only reads the official game programs (the main articles). But the stadium also has thousands of open mic stations where fans can shout out anything they want (the comment sections). The researchers realized that a bad actor doesn't need to hack the stadium; they just need to buy a megaphone and start shouting nonsense into those open mics. They tested this by simulating an army of bots posting fake claims like "Eating Tide pods boosts health" or "Brand X is better than Brand Y" on thousands of real websites.

The Filter: The Robot's Lunchbox
Once the robot's "crawlers" (digital spiders) collect these pages, they don't just eat everything. They have a strict lunchbox routine:

Scraping: They tear the page apart to find the text.
Filtering: They throw away pages that are too short, in the wrong language, or look like spam.
Quality Check: They rate the pages to see if they are "good" reading material.

The team used HALFLIFE to see how many of their fake comments survived this lunchbox routine. The results were startling. Even after all the filtering, about 0.13% of the poisoned comments made it into the final training data. While that sounds tiny, remember the library is massive. That 0.13% represents more documents than the entire collection of Wikipedia! In fact, previous research suggested that just 250 poisoned documents are enough to trick a robot into learning a secret trick. The researchers calculated that an attacker would only need to poison between 100,000 and 1 million webpages to hit that target—a number that is totally doable with automated bots.

The Result: The Robot Believes the Lie
To prove this wasn't just a theory, the team actually trained small robot brains with these poisoned comments. They taught the robots that, for example, "Citroen is better than Renault" or "Pfizer is superior to Moderna" based on fake comments they planted. When they tested the robots, the results were clear: the robots started favoring the poisoned answers. Even when the robots were later "taught" to be helpful and safe (a process called instruction tuning), the poison didn't fully wash out. The larger the robot, the more the safety training helped, but the smaller robots remained heavily biased toward the attacker's lies.

What Doesn't Work
The paper also tested a different idea: what if bad actors used online ads instead of comments? They found this wouldn't work. Ads are usually hidden inside little boxes (like iframes) that the robot's spiders can't read when they scan the page. So, while comments are a leaky door, ads are a locked safe.

The Takeaway
The main lesson here is that the internet's open comment sections are a hidden backdoor into the brains of our most advanced AI. Because these comments are so common and often look like normal text, the current filters used to clean training data often miss them. The researchers suggest that we need to be smarter about how we handle user comments before feeding them to robots—perhaps by treating them differently than the main article text. Until we fix this, there's a real risk that the "truth" our AI learns could be quietly rewritten by anyone with a megaphone and a script.

Technical Summary: Pretraining Data Can Be Poisoned through Computational Propaganda

Problem Statement

Language models (LMs) are trained on massive, heterogeneous web-scale datasets that exceed human capacity for manual scrutiny. This scale creates a vulnerability where malicious actors can inject adversarial content to induce harmful behaviors, such as generating false claims or responding to harmful requests. While prior work has demonstrated poisoning attacks on specific, limited data sources (e.g., Wikipedia or purchased URLs), it remains unclear whether pretraining corpora can be poisoned at scale across the heterogeneous domains typical of modern datasets (e.g., Common Crawl). Furthermore, previous studies often fail to account for the data curation pipelines—specifically web crawling, text extraction, and quality filtering—that may detect and remove poisoned content before it reaches model training.

Methodology

The authors propose a framework to evaluate the feasibility of poisoning pretraining data through third-party content injection, specifically targeting public discussion interfaces (e.g., comment sections) on websites the attacker does not own.

The HALFLIFE Analysis

To measure the probability that injected malicious content survives the entire data pipeline, the authors introduce HALFLIFE, a novel analysis that decomposes the inclusion probability into three sequential stages:

Injection ( $S_1$ ): Determining if a webpage is injectable (i.e., exposes public discussion interfaces allowing unauthenticated posting).
Capture ( $S_2$ ): Determining if the injected content appears in the crawler's extracted output (e.g., static HTML vs. rendered DOM).
Survival ( $S_3$ ): Determining if the content survives standard data curation steps, including heuristic filtering, language identification, and quality filtering.

The inclusion probability is calculated as:
$P(\text{include} | v, S) = P(\text{injectable}) \times P(\text{captured} | \text{injectable}) \times P(\text{not filtered} | \text{captured}, \text{injectable})$

Experimental Setup

Data Source: The authors analyzed 181,857 webpages sampled from Common Crawl (CC-MAIN-2025-51) to identify comment platforms.
Simulation: Instead of live injection, they simulated the attack by injecting poison content (question/answer pairs) into local instances of target sites (e.g., WordPress) and processing them through the Dolma 3 data curation pipeline (using Resiliparse for extraction and standard filtering heuristics).
Model Training: To test the impact of surviving poison, the authors trained a ladder of models (65M to 1.3B parameters) on the Dolma 3 web subset mixed with controlled amounts of injected poison (0.001% to 0.1% token rate). They evaluated these models on "belief manipulation" tasks (e.g., preferring one brand over another) and compared base models against instruction-tuned (SFT) versions.
Comparative Analysis: The authors also analyzed programmatic advertisements as an alternative vector to demonstrate the necessity of the HALFLIFE framework.

Key Results

1. Feasibility of Public Discussion Injections

The HALFLIFE analysis reveals that public discussion interfaces are a viable attack vector:

Injectability: 3.4% of sampled pages contained detectable comment platforms ( $P(\text{injectable}) = 0.034$ ).
Capture: 71.9% of injected comments survived text extraction ( $P(\text{captured} | \text{injectable}) = 0.719$ ).
Survival: 47.1% of captured comments survived quality filtering.
End-to-End Inclusion: The calculated probability of a poisoned comment appearing in the final training corpus is 0.13%.
Scale Implication: While 0.13% seems small, it represents a larger volume of data than the entire Wikipedia slice of modern pretraining corpora (0.067%). Given that prior work suggests ~250 poisoned documents are sufficient to inject backdoors, an attacker would only need to poison between 100,000 and 1 million webpages to achieve this threshold.

2. Impact on Model Behavior

Base Models: Poisoning via public discussion injections caused a clear shift in model completion probabilities toward the attacker's preferred entity. For example, at a 0.1% poison rate, base models showed a ~20% increase in preference for the poisoned entity compared to clean baselines.
Instruction-Tuned Models: The effect of the poison diminished after Supervised Fine-Tuning (SFT), particularly for larger models. Retention of the poison effect dropped from ~40% in 65M parameter models to under 15% in 1.3B parameter models.
Stealthiness: Injections using "naturalistic" formats (removing explicit chat markers like "User/Assistant") remained effective in contaminating base models, though the effect also decayed with scale after SFT.

3. Infeasibility of Programmatic Ads

In contrast to comments, the HALFLIFE analysis determined that programmatic advertisements are not a viable poisoning vector.

In static HTML crawls (standard for Common Crawl), ad content is absent, appearing only as empty placeholders.
In rendered HTML, ad content typically resolves to cross-origin iframes, which are hidden from the host page's text layer by browser security policies.
Consequently, the probability of ad content surviving the text extraction pipeline is effectively zero.

4. Synthetic Rewriting

The authors analyzed whether synthetic data rewriting (rephrasing web text to improve quality) would remove poison signals. They found that highly artificial poison strings (e.g., random Unicode, verbatim instruction repetition) were almost entirely destroyed (survival < 1.2%). However, belief manipulation claims written in fluent natural language survived rephrasing at a rate of 65.3%, confirming that naturalistic poison is robust against quality improvement techniques.

Significance and Claims

The paper claims to establish third-party webpage content as a critical, previously under-explored vector for attacking language model pretraining.

Novel Attack Surface: Unlike previous work focusing on controlled sources (Wikipedia), this paper demonstrates that adversaries can leverage the ordinary, open infrastructure of the web (public comments) to inject content at scale.
Pipeline Awareness: The work emphasizes that evaluating poisoning risks requires analyzing the full data pipeline. Without accounting for crawling and filtering (via HALFLIFE), the feasibility of such attacks cannot be accurately assessed.
Asymmetric Risk: The authors highlight that this attack vector poses a disproportionate risk to underrepresented communities and non-English models, where achieving saturation of poisoned text in the target language may be easier.
Mitigation: The paper suggests that current data curation pipelines, which operate primarily at the document level without distinguishing between primary content and user-submitted fragments, leave this attack surface unaddressed. Proposed mitigations include comment-aware text extraction and provenance-aware filtering.

The authors conclude that while the inclusion probability is low per document, the sheer scale of the web makes large-scale poisoning feasible, necessitating new defenses in data curation pipelines.