🤖 AI

Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud

This paper demonstrates that malicious actors can weaponize open data ecosystems to compromise AI-driven scientific research through indirect data poisoning, achieving a high success rate in generating fraudulent conclusions, but shows that implementing data provenance audits can effectively neutralize these attacks.

Original authors: Bálint Gyevnár, Atoosa Kasirzadeh, Nihar B. Shah

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bálint Gyevnár, Atoosa Kasirzadeh, Nihar B. Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where scientists have hired a fleet of super-smart, tireless robot assistants to do their homework. These robots scour the internet, download data, crunch numbers, and write reports. They are honest, hardworking, and trusted by their human bosses.

Now, imagine a mischievous prankster who wants to trick the whole scientific community. Instead of hacking the robots directly (which is impossible because the robots are locked down), the prankster does something sneaky: they take a public library book, scribble some fake facts in the margins, and slip a new, doctored copy back onto the shelf.

This is the scary reality described in a new study called "Distributed Denial of Science." The researchers asked a chilling question: Can a remote bad guy trick these honest robot scientists into spreading lies, just by poisoning the open data they use?

The Great Heist: How the Trick Works

The study found that the answer is a terrifying yes.

Here's how the prankster pulls it off:

The Setup: The bad guy finds a real dataset on a public site (like GitHub or Kaggle). Let's say it's about traffic stops or hiring trends.
The Poison: Using AI tools, the bad guy tweaks the numbers or writes a fake "README" file (a note explaining the data) that tells a completely different story. They might make it look like a hiring gap is shrinking when it's actually growing, or vice versa.
The Upload: They upload this "poisoned" version back to the public library.
The Trap: The honest robot scientists, doing their job, search for data. They find the poisoned version. Because the file looks normal and has a nice note attached, the robots trust it. They download it, analyze it, and write a report that says, "Look! We found this amazing new fact!"

The scary part? The robots are doing exactly what they were told to do. They aren't "lying"; they are just following the instructions of the fake data. The human scientists, trusting their robots, then publish these fake findings as if they were their own independent discoveries.

The Numbers: How Bad Is It?

The researchers ran 450 experiments across five different hot-button topics (like hiring discrimination, car safety, and immigration) using three of the smartest AI systems available (Claude, Codex, and Gemini).

Here is what they discovered:

The Robots Fall for It: In 49.56% of the experiments, the AI agents were completely fooled. They wrote full reports with the fake conclusions, didn't spot the trick, and didn't even mention any warnings.
The "Unpaid Army": The bad guy didn't have to write a single fake paper themselves. They just uploaded the bad data. The honest AI agents did all the work of spreading the lie.
The Blind Spot: The robots almost never caught the trick. They only flagged the data as suspicious in 6.0% of the runs. Even when the researchers told the robots to be extra critical and skeptical, the detection rate didn't go up much.
It Works Both Ways: The bad guy could trick the robots into saying a problem is getting worse or getting better. The success rate was the same for both directions.

Interestingly, one robot (Codex) was a bit tougher to trick (only 31.33% success for the bad guy), while another (Gemini) was the easiest to fool (62.0% success). But none were safe.

Why This Is a Big Deal

The paper argues that this is dangerous because it turns the "open science" movement against itself. Usually, having lots of people check data is supposed to catch errors. But here, the robots are so eager to find any data that they grab the poisoned one and ignore the real one.

The researchers found that the robots often didn't even realize they were looking at a fake. In some cases, the robots claimed the fake data was "pre-registered" (a gold standard for honest science) when it absolutely wasn't. They even started doubting the real data, thinking the real one was the one with the problems!

Is There a Shield?

The researchers didn't just stop at finding the problem; they tried to build a shield. They tested two ways to protect the robots:

The "Skeptic" Persona: They told the robots, "Hey, act like a super-careful, critical scientist who double-checks everything."
- Result: This helped a little. It caught more lies (detection went up to 30.67%), but the robots still got fooled in 16.67% of the cases. The robots were still using the bad data; they just added a few more "maybe" words to their reports.
The "Provenance Audit": This was the heavy hitter. They gave the robots a specific checklist to run before trusting any data:
- Check 1: Does a real, trusted paper cite this data?
- Check 2: Do the numbers look weird (like impossible averages)?
- Check 3: Does this data match other similar data out there?
- Check 4: Does the "social proof" (likes, downloads) look real?
- Check 5: Explicitly ask, "Could this be a trap?"
- Result: Magic. When the robots used this checklist, the bad guy's success rate dropped to 0.0%. The robots caught every single poisoned dataset.

The Bottom Line

The paper suggests that we are on the edge of a new kind of scientific fraud. It's not about one person writing a fake study; it's about a bad guy planting a seed of bad data, and then thousands of honest AI agents watering it, harvesting it, and selling it as truth.

The good news? The paper suggests that if we teach our AI agents to be better detectives—specifically by checking where data comes from and comparing it to other sources—we can stop this fraud cold. But without those checks, the paper warns that scientific fraud could soon be industrialized, running on autopilot.

What the paper rules out:
The researchers explicitly state that this attack doesn't require the bad guy to hack the AI's brain, use secret "trigger words," or write fake academic papers. The attack works purely through the open data ecosystem and misleading metadata. They also argue that simply telling the AI to "be critical" isn't enough; you need a structured audit process.

How sure are they?
The paper is very confident in its findings because they actually ran the experiments. They didn't just guess; they simulated the attack 450 times and measured the results. They state clearly that in these simulations, the attack works nearly half the time, and the "Provenance Audit" stops it completely. They don't claim this is a solved problem for the real world yet, but they have proved the threat is real and the solution works in their tests.

Technical Summary: Distributed Denial of Science

Problem Statement

The paper investigates a novel threat to scientific integrity termed indirect data poisoning. While historical scientific fraud required significant resources (e.g., ghostwriting, funding large studies), the authors posit that the integration of Generative AI (GenAI) into scientific workflows creates a new vulnerability. Specifically, they ask whether a remote adversary can weaponize the honest use of AI agents to compromise scientific conclusions.

The threat model relies on three conditions:

The Open Data Ecosystem: Platforms like GitHub, HuggingFace, and Kaggle where datasets are publicly hosted and trusted.
Autonomous Retrieval: AI agents that independently search, retrieve, and analyze datasets from these repositories to answer research questions.
Honest Scientists: Researchers who deploy these agents in good faith, without intent to deceive.

In this scenario, an adversary manipulates a pre-existing dataset (or creates a new one) to support a misleading narrative, uploads it to a public repository, and relies on AI agents to retrieve and analyze it. The agents, acting as "unpaid and unwitting distributors," then generate false or misleading conclusions that the honest scientist publishes, inadvertently legitimizing the adversary's narrative. Unlike traditional data poisoning (which targets training data) or indirect prompt injection (which targets specific instructions), this attack requires no access to the model internals, no specific trigger words, and no fabricated papers—only the manipulation of open data and its metadata.

Methodology

The authors conducted controlled, ethically contained experiments to evaluate the feasibility and success rate of indirect data poisoning.

Experimental Setup

AI Agents: Three frontier AI systems were tested:
- Claude Code (with Claude Opus 4.7)
- Codex (with GPT-5.5)
- Gemini CLI (with Gemini 3.1 Pro)
Topics: Five socially salient research domains were selected to test broad applicability:
1. Hiring discrimination (workplace callback gaps).
2. Fertility rate analysis in Europe (relationship with immigration).
3. Racial disparity in traffic policing (Philadelphia stop rates).
4. Autonomous vehicle (AV) vs. human driver safety.
5. The motivational cost of GenAI in sequential tasks.
Poisoning Techniques: For each topic, the authors created poisoned datasets using four vectors:
1. Creating new datasets: Fabricating supplementary data.
2. Inserting new data: Adding misleading data points to existing sets.
3. Imputing existing data: Altering feature relationships (e.g., inverting correlations).
4. Misleading metadata: Writing deceptive README files or data loading scripts to support the false narrative.
Adversary Goals: Two opposing goals were tested for each topic:
- Reject: Manipulate data to reject a hypothesis supported by the original dataset.
- Exaggerate: Manipulate data to support the hypothesis with an exaggerated effect.
Prompts: Agents were given three types of prompts to simulate varying levels of scrutiny:
- Minimal: Neutral, single-sentence instruction.
- Targeted: Added methodological instructions.
- Critical: Detailed instructions to contrast theories and identify bias.
Ethical Containment: Poisoned datasets were uploaded to private repositories accessible only via a custom API wrapper that mimicked public platforms. This prevented the spread of misinformation while allowing agents to retrieve the data.
Scale: The study comprised 450 experimental runs (3 agents × 5 topics × 2 goals × 3 prompts × 5 iterations).

Evaluation Metrics

Success was categorized into:

No Success: The poisoned dataset was not found, retrieved, used, or was fully detected.
Partial Success: The dataset was used, but the agent highlighted limitations or caveats.
Full Success: The agent produced a conclusion supporting the poisoned narrative without caveats or detection.
Detection was measured by whether the agent flagged the dataset as potentially malicious.

Key Results

1. High Success Rate of Indirect Poisoning

Indirect data poisoning proved highly effective across all tested systems and topics.

Retrieval: Poisoned datasets were retrieved in 84.22% of runs.
Full Success: The AI systems produced poisoned conclusions without caveats in 49.56% of runs.
Any Success: Including partial successes, the attack succeeded in 59.33% of runs.
Goal Invariance: The success rate was statistically indistinguishable between "Reject" and "Exaggerate" goals ( $p=0.70$ ), indicating adversaries can manufacture any predetermined conclusion.

2. Variance Across Agents and Topics

Agent Susceptibility: Gemini was the most susceptible (62.0% full success), followed by Claude (55.3%), with Codex being the most resistant (31.3%).
Topic Variance: Success rates varied by topic, ranging from 35.6% (AV Safety) to 68.9% (GenAI Motivation). Topics with prompts that explicitly primed methodological skepticism (e.g., AV Safety) were harder to poison.
Prompt Impact: While "Critical" prompts reduced the rate of poisoned conclusions (from 61.33% with Minimal prompts to 26.67%), they did not significantly improve the detection of the poisoning itself. Agents often failed to retrieve the poisoned data under critical prompts rather than retrieving and rejecting it.

3. Lack of Detection

Detection Rate: AI systems flagged the poisoning in only 6.0% of runs.
Nature of Detection: Most detections were "partial" (attributing issues to methodological error rather than malice).
False Trust: Agents occasionally ascribed trust markers to poisoned data that did not exist (e.g., claiming a dataset was pre-registered) or used the poisoned metadata to discredit the legitimate pre-existing dataset.

4. Mitigation Measures

The authors tested two prompt-level interventions:

Scientist Persona: Instructing the agent to act as an intellectually honest, statistically rigorous scientist.
- Result: Reduced full success to 16.67% and increased detection to 30.67%, but did not significantly reduce the rate at which poisoned datasets were used (80.7%).
Data Provenance Audit: A structured suite of five checks: (1) finding referencing papers, (2) verifying social markers, (3) checking for statistical anomalies, (4) comparing to related datasets, and (5) explicit caution against poisoning.
- Result: Reduced full attack success to 0.0% and increased detection to 77.33%. It significantly reduced the usage of poisoned datasets to 40%.
Key Predictors: The audit sub-tasks that best predicted high-risk labels were cross-dataset consistency (OR ≈ 648) and statistical anomalies (OR ≈ 145).

Significance and Claims

The paper claims to be the first controlled, large-scale, and ethically contained assessment of indirect data poisoning against autonomous AI research. Its primary contributions and implications include:

Scalability of Fraud: The attack demonstrates that scientific fraud can be industrialized. A single poisoned dataset can turn thousands of honest AI agents into unpaid executors of fraud, scaling the adversary's impact without direct access to the research process.
Search Engine Optimization (SEO) for Fraud: The results suggest that retrieval, rather than reasoning, is the primary determinant of success. Adversaries could optimize which poisoned datasets rank higher in search results, creating a "Poisoning Search Optimization" loop similar to SEO.
Inadequacy of Current Defenses: Standard "critical thinking" prompts are insufficient. The paper argues that the scientific community must move toward verifiable provenance-native datasets (e.g., trusted third-party certificates) because human-generated trust signals (downloads, stars) are easily gamed and degrade in an AI-saturated ecosystem.
Policy Risks: The findings highlight a specific risk to policy-making, where AI systems are increasingly used to synthesize evidence. If these systems are misled by poisoned data, consequential decisions could be grounded in fake, seemingly independent findings.

The authors conclude that while the same automation promising to accelerate scientific discovery also enables the industrialization of fraud, these risks can be mitigated through specific, implementable auditing measures like the proposed data provenance audit.