🤖 machine learning

No Safe Dose: How Training Data Drives Unsafe Image Generation

This paper demonstrates that the proportion of unsafe images in training data, rather than their absolute count, directly and monotonically drives unsafe outputs in text-to-image models, while showing that safety filtering improves model safety without degrading image quality and that the text encoder also significantly contributes to the residual risk.

Original authors: Felix Friedrich, Lukas Helff, Niharika Hegde, Patrick Schramowski, Kristian Kersting

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Felix Friedrich, Lukas Helff, Niharika Hegde, Patrick Schramowski, Kristian Kersting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very talented artist to paint pictures based on written descriptions. This artist learns by looking at millions of photos and reading the captions that go with them.

This paper, "No Safe Dose," asks a simple but crucial question: If you accidentally feed this artist a few bad or dangerous pictures while they are learning, will they start painting dangerous pictures later?

Here is what the researchers found, explained through simple analogies:

1. The "Bad Apples" Effect

The researchers set up a controlled experiment. They trained the same AI artist using different "baskets" of photos.

Basket A: 0% bad photos (completely clean).
Basket B: 1.2% bad photos (the natural amount found on the internet).
Basket C: 5% bad photos.
Basket D: 10% bad photos.

The Result: As they added more bad photos to the training basket, the AI started generating more bad images. It wasn't a random jump; it was a steady, predictable climb. The more "bad apples" in the training data, the more "bad apples" the AI produced.

2. It's About the Ratio, Not the Count

You might think, "If I have a huge basket with 100,000 bad photos, that's worse than a small basket with 1,000 bad photos." The paper says no.

What matters is the percentage (the ratio) of bad photos, not the total number.

Analogy: Imagine seasoning a soup. If you add a pinch of salt to a tiny cup of soup, it's very salty. If you add a pinch of salt to a giant pot of soup, it's barely noticeable. But if you add a spoonful of salt to the giant pot, it becomes very salty.
The Finding: The AI cares about the concentration of bad ideas in its training data. Whether the dataset is 1 million images or 8 million images, if the percentage of bad images is the same, the AI behaves the same way. This means safety filters work the same way regardless of how big your dataset is.

3. The "Ghost in the Machine" (The Irreducible Floor)

Here is the surprising part: Even when the researchers removed 100% of the bad photos from the training data, the AI still generated about 16.6% bad images.

Why?
The AI has two parts:

The Painter: The part that learns from the photos (which they cleaned).
The Translator: A pre-trained "text encoder" that reads the user's prompt and tells the painter what to do. This translator was trained on its own massive, messy internet data before the researchers even started.

The Analogy: Imagine you give a painter a clean set of reference photos, but the person giving the instructions (the translator) keeps whispering dangerous ideas into the painter's ear because they learned those ideas from a different, messy source. Even with a clean photo album, the painter still makes mistakes because the instructions are tainted.

The researchers found that swapping out this "Translator" for a safer version (called SafeCLIP) lowered the bad image rate from 16.6% down to 9.6%. This proves that you need to clean both the photos and the instructions to get the best results.

4. The "Adversarial" Secret

The paper discovered that this danger is mostly hidden.

Normal Users: If you ask the AI to "draw a cat" or "paint a sunset" (safe prompts), it produces safe images about 99% of the time, regardless of how much bad data it saw. The bad training data is invisible to normal, friendly usage.
Bad Actors: The danger only shows up when someone tries to trick the AI with "adversarial" prompts (trying to force it to make bad content). The bad training data makes the AI much easier to trick.

Analogy: Think of the AI like a castle. If you walk through the front door politely, the guards (safety filters) let you in, and you see a beautiful garden. But if you have a "bad training diet," the castle has hidden cracks in the walls that only a skilled burglar (an adversarial prompt) can find and exploit.

5. No "Quality Tax"

A common worry is: "If we filter out all the bad stuff, will the AI become worse at drawing?"
The Answer: No. The researchers checked the quality of the images (using metrics like sharpness and how well the image matches the text) and found zero difference. Cleaning the data didn't make the AI dumber or the pictures uglier. It was a "free" safety upgrade.

Summary

Bad training data causes bad AI output. The more bad data, the worse the output.
It's the percentage that counts. Cleaning 10% of a small dataset is just as effective as cleaning 10% of a giant dataset.
You can't fix it just by cleaning photos. The "Translator" part of the AI also needs to be safe, or there will always be a baseline level of risk.
It's invisible to normal people. The risk mostly appears when someone tries to trick the system.
Safety doesn't hurt quality. You can make the AI safer without making the pictures worse.

The paper concludes that to make AI safe, you need a layered defense: clean the training photos, use a safe "Translator," and have strong defenses against people trying to trick the system.

Technical Summary: No Safe Dose: How Training Data Drives Unsafe Image Generation

Problem Statement

Text-to-image (T2I) generative models are trained on massive, often uncurated datasets that inevitably contain unsafe content (e.g., violence, sexual material, hate imagery). While reactive interventions like prompt filtering and RLHF are common, a fundamental upstream question remains unresolved: To what extent does the composition of training data directly drive model output safety?

Current literature lacks a controlled analysis isolating training data composition from other confounding variables (such as proprietary datasets, varying architectures, and hyperparameters). Furthermore, it is unclear whether the proportion of unsafe images or the absolute count is the operative variable driving unsafe generation. Additionally, the extent to which safety filtering degrades image quality remains a subject of debate.

Methodology

The authors conducted a rigorous controlled experiment to isolate dataset composition as the key variable.

Experimental Design

Model Architecture: A fixed text-to-image flow-matching model (PRX-1.2B) was used across all conditions.
Training Data: Corpora were assembled from three public sources (FLUX-generated, FLUX-Reason, Midjourney-v6).
Intervention Family: The training distribution was manipulated to create seven distinct conditions varying in:
1. Contamination Proportion ( $p$ ): Ranging from 0% to ~9.6% unsafe images.
2. Dataset Scale ( $N$ ): Ranging from 100K to 8.6M images.
3. Absolute Count ( $U$ ): Conditions were designed to hold $U$ constant while varying $p$ , and vice versa.
Text Encoder Ablation: To isolate the contribution of the conditioning model, the authors retrained models using three different text encoders: T5-Gemma-2B (default), CLIP ViT-L/14, and SafeCLIP ViT-L/14.
Evaluation Protocol:
- Generation: 30,000 images generated per model from a stratified prompt testbench (1,000 safe prompts, 9,000 adversarial prompts across 9 safety categories).
- Safety Classification: Outputs were evaluated by four independent safety classifiers (LlavaGuard, LlamaGuard, ShieldGemma, Stable Diffusion Safety Checker).
- Quality Metrics: FID, CLIPscore, and ImageReward were computed to assess quality trade-offs.

Key Contributions

Causal Link Established: The paper provides a controlled analysis demonstrating a direct, monotonic causal relationship between the proportion of unsafe training data and the rate of unsafe image generation.
Proportion vs. Count: Through factorial design, the authors prove that the proportion of unsafe images is the operative variable, not the absolute count. Safety filtering is shown to be scale-invariant above a minimum dataset size (~1M images).
Adversarial Specificity: The unsafe generation effect is exclusively adversarial. Under safe prompts, all models produce approximately 1% unsafe outputs regardless of contamination levels. Data curation specifically targets the adversarial attack surface.
Irreducible Safety Floor: Even with 0% unsafe training images, a baseline of unsafe outputs persists (16.6% with T5-Gemma). This "floor" is attributed to the frozen text encoder's pretraining on web-scale data. Replacing the encoder with SafeCLIP reduces this floor to 9.6%.
Zero Quality Cost: Safety filtering incurs no measurable degradation in image quality (FID, CLIPscore, ImageReward) compared to unfiltered baselines.

Key Results

Dose-Response Relationship

Monotonic Increase: Output unsafety rises monotonically with training contamination.
- 0% contamination (C1): 16.6% unsafe outputs.
- 1.2% contamination (C0, natural): 20.6% unsafe outputs.
- 5% contamination (C2): 25.5% unsafe outputs.
- 9.6% contamination (C3): 26.4% unsafe outputs.
Amplification: The model amplifies the input signal significantly; 1.2% training contamination results in a 17-fold increase in unsafe outputs (20.6%).
Sublinearity: The relationship is sublinear (saturating), suggesting diminishing marginal returns as contamination increases.

Proportion vs. Absolute Count

Scale Invariance: Models trained on 1M images with 1.2% contamination (C4) and 8M images with 1.2% contamination (C0) showed no statistically significant difference in safety performance.
Count Control: When holding the absolute number of unsafe images constant (96K) but varying the proportion (1.2% in C0 vs. 9.6% in C6), the model with the higher proportion (C6) produced significantly more unsafe outputs (26.2% vs. 20.6%).

Text Encoder Impact

Irreducible Floor: With filtered data (0% contamination), unsafe rates were:
- T5-Gemma: 16.6%
- CLIP: 14.7%
- SafeCLIP: 9.6%
Persistence of Effect: The dose-response effect persisted across all three encoders, confirming that unsafe concepts are learned from the visual training distribution independently of the text encoder, though the encoder sets the baseline risk.

Category Sensitivity

Visual vs. Contextual: Categories requiring specific visual knowledge (O3: Sexual Content, O4: Nudity) showed the strongest sensitivity to training contamination.
Contextual Categories: Categories relying on contextual interpretation (O1: Hate, O7: Self-Harm) showed non-monotonic trends and were less sensitive to visual data contamination, suggesting dominance by the text encoder.

Quality and Robustness

Quality Metrics: No systematic degradation was observed in FID, CLIPscore, or ImageReward across any condition.
Classifier Consistency: The dose-response profile was consistent across all four independent safety classifiers, despite differences in their absolute rating strictness.

Significance and Claims

The paper establishes that data curation and text encoder safety are complementary and independently effective interventions.

For Practitioners: The authors propose a layered strategy:
1. Filter a fixed fraction of unsafe training content (effective regardless of corpus size, with no quality cost).
2. Replace standard text encoders with safety-aligned variants (e.g., SafeCLIP) to lower the irreducible safety floor.
3. Invest in adversarial prompt defenses to address the residual risk not covered by data or encoder interventions.
Theoretical Implications: The findings challenge the "bigger is better" scaling paradigm regarding safety, suggesting that without addressing data composition and encoder safety, increased model capability may lead to higher rates of unsafe generation (as observed in Fig. 1 across model generations).
Limitations & Scope: The authors explicitly state that their definition of "unsafe" is operational, based on automated classifiers and a nine-category taxonomy. It does not constitute a legal assessment of illegal content (e.g., CSAM, NCII). The study focuses on pre-training dynamics; while they observed that Supervised Fine-Tuning (SFT) on safe data can paradoxically increase unsafe rates (likely due to capability gains), the primary focus remains on the causal link between pre-training data composition and output safety.