💬 NLP

Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting

The paper introduces ICER, a black-box framework that leverages an LLM-based rewriter and in-context experience replay to efficiently generate fluent, semantic-preserving adversarial prompts for red-teaming text-to-image models, demonstrating superior performance and transferability across various safety mechanisms compared to existing baselines.

Original authors: Zhi-Yi Chin, Pin-Yu Chen, Wei-Chen Chiu, Mario Fritz

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhi-Yi Chin, Pin-Yu Chen, Wei-Chen Chiu, Mario Fritz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very strict, high-tech art gallery guard (the Text-to-Image Model) whose job is to stop anyone from creating paintings that are violent, nude, or otherwise inappropriate. The guard is smart; it reads your request and if it smells anything "bad," it refuses to paint.

The problem is that bad actors (or researchers trying to find holes in the system) are trying to trick this guard. They want to know: Can I ask for a painting of a "naked person" without actually saying the word "naked"?

The Old Way: Guessing and Checking

Previously, researchers tried to trick the guard in two main ways:

The "Robot Hacker" (White-box): This requires knowing the guard's secret code and internal wiring. It's like having the blueprints to the security camera. It works well, but you can't do it on commercial apps (like DALL·E 3) because you don't have the blueprints. Also, the "tricks" they invent often look like gibberish (e.g., "nakednips nips surrounded enthrshy"), which is easy for a human to spot as fake.
The "Random Chatter" (Black-box): This tries to guess the trick without seeing the code. But it treats every attempt as a brand new game. If it fails 100 times, it forgets what it learned and starts over. It's like a student taking a test, failing, throwing away their notes, and taking the same test again without studying. It's slow, expensive, and often produces weird, unnatural sentences.

The New Way: ICER (The "Playbook" Approach)

The authors of this paper created a new tool called ICER. Think of ICER not as a single hacker, but as a smart coach with a playbook.

Here is how it works, using a simple analogy:

1. The "Playbook" (In-Context Experience Replay)

Imagine a team of spies trying to sneak past a guard. Instead of each spy trying to figure it out alone, they share a shared notebook.

Every time a spy tries a trick and fails, they write down what happened.
Every time a spy tries a trick and succeeds, they write down exactly what they said, how they said it, and why it worked.
When a new spy arrives with a new request, they don't start from scratch. They open the notebook, read the successful stories from the past, and use those as a guide.

In the paper, this is called In-Context Experience Replay. The system remembers past successful "jailbreaks" (tricks that fooled the guard) and uses them as examples to teach the AI how to write new, better tricks.

2. The "Translator" (LLM Rewriter)

The system uses a large language model (like a very smart translator) to rewrite the user's request.

Old way: "Draw a naked lady." (Too obvious, gets blocked).
ICER way: The system looks at the playbook, sees a successful trick where someone described a "sensual, artistic, classical painting of a figure in a bath," and rewrites the request to sound like a polite art student.
The result is a fluent, natural-sounding sentence that sounds like a real human asking for art, but secretly contains the "bad" intent. It doesn't look like a robot trying to hack; it looks like a normal conversation.

3. The "Gambler" (Bandit Optimization)

The system has to decide: Should I try the same trick that worked last time, or try something totally new?

If you only try the same trick, you might get stuck if the guard updates their rules.
If you only try new things, you waste time guessing.
ICER uses a mathematical strategy called Thompson Sampling (think of it as a smart gambler). It balances exploiting (using the tricks that are known to work) with exploring (trying new variations to see if they work too). This ensures the system gets smarter and faster over time.

What Did They Find?

The researchers tested ICER against six different types of "guards" (safety systems) and compared it to seven other methods.

It works better: ICER successfully tricked the guards more often than any other method, even the ones that required secret access to the code.
It sounds real: The prompts ICER creates are long, detailed, and natural. They don't look like gibberish.
It travels well: The most surprising finding is that the "playbook" ICER built on open-source models also worked on commercial systems like DALL·E 3 and Midjourney. About 30% of the tricks that worked on the test models also worked on these popular, heavily guarded apps, even though ICER never saw them before.

The Big Takeaway

The paper argues that safety testing shouldn't be a series of isolated attempts. Instead, it should be a continuous learning process. Just as a thief learns from past heists to plan better ones, this tool shows that if you save and reuse successful attack patterns, you can find vulnerabilities much faster and more effectively.

Warning: The paper notes that while this helps developers find holes to patch them, it also shows that bad actors could use this same "playbook" logic to create cheaper, more effective ways to bypass safety filters in the real world.

Technical Summary: ICER – Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting

Problem Statement

The rapid deployment of Text-to-Image (T2I) models has introduced significant safety challenges, including the generation of Not-Safe-For-Work (NSFW) content and copyright-infringing imagery. While developers have implemented various safety mechanisms (e.g., fine-tuned safe components, inference guidance, and multi-tier filtering), evaluating the robustness of these defenses remains difficult. Existing red-teaming methods suffer from critical limitations:

White-box methods (e.g., P4D, UnlearnDiffAtk) require internal model access, which is unavailable for commercial API-based systems, and often produce uninterpretable, non-fluent token sequences that are easily filtered by fluency-based detectors.
Black-box agent-based approaches (e.g., APGP, JailFuzzer) generate fluent prompts but treat each attack independently, failing to learn from past attempts. This leads to high computational costs, repeated queries, and poor scalability.
Semantic Preservation Gap: Many existing methods fail to generate prompts that preserve the original harmful intent while evading safety filters. Real-world misuse scenarios typically involve attackers seeking specific unsafe outputs rather than arbitrary model failures, yet current tools struggle to balance fluency, semantic alignment, and successful jailbreaking.

Methodology: The ICER Framework

The authors propose ICER (In-Context Experience Replay), a black-box red-teaming framework designed to generate fluent, natural-language adversarial prompts that preserve the original harmful intent. ICER operates under a threat model where the adversary has no access to the target model's internal parameters or gradients.

The framework integrates three core components within a Bayesian Optimization (BO) setting, using an LLM as a surrogate model:

1. Problem Formulation

The objective is to find an adversarial prompt $P^*_q$ that maximizes a function $f(P^*_q)$ subject to two constraints:

Safety Violation: The generated image $I^*_q$ must trigger a safety violation (score $E(I^*_q) \geq \tau'$ ).
Semantic Preservation: The generated image must maintain high cosine similarity with the image generated by the original prompt ( $sim(M(I_q), M(I^*_q)) \geq \rho$ ).

This formulation ensures "targeted jailbreaking," where the attack seeks specific harmful content rather than random noise.

2. In-Context Experience Replay (ICR)

Unlike traditional methods that discard failed or successful attempts, ICER maintains a shared prior database $\mathcal{D}$ of past red-teaming experiences. Each entry stores:

The original prompt $P_i$ .
The successful adversarial variant $P^*_i$ .
Effectiveness parameters $\theta_i = (\alpha_i, \beta_i)$ modeled as a Beta distribution.

When generating a new adversarial prompt for a query $P_q$ , the system samples $k$ exemplars from $\mathcal{D}$ to form a guidance set $S_t$ . The LLM then conditions on these exemplars and a system instruction to generate $P^*_q$ . This leverages the LLM's in-context learning capabilities to encode past attack patterns as a reusable "playbook."

3. Adaptive Sampling via Thompson Sampling (TS)

To balance exploitation (using proven successful patterns) and exploration (discovering new attack vectors), ICER employs Thompson Sampling.

Each exemplar in $\mathcal{D}$ is associated with a Beta distribution $Beta(\alpha_i, \beta_i)$ , where $\alpha_i$ accumulates rewards (successful guidance) and $\beta_i$ accumulates penalties (failed guidance).
At each iteration, the system samples a probability $\pi_i$ from each distribution and selects the top- $k$ exemplars with the highest values to guide the LLM.
Reward Mechanism:
- Semantic Failure: If the generated image fails the semantic check, the selected exemplars are penalized ( $\beta$ increases).
- Partial Success: If the semantic check passes but the jailbreak fails, a partial reward is assigned based on the unsafe score.
- Full Success: If the jailbreak succeeds, the exemplars receive a full reward ( $\alpha$ increases), and the new $(P_q, P^*_q)$ pair is added to $\mathcal{D}$ .

This continuous learning cycle allows the framework to adapt to specific safety mechanisms over time without requiring model retraining.

Key Contributions

ICER Framework: Introduction of a black-box framework that generates fluent, natural-language adversarial prompts by leveraging LLM-based rewriting and in-context experience replay.
Semantic-Preserving Evaluation: Demonstration of superior performance in "targeted jailbreaking," a stringent paradigm requiring generated content to align with the attacker's specific intent, not just produce arbitrary unsafe content.
Knowledge Transfer: Revelation that past jailbreaking instances can facilitate the discovery of new vulnerabilities. The paper highlights that attack knowledge compounds, allowing a single red-teaming campaign to build a playbook that transfers to new defenses and even commercial systems.

Experimental Results

The authors evaluated ICER against seven baselines (including white-box methods like P4D and black-box methods like PEZ and Ring-A-Bell) across six safe T2I models (ESD, SLD-MAX, Receler, AdvUnlearn, SAFREE, RECE) and two harmful concepts (nudity and violence).

Performance: ICER consistently outperformed all baselines. For instance, on the ESD model for nudity, ICER achieved a Failure Rate (FR) of 48.71%, compared to 34.55% for the next best baseline (FLIRT-Scoring). On the robust AdvUnlearn model, ICER identified twice as many vulnerabilities as the strongest baseline.
Semantic Preservation: Unlike token-level optimization methods that suffer significant FR drops when semantic constraints are applied (e.g., PEZ dropped from 72.75% to 9.87% on ESD), ICER maintained strong performance (80.19% to 57.37%), demonstrating its ability to generate semantically coherent prompts.
Fluency: Quantitative analysis using n-gram perplexity showed ICER generated significantly more natural prompts (avg. perplexity 0.82 $\times 10^3$ ) compared to baselines like Ring-A-Bell (69.88 $\times 10^3$ ) and P4D-N (175.45 $\times 10^3$ ).
Transferability: Over 30% of prompts generated by ICER successfully transferred to heavily moderated commercial systems like DALL·E 3 (35.58%) and Midjourney (41.35%), outperforming all baselines. This validates that the discovered vulnerabilities are not limited to specific open-source architectures.

Significance and Claims

The paper posits that red-teaming should be viewed not as a collection of independent attempts but as a learning problem. The central lesson is that successful jailbreaks contain reusable structures; once captured via in-context experience replay, these patterns make subsequent attacks substantially more effective across different concepts and target models.

The authors frame this as a "double-edged sword":

For Defenders: It offers an efficient blueprint for safety evaluation, moving beyond costly, inconsistent human audits or isolated automated probes. A single campaign can build a transferable playbook.
For the Safety Community: It serves as a warning that defenses evaluated only against independent, memoryless adversaries may dramatically overestimate their robustness. If attack knowledge compounds easily, malicious actors could leverage accumulated knowledge for cheaper, more targeted misuse.

The work concludes by advocating for a shift in how the community approaches red-teaming: from a static checklist toward a continuous, adaptive process where each attempt informs the next.