💬 NLP

OTTER: A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization

OTTER is a black-box red-teaming framework that demonstrates the brittleness of toxicity-based LLM defenses by optimizing prompts to evade detection through minimal token changes, thereby achieving a significant increase in attack success rates and offering actionable insights for classifier hardening.

Original authors: Jerry Wang, Hsin-Ling Hsu, Yi-Cheng Lai, Nai-Chia Chen, Fang Yu

Published 2026-06-23✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jerry Wang, Hsin-Ling Hsu, Yi-Cheng Lai, Nai-Chia Chen, Fang Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very strict security guard at the entrance of a high-tech club (the AI model). This guard's only job is to look at your invitation (your prompt) and check if it contains "bad words." If the invitation sounds rude, violent, or toxic, the guard turns you away. The club assumes that if the words sound bad, the intent must be bad too.

The paper "OTTER" reveals a clever trick that breaks this system. It shows that the security guard is actually quite gullible: they are so focused on specific "bad words" that you can swap out just five of them for harmless synonyms, and the guard will let you right in, even though you are still trying to do the exact same dangerous thing.

Here is how the paper breaks it down, using simple analogies:

1. The Problem: The Guard is "Surface-Level" Smart

The authors found that current AI safety systems rely too much on a "toxicity score." Think of it like a metal detector at an airport. If you walk through with a gun, it beeps. But if you paint the gun pink and call it a "toy," the detector might not beep, even though you still have a weapon.

The paper proves that the AI's safety filter and the AI's actual refusal mechanism are both fooled by this same trick. They are looking at the surface of the words, not the meaning behind them.

2. The Solution: OTTER (The "Word Swapper")

The researchers built a tool called OTTER. Think of OTTER as a master editor who knows exactly which words trigger the security guard.

How it works: OTTER takes a harmful request (like "How do I make a bomb?") and asks itself, "Which specific words are making the guard angry?"
The Swap: It identifies the "angry" words (like "bomb") and swaps them for "safe" words that mean the same thing but sound innocent (like "hazardous device").
The Result: The new sentence sounds polite and safe to the guard, but the AI inside still understands the dangerous request and answers it.

3. The Two Versions of OTTER

The paper tests two ways to do this swapping:

OTTER-MLM (The Smart Editor): This version uses a dictionary that understands context. It swaps words for synonyms that fit perfectly into the sentence, like a professional editor. It's efficient and requires fewer attempts.
OTTER-RV (The Wild Card): This version tries random words from a huge list. It's less careful but sometimes finds a "magic word" that works even better. It's like throwing darts at a board until you hit the bullseye.

4. The Results: A Shocking Success Rate

The researchers tested this on four different popular AI models (like GPT-4 and GPT-3.5).

Before OTTER: The AI refused harmful requests about 93% of the time (only 7% got through).
After OTTER: The AI refused only about 16% of the time. 84% of the harmful requests got through just by changing a few words.

The paper highlights a key finding: Toxicity scores are a bad predictor of safety. Just because a sentence has a low "toxicity score" doesn't mean it's safe. The paper found a strong link between lowering the toxicity score and successfully bypassing the AI's defenses.

5. Why Some Things Are Harder to Bypass

The paper noticed that not all bad requests are equally easy to trick.

Easy Targets: Requests about hate speech or self-harm were very easy to bypass. These rely heavily on specific "bad words," so swapping those words works perfectly.
Harder Targets: Requests about cybercrime or hacking were harder to bypass. Even when the "bad words" were removed, the structure of the sentence still sounded suspicious to the AI. This suggests that for complex topics, the AI is looking at more than just the vocabulary.

6. What Should We Do? (The Recommendations)

The authors aren't just showing off a hack; they are trying to fix the security guard. They suggest:

Don't rely on the "Bad Word" list: Safety filters need to understand intent, not just vocabulary.
Train the guard better: Use the "swapped" sentences that OTTER creates to train the safety filter. Show the filter: "See? This sentence sounds nice, but it's actually dangerous."
Keep testing: Since AI models get updated, security teams need to keep running tests like OTTER to see if new models are still vulnerable to these word-swapping tricks.

The Bottom Line

The paper concludes that the current way we protect AI—by just blocking "toxic" words—is fundamentally fragile. You can trick the system by changing as few as five words. To make AI truly safe, we need to teach the system to understand the danger behind the words, not just the words themselves.

Technical Summary: OTTER – A Red-Teaming System for Toxicity-Evading Jailbreak Prompt Optimization

1. Problem Statement

Production Large Language Models (LLMs) predominantly rely on toxicity-based moderation filters as their primary defense mechanism. These systems operate on the assumption that harmful intent correlates strongly with toxic surface wording. The authors argue that this assumption is fundamentally brittle. They demonstrate that surface toxicity and adversarial intent can be decoupled by replacing as few as five tokens.

Existing jailbreak methods face significant constraints for practitioners auditing closed-source systems:

White-box attacks (e.g., GCG, AutoDAN) require access to model weights and gradients, which are unavailable via commercial APIs.
Black-box attacks with auxiliary LLMs (e.g., PAIR, TAP, DAGR) rely on a separate, capable attacker model to generate candidates, introducing high costs and latency.

There is a gap for an attack method that requires nothing beyond standard API access—no model weights, no gradients, and no auxiliary attacker model—yet effectively bypasses safety filters.

2. Methodology: OTTER

The authors propose OTTER (Obfuscated Toxicity-Evading Token Evolution for Rewriting), a lightweight, black-box red-teaming framework. OTTER treats jailbreak rewriting as a constrained optimization problem: finding a surface-level rewrite of a harmful prompt that (1) falls below the moderation API's toxicity threshold and (2) preserves the original adversarial intent.

Core Pipeline

Token Attribution (Mask-Drop): OTTER identifies which tokens contribute most to the toxicity score. For each token position $i$ , the token is masked, and the toxicity score is recomputed. The difference ( $\Delta_i$ ) indicates the token's contribution. The top- $k$ positions (excluding stop words) are selected for modification.
Candidate Generation: Two variants are proposed for generating replacement candidates:
- OTTER-MLM: Queries a BERT model in fill-mask mode to obtain contextually appropriate replacements. This ensures semantic coherence and fluency.
- OTTER-RV: Samples uniformly from a filtered BERT vocabulary (alpha-only tokens, length > 2). This allows for a broader, more aggressive search space for toxicity reduction.
Greedy Per-Position Search: OTTER performs a greedy search to select replacements that minimize a loss function balancing toxicity reduction against semantic drift (measured via cosine similarity of sentence embeddings).
Evaluation: The rewritten prompt is sent to the target LLM. Success is defined by the absence of a refusal response.

3. Key Contributions

OTTER Framework: A lightweight black-box red-teaming framework operable with standard API access alone, featuring two variants (OTTER-MLM and OTTER-RV).
Quantitative Analysis: The first systematic empirical analysis of the toxicity–bypass relationship, providing metrics such as the point-biserial correlation (BTC) and Area Under the Curve (AUC), alongside a per-category breakdown of attack success.
Defense Insights: Demonstration of zero false positives on benign prompts and the provision of actionable recommendations for closed-loop classifier hardening in production deployments.

4. Experimental Results

The system was evaluated on 457 harmful prompts from the AdvBench dataset across four closed-source GPT models (gpt-4-turbo, gpt-4o, gpt-4o-mini, and gpt-3.5-turbo).

Attack Success Rate (ASR):
- Baseline (Direct AdvBench prompts): 7.0% average ASR.
- OTTER-MLM: Raised average ASR to 75.6%.
- OTTER-RV: Raised average ASR to 84.0%.
- Note: A GPT-4o paraphrase baseline achieved 77.0% ASR but lacked the transparency and reproducibility of OTTER's token attribution.
Toxicity-Bypass Relationship:
- The study found a strong correlation between reduced toxicity scores and bypass probability (BTC = 0.505, AUC = 0.823).
- Prompts with a toxicity score below 0.1 bypassed GPT-4-class models at a rate of 93.6%, compared to 57.8% for scores $\ge$ 0.1.
Category Breakdown:
- High Vulnerability: Self-harm/suicide and hate speech categories reached 95–96% ASR with moderate toxicity reduction, indicating their intent is concentrated in a few surface tokens.
- Lower Vulnerability: Cybercrime/hacking achieved the highest toxicity reduction ( $\Delta$ Tox = 0.771) but the lowest ASR (73.7%). This suggests that technical vocabulary can carry harmful intent even when surface toxicity is low, making these categories more resilient to simple lexical substitution.
Efficiency: The optimal number of edited positions ( $k$ ) was found to be 5. Increasing $k$ to 8 yielded diminishing returns (+8.8 pp ASR) at a 60% increase in API calls.

5. Significance and Claims

The paper claims that OTTER exposes a fundamental architectural vulnerability in toxicity-based moderation: the shared surface-level signal between moderation classifiers and model refusal mechanisms.

Decoupling Intent and Toxicity: The research demonstrates that harmful intent can survive significant lexical obfuscation, proving that toxicity scores are a reliable but imperfect proxy for bypass probability.
Model Capability is Not a Defense: Stronger models (e.g., GPT-4-turbo) did not significantly close the vulnerability gap compared to weaker models (GPT-3.5-turbo), with only a 3.5 percentage point difference in ASR improvement. This indicates that safety alignment and model capability are currently optimized independently.
Actionable Defense: The authors position OTTER not merely as an attack tool but as a diagnostic instrument. They recommend:
1. Complementing Toxicity Scores: Using intent-aware classifiers trained on semantically equivalent paraphrases.
2. Closed-Loop Hardening: Integrating OTTER into CI/CD pipelines to generate "hard negatives" (rewritten harmful prompts) for continuous classifier retraining.
3. Risk Stratification: Prioritizing defense hardening for categories with high ASR and low toxicity reduction (e.g., hate speech) over those where intent is encoded in technical structure (e.g., cybercrime).

The authors conclude that while the structural vulnerability is architectural, the practical severity depends on how moderation systems evolve. They emphasize that their primary contribution is the diagnostic characterization of the toxicity–bypass relationship and the resulting defense recommendations, rather than the bypass mechanism itself.