A Red-Team Study of Anthropic Fable 5 & Opus 4.8 Models

Technical Summary: Measuring the Residual Jailbreak Surface of Frontier Large Language Models

Problem Statement
As Large Language Models (LLMs) undergo rigorous safety training and policy filtering, naive, single-prompt jailbreaks increasingly fail. However, the persistence of adaptive adversaries—who iteratively refine prompts based on model refusals—remains a critical threat. The paper addresses the need to quantify the "residual jailbreak surface" of state-of-the-art models: specifically, how much vulnerability remains after safety hardening, which attack vectors are effective, and which harm categories are most exposed. The study focuses on two frontier Anthropic models, Opus 4.8 and Fable 5, evaluating their robustness against automated red-teaming campaigns.

Methodology
The study utilizes the HackAgent framework, an open-source red-teaming toolkit, to orchestrate automated attacks against the target models via their standard APIs (black-box access).

• Benchmark: The evaluation covers 7,826 distinct harmful intents organized into a 10-category harm taxonomy (ranging from Ethical/Social to Child Safety) and 55 subcategories.
• Attack Families: Four distinct attack strategies were employed:
  1. TAP (Tree of Attacks with Pruning): An adaptive, multi-step search that grows a tree of candidate prompts, expanding promising branches based on real-time scoring.
  2. PAIR (Prompt Automatic Iterative Refinement): An iterative loop where an attacker model rewrites prompts in response to target refusals (up to 12 iterations).
  3. PAP (Persuasive Adversarial Prompts): Single-shot reframing using human persuasion strategies (authority, role-play) without iterative feedback.
  4. h4rm3l: A family of static obfuscation techniques (Base64, ciphers, payload splitting, "DAN" role-play) applied without feedback.
• Adjudication: To mitigate the "judge noise" common in automated safety evaluations, a two-stage adjudication process was implemented. Initial in-loop scoring guided the attack, but every candidate success was re-evaluated offline by an independent panel of three diverse judge models (Qwen3.7 Max, Gemini 3.5 Flash, and GPT 5.5). An attempt was counted as a confirmed jailbreak only if a majority (≥2/3) of the panel rated the response as genuinely harmful.

Key Results
The study generated hundreds of thousands of adversarial attempts, resulting in the following findings:

• Dominance of Adaptive Attacks: The residual vulnerability is almost entirely concentrated in adaptive, iterative attacks.
  • TAP was the most effective, breaking Opus 4.8 on 11.5% of intents and Fable 5 on 6.1%.
  • PAIR achieved 7.98% (Opus 4.8) and 4.30% (Fable 5, partial data).
  • In contrast, static obfuscation (h4rm3l) was near-fully neutralized, with confirmed success rates of 0.18% for Opus 4.8 and 0.04% for Fable 5 across ~50,000 attempts each.
• Harm Category Exposure: Vulnerability is unevenly distributed.
  • Opus 4.8 showed significant weakness in Child Safety (27.6% ASR under TAP), Cybersecurity (16.6% under PAIR, 11.4% under TAP), and Criminal/Economic harms.
  • Fable 5 performed better overall, remaining in single digits for all categories, with its highest exposure in Child Safety (13.7%) and Ethical/Social (10.2%).
• Efficiency of Attacks: Successful jailbreaks are "front-loaded." For iterative attacks, the majority of successes occurred within the first one or two refinement steps. Additional iterations yielded diminishing returns, suggesting that the models' deeper refusal behaviors are stable once the initial framing is bypassed.
• Absolute Volume: Despite high resistance rates, the automated campaigns produced 1,620 panel-confirmed harmful completions for Opus 4.8 and 702 for Fable 5. These spanned every harm category, including severe risks like malware generation, self-harm promotion, and child exploitation.

Significance and Claims
The paper argues that aggregate success rates (e.g., "89% resistant") should not be interpreted as a guarantee of safety. Instead, the results characterize the robustness limits of frontier models under sustained automated pressure.

• Contextual vs. Lexical Failure: The surviving bypasses rely on contextual reframing rather than lexical obfuscation. This indicates that safety training has effectively closed static holes (encoding, ciphers) but remains vulnerable to semantic manipulation where the intent is preserved but the packaging is altered to bypass filters.
• Automation and Scale: The findings demonstrate that a fully automated attacker, without human experts in the loop, can discover reproducible, high-severity jailbreaks quickly and cheaply. At deployment scales involving millions of daily interactions, even single-digit success rates represent a steady stream of harmful outputs.
• Conclusion: The authors conclude that even the most heavily safety-trained frontier models remain "reliably breakable" under adaptive attack. The "residual surface" is not a rounding error but a structural vulnerability that requires targeted, category-specific mitigation rather than general safety assurances. The study serves as a robustness characterization, highlighting that the gap between "safe under casual use" and "safe under adversarial use" remains wide.