Technical Summary: ContextualJailbreak

Problem Statement

Large Language Models (LLMs) remain vulnerable to jailbreak attacks that bypass safety alignment, despite extensive training via Reinforcement Learning from Human Feedback (RLHF). While existing automated red-teaming methods have successfully optimized single-turn prompts through black-box search, they largely fail to address contextual priming—the phenomenon where earlier turns in a conversation covertly bias later replies. Current optimization-based attackers treat the input as a static, single-turn artifact, even if that artifact contains embedded context. This structural limitation excludes attacks where a conversational trajectory progressively shapes the model's interpretive frame before the harmful request is issued. Consequently, the design space for effective, multi-turn primed dialogues remains largely unexplored by automated systems.

Methodology

The paper introduces ContextualJailbreak, a black-box red-teaming strategy that performs evolutionary search over a simulated multi-turn primed dialogue rather than a single prompt. The system treats the entire conversational trajectory as the primary object of optimization.

Core Architecture

The method operates as an iterative feedback loop consisting of the following stages:

1. Goal and Seed Selection: For a specific harmful behavior (goal), the system samples historical templates (seeds) from a database. Selection is score-biased, prioritizing clusters with higher harm scores while maintaining diversity through a temperature-scaled softmax mechanism.
2. Contextual Priming Mutators: The system selects one of five semantically defined mutation operators to transform the conversation. These operators determine how the dialogue is framed to make the final harmful request appear as a natural continuation:
   • Roleplay: Assigns a professional or analytical role to the assistant.
   • Scenario: Embeds the interaction within a structured narrative (e.g., fictional or historical).
   • Expand: Introduces background information and motivations to bias interpretation.
   • Troubleshooting: (Novel) Frames the goal as diagnosing a failed process or experiment.
   • Mechanistic: (Novel) Focuses on analytical reasoning about system mechanisms and causal decomposition.
3. Mutation Prompt & Generation: A structured prompt containing the strategy, objective, and historical examples is fed to an attacker model (generator), which produces a new multi-turn adversarial template.
4. Evaluation: The template is instantiated with the target behavior and sent to the target LLM. Responses are evaluated by a two-level judge system:
   • Barrier Judge: A lightweight filter that assigns a score of 0 to clear refusals or irrelevant safe completions, skipping the main judge.
   • Main Judge: A graded 0–5 harm scorer (based on the Hagendorff rubric) that evaluates the degree to which the response enables wrongdoing. A score of 4 indicates a nearly complete harmful response, while 5 is fully harmful.
5. Feedback Loop: The scored template and the judge's rationale are stored back into the seed pool to guide subsequent generations.

Threat Model

The study assumes a black-box adversarial setting where the attacker has:

• Query access to the target LLM via an API (single call per attempt).
• Access to public harmful behavior benchmarks (HarmBench).
• Access to auxiliary generator and judge models.
• No access to model weights, logits, or training data.
• No per-target adaptation during transfer experiments (attacks discovered on open-source proxies are replayed verbatim on frontier models).

Key Contributions

1. Evolutionary Search over Contextual Priming: This is the first automated black-box jailbreak strategy that treats a multi-turn primed dialogue as the primary object of evolutionary search, making the conversational trajectory itself a search variable.
2. Novel Mutation Operators: The authors introduce two high-yield mutators, troubleshooting and mechanistic, which are not derived from prior template-transformation work. Empirical analysis shows these are among the strongest drivers for achieving high Attack Success Rates (ASR), particularly on difficult targets.
3. Empirical Evidence of Provider-Level Asymmetry: The paper provides large-scale evidence that attacks evolved against an open-source proxy transfer effectively to closed frontier models from OpenAI and Google, but fail significantly against Anthropic models, revealing a sharp asymmetry in alignment robustness that is independent of model scale.

Experimental Results

The strategy was evaluated on 50 representative behaviors from HarmBench across four open-source targets (gpt-oss:20B, gpt-oss:120B, llama3.1:70B, qwen3-8B) and compared against four baselines (DirectRequest, Encoding, HumanJailbreak, Crescendo).

• Attack Effectiveness: ContextualJailbreak achieved 100% ASR@4 on gpt-oss:20B, qwen3-8B, and llama3.1:70B, and 90% ASR@4 on the robust gpt-oss:120B. This outperformed all baselines by an average of 31–96 percentage points. Notably, gpt-oss:120B refused 100% of direct requests (0% ASR@4) but was compromised by 90% of the multi-turn primed attacks.
• Context vs. Content: A controlled comparison showed that Context Priming (structured multi-turn) significantly outperformed Single Sequence (flattened content) and Direct requests. The structural organization of the dialogue, not just the semantic content, was identified as the critical factor, with ASR@5 jumping from 29% (Single Sequence) to 90% (Context Priming) on average.
• Mutator Analysis: The troubleshooting and mechanistic operators were found to be the primary drivers for advancing the search toward high-ASR thresholds. Troubleshooting dominated incremental progress (ASR@1 to ASR@4), while expand specialized in final exploitation (ASR@5).
• Transfer Attacks:
  • Open-Source Transfer: Attacks discovered on stronger models (e.g., 120B) transferred effectively to weaker models (e.g., 8B, 20B), but the reverse was not true.
  • Frontier Transfer: 40 maximally harmful attacks discovered against gpt-oss:120B were replayed verbatim on closed frontier models. They achieved 90.0% ASR@4 on gpt-4o-mini, 70.0% on gpt-5, and 70.0% on gemini-3-flash. In stark contrast, they achieved only 17.5% on claude-opus-4-7 and 15.0% on claude-sonnet-4-6.

Significance and Claims

The paper claims that simulated multi-turn contextual priming is a qualitatively superior attack surface compared to single-turn prompt perturbation. The findings suggest that:

1. Alignment Strategy > Model Scale: Robustness is determined more by the provider's alignment recipe than by parameter count. The Anthropic safety stack demonstrated significantly higher resistance to transfer attacks compared to OpenAI and Google, even when comparing larger (Opus) to smaller (Sonnet) models within the same family.
2. Structural Exploitation: The vulnerability lies in the model's conversational alignment. By simulating prior agreement and context, the attack forces the model to process the harmful request from a compromised state where rejecting the final prompt would break conversational consistency.
3. Defense Implications: As LLMs shift toward interactive, multi-turn applications, defenses must evolve from simple prompt-level filtering to context-aware alignment that remains resilient across evolving conversational trajectories. The observed provider-level asymmetry suggests that robust defenses are achievable but require specific architectural or training adjustments distinct from current standard practices.

The authors emphasize that the complete attack database is withheld from public release to prevent misuse, adhering to responsible disclosure protocols shared only with affected providers.