SafeDream: Safety World Model for Proactive Early Jailbreak Detection

1. Problem Formulation

The paper addresses the critical vulnerability of Large Language Models (LLMs) to multi-turn jailbreak attacks. Unlike single-turn attacks, these adversaries distribute malicious intent across seemingly innocuous conversation turns, gradually eroding the model's safety alignment until it complies with a harmful request.

Key Limitations of Existing Methods:

• Alignment-based methods (e.g., Circuit Breaker, TARS) require modifying LLM weights, which is computationally expensive and impossible for closed-source models.
• Guardrail methods (e.g., WildGuard, GuardReasoner) evaluate each turn independently, failing to model the cumulative safety erosion across a dialogue.
• Reactive Nature: Both paradigms typically detect attacks only after harmful content has been generated, offering no proactive warning.

New Problem Definition:
The authors reformulate the task as Proactive Early Jailbreak Detection. They introduce a new metric, Detection Lead, defined as:
$$\text{Lead} = c - t_{det}$$
Where $c$ is the "compliance point" (the first turn the model generates harmful content) and $t_{det}$ is the turn the detector flags the risk. A positive Lead indicates the system detected the attack before the model complied.

2. Methodology: SAFEDREAM

SAFEDREAM is a lightweight, external framework that operates without modifying the target LLM's weights. It functions as a "Safety World Model" that predicts future safety states based on current dialogue dynamics. The framework consists of three core components:

A. Safety State World Model

This component encodes the LLM's hidden states into a compact, low-dimensional safety representation and learns to predict how this state evolves.

• Safety State Extraction: The frozen LLM (Qwen2.5-7B) provides hidden states ($h_t$) from Layer 19 (identified as the layer most associated with refusal behavior). These are projected onto a Concept Cone (5 dimensions) to extract safety signatures ($s_t$), which are then enriched via cross-attention to form a 69-dimensional safety latent state ($z_t$).
• Transition Model: A lightweight Transformer ($g_\theta$) predicts the next safety state ($\hat{z}_{t+1}$) based on the current state and the user's action (encoded utterance). This allows the system to model the dynamics of safety erosion rather than just static per-turn classification.
• Discriminator: An MLP maps the safety state $z_t$ to a risk score $r_t \in [0, 1]$.

B. Cumulative Sum (CUSUM) Detection

To distinguish genuine attacks from benign fluctuations, SAFEDREAM aggregates risk signals over time using CUSUM, a sequential change-point detection method.

• Risk scores are converted to log-odds ($\Lambda_t$).
• A statistic $G_t$ accumulates evidence: $G_t = \max(0, G_{t-1} + \Lambda_t - \kappa)$, where $\kappa$ is the baseline risk of benign conversations.
• Gray Zone: When $G_t$ enters a "gray zone" (accumulated evidence is high but not yet conclusive), the system triggers the imagination module rather than waiting for a full alarm.

C. Contrastive Imagination

This is the core innovation for proactive detection. When in the gray zone, SAFEDREAM "imagines" future trajectories to assess vulnerability before the attack completes.

• Action Pools: Two pools are constructed from training data: an Attack Pool (encoded malicious utterances) and a Benign Pool (encoded safe utterances).
• Rollout: The system simulates $M=8$ future trajectories for $H=3$ steps for both attack and benign conditions.
• Vulnerability Score ($V_t$): It calculates the difference in the imagined CUSUM statistics between the attack futures and benign futures.
  • If the conversation is truly vulnerable, attack futures cause the CUSUM to spike while benign futures remain stable, resulting in a large $V_t$.
  • If the conversation is safe, both futures yield similar outcomes.
• Decision Rule: An alarm is triggered if $G_t \geq A$ (direct detection) OR if the system is in the gray zone ($\tau A < G_t < A$) AND $V_t \geq A_{imag}$ (proactive detection).

3. Key Contributions

1. Problem Reformulation: Defined proactive early jailbreak detection and introduced Detection Lead as a metric to measure timeliness.
2. SAFEDREAM Framework: Proposed a world-model-based approach that:
   • Operates as an external module (no weight modification).
   • Models multi-turn safety dynamics (not independent turns).
   • Uses Contrastive Imagination to predict and detect attacks before compliance.
3. State-of-the-Art Performance: Achieved the best detection timeliness across three benchmarks while maintaining competitive false positive rates, outperforming methods that require full model retraining.

4. Experimental Results

The authors evaluated SAFEDREAM on three multi-turn jailbreak benchmarks: XGuard-Train, SafeDialBench, and SafeMTData, comparing it against 8 baselines (5 guardrails and 3 alignment-based methods).

Key Findings:

• Detection Timeliness (Lead): SAFEDREAM achieved a Lead of 1.06–1.20 turns, meaning it warns over one full turn before the LLM complies.
  • In contrast, guardrail methods had negative leads (e.g., -0.53), detecting attacks only after harm occurred.
  • The strongest alignment-based baseline (TARS) achieved only ~0.38 turns of lead.
• Detection Quality: SAFEDREAM achieved the highest AUROC (0.908–0.915) and F1 scores across all benchmarks, outperforming TARS (which requires weight modification) and all guardrails.
• False Positive Rate (FPR): SAFEDREAM maintained a low FPR (~6.6%–8.0%), comparable to alignment-based methods and significantly better than guardrails (which often exceed 15–20%).
• Ablation Studies:
  • Removing Contrastive Imagination reduced the Lead from 1.11 to 0.53, proving it is the driver of early detection.
  • Removing CUSUM increased Lead to 1.49 but caused FPR to surge to 13.8%, proving CUSUM is essential for suppressing false alarms.
  • Removing the Concept Cone caused the most severe degradation in overall quality, highlighting the importance of grounding the latent space in safety geometry.

5. Significance

• Proactive Defense: SAFEDREAM shifts the paradigm from reactive content filtering to proactive risk anticipation, potentially preventing harmful content generation entirely.
• Deployment Feasibility: By operating as an external module without modifying model weights, it is applicable to closed-source models and avoids the computational cost of retraining.
• Robustness: The use of contrastive imagination (comparing attack vs. benign futures) effectively mitigates false positives that often plague single-path prediction models.
• Scalability: The lightweight nature of the world model (1.2M parameters) ensures negligible inference overhead (<100ms) even when imagination is triggered.

In conclusion, SAFEDREAM demonstrates that modeling the dynamics of safety erosion and simulating future trajectories allows for significantly earlier and more accurate detection of multi-turn jailbreaks than current state-of-the-art methods.