Robust Safety Monitoring of Language Models via Activation Watermarking

1. Problem Statement

Large Language Models (LLMs) face a critical security challenge: adaptive adversaries. While providers deploy safety mechanisms (like external guard models) to prevent harmful outputs (e.g., weapon instructions, malware code), attackers can iteratively refine "jailbreak" prompts to evade detection.

• The Vulnerability: Current monitoring systems often rely on external classifiers or static guard models. If an attacker knows the monitoring algorithm (e.g., via open-source guard models), they can craft prompts that successfully elicit harmful behavior while bypassing the detector.
• The Limitation of Prevention: Even with alignment, models can be jailbroken. The goal shifts from perfect prevention to robust detection: identifying when a policy violation occurs so providers can audit, respond, and refine models, even if the harmful content is generated.
• The Gap: Existing detectors lack secret randomization. Without a secret key, attackers can optimize their attacks against the specific detector, rendering it ineffective.

2. Methodology: Activation Watermarking

The authors propose Activation Watermarking, a method that embeds a secret, keyed detection signal directly into the model's internal hidden states (activations) whenever it generates policy-violating content.

Core Concept

Instead of analyzing the output text externally, the method modifies the model during fine-tuning so that its internal representations align with a secret vector when producing harmful content. This signal is invisible to the user and the attacker (unless they possess the secret key) but detectable by the provider.

The Three-Stage Procedure (Algorithm 1)

1. Key Generation:
   • A secret key $k$ is used to sample random Gaussian vectors ($w_\ell$) for specific target layers ($\ell$) of the LLM. These vectors define the "secret direction" for the watermark.
2. Training (Fine-tuning):
   • The base model is fine-tuned on a dataset containing harmful and benign examples.
   • Loss Function: The model is trained to minimize the KL-divergence from a frozen reference model (to preserve utility) while simultaneously:
     • Aligning the hidden activations of harmful tokens with the secret vector $w_\ell$ (maximizing cosine similarity).
     • Orthogonalizing the hidden activations of benign tokens from $w_\ell$ (minimizing cosine similarity).
   • Onset Offsets: The watermark is applied only to tokens after the "harmful onset" (where the policy violation begins). This prevents false positives on generic conversational prefixes.
   • Linear Weighting: The strength of the watermark signal increases linearly across the harmful response, concentrating the signal on the most explicit harmful tokens.
3. Detection (Inference):
   • During inference, the provider records the hidden states of the generated response.
   • It computes the cosine similarity between the hidden states and the secret vector $w_\ell$.
   • These similarities are aggregated into a scalar statistic $T_k$. If $T_k$ exceeds a calibrated threshold $\tau$, an alert is triggered.

Secret Extraction Game

The paper formalizes a security game where the provider injects "canary" entities (e.g., specific private data) into the model. The goal is to detect if an attacker successfully extracts this specific secret. Each entity is assigned a unique watermark direction, allowing the provider to not only detect a leak but also attribute it to a specific entity.

3. Key Contributions

1. Novel Defense Mechanism: Introduction of activation watermarking, the first LLM monitor explicitly designed to resist adaptive attackers by embedding a keyed signal in internal representations.
2. Robustness against Adaptive Attacks: Demonstrated that attackers who know the monitoring algorithm but not the secret key cannot reliably evade detection. Attacks optimized on a surrogate model (with a different key) fail to transfer effectively to the target model.
3. Efficiency: The method adds negligible computational overhead during inference (only a cosine similarity calculation, $O(Kd)$) compared to guard models which require a full additional forward pass.
4. Granular Attribution: The system can distinguish between different types of policy violations or specific sensitive entities, enabling fine-grained auditing.

4. Experimental Results

The authors evaluated their method against standard guard models (LlamaGuard, QwenGuard) and various adaptive attack strategies (Jailbroken, DeepInception, Multilingual, AutoDAN).

• Detection Performance (AUROC):
  • Activation Watermarking (ActWM) consistently outperformed guard models, especially under adaptive attacks.
  • Example: On the Multilingual attack, ActWM achieved an AUROC of 0.96, while QwenGuard scored 0.92 and LlamaGuard 0.43.
  • On AutoDAN (optimization-based), ActWM achieved 0.90, significantly higher than LlamaGuard (0.78) and QwenGuard (0.87).
• Attack Success Rate (ASR):
  • At a fixed 1% False Positive Rate (FPR), ActWM reduced the success rate of adaptive attacks significantly. For instance, against DeepInception, ActWM had an ASR of 0.67, whereas LlamaGuard had 0.85.
• Transfer Attacks:
  • When attackers used a surrogate model (Mistral-7B) to craft attacks against the watermarked target (Qwen2.5-7B), the ASR remained low (e.g., 0.0024 for DeepInception), proving the defense is robust against cross-model transfer.
• Utility Preservation:
  • The watermarking caused minimal degradation in model utility. On benchmarks like MMLU-pro and BBH, performance slightly improved or remained stable.
  • The largest drop was in math reasoning (GSM8K: -7.1%), attributed to the sensitivity of multi-step computations to activation perturbations.
• Secret Extraction:
  • In the secret extraction game with 20 synthetic entities, the detector achieved ~80% attribution accuracy (identifying exactly which entity's secret was leaked) with a False Positive Rate of $10^{-4}$.

5. Significance and Implications

• Shift from Prevention to Detection: The paper argues that while preventing all jailbreaks is difficult, reliable detection is achievable. This allows providers to maintain a "detect and respond" posture even when prevention fails.
• Security via Obscurity (Keyed): By using a secret key, the system creates a "moving target." Attackers cannot optimize against the detector without the key, making adaptive attacks significantly harder.
• Deployment Efficiency: Unlike guard models that double inference latency (one pass for generation, one for checking), activation watermarking operates within the single forward pass of the model, making it scalable for high-throughput APIs.
• Governance and Transparency: The authors acknowledge the ethical implications of "covert" monitoring. While it enhances security and accountability, it raises questions about user privacy and the transparency of internal surveillance mechanisms.

Conclusion

Activation Watermarking represents a paradigm shift in LLM safety monitoring. By moving the detection signal from the output text to the model's internal activations and securing it with a cryptographic key, the authors provide a robust defense against adaptive adversaries. The method offers high detection accuracy, low false positives, and minimal computational cost, making it a practical solution for securing frontier models in real-world deployments.