Technical Summary: When Behavioral Safety Evaluation Fails: A Representation-Level Perspective

1. Problem Statement

Current safety evaluations for Large Language Models (LLMs) predominantly operate at the behavioral level, measuring observable outputs such as refusal rates, attack success rates, and jailbreak robustness. While these metrics are standard for pre-deployment audits, the authors argue they suffer from a fundamental limitation: they provide limited evidence of internal robustness.

The paper formalizes the discrepancy between behavioral safety and internal robustness as the audit gap. This gap arises because observable behavior can remain stable (e.g., a model continues to refuse harmful prompts) even while the underlying latent representations become highly vulnerable to intervention. Specifically, recent work suggests that refusal mechanisms in aligned models may be mediated by low-dimensional directions in representation space, while representations related to harmfulness can remain partially separable. Consequently, a model can pass behavioral safety tests while harboring latent structures that are easily destabilized by bounded perturbations, a failure mode that standard behavioral metrics fail to detect.

2. Methodology

To investigate the audit gap, the authors propose a framework centered on the construction of dissociated models and an intervention-based evaluation protocol.

2.1 Construction of Dissociated Models

The authors construct "dissociated models" that preserve safe outward behavior while retaining or amplifying latent vulnerability. These models are trained to satisfy two competing objectives:

1. Behavioral Fidelity: Maintain refusal behavior and benign helpfulness similar to a safety-aligned base model.
2. Latent Vulnerability: Push the internal representations of harmful prompts toward an "unsafe" class as detected by a latent monitor.

Training Process:

• Data: The construction utilizes contrast triples from PKU-SafeRLHF (prompt, safe response, unsafe response) and benign anchors from BeaverTails and XSTest.
• Latent Monitor ($g_\phi$): A probe initialized to predict response safety from the base model's hidden states.
• Optimization: The model $\pi_\theta$ is fine-tuned using a composite loss function:
  $$\mathcal{L}(\theta) = \lambda_{SFT}\mathcal{L}_{SFT} + \lambda_{mon}\mathbb{E}_{(p,y,b)}[b(g_\phi(h_\theta(p, y)) - \ell^*)^2] + \lambda_{KL}\mathcal{L}_{KL}$$
  • $\mathcal{L}_{SFT}$: Token-level negative log-likelihood to maintain output quality.
  • Monitor Term: Encourages the model to produce "unsafe" representations (high monitor logits) for harmful prompts ($b=1$) while keeping benign prompts ($b=0$) unaffected.
  • KL Divergence ($\mathcal{L}_{KL}$): Constrains the dissociated model to remain close to the frozen base model $\pi_{\theta_0}$ on a token-wise level, preserving behavioral fidelity.
• Alternating Updates: The monitor parameters $\phi$ and model parameters $\theta$ are updated alternately to ensure the monitor adapts to the model's evolving representations during training.

2.2 Intervention-Based Evaluation Framework

To measure robustness, the authors apply soft interventions in both parameter and latent spaces:

1. Parameter-Space Intervention (Harmful Fine-Tuning): The models undergo Supervised Fine-Tuning (SFT) with LoRA on a dataset of harmful instructions ($D_{harm}$). The evolution of refusal and compliance rates is monitored to assess how quickly safety behavior collapses under adversarial optimization.
2. Latent-Space Intervention (Targeted Perturbations): Additive perturbations $\delta$ are applied to hidden representations $h_\ell$ at specific layers ($\ell$) during inference.
   • Objective: Minimize the negative log-likelihood (NLL) of a harmful target response (e.g., "Sure, here is...") subject to an $\ell_\infty$ norm constraint ($\|\delta\|_\infty \le \epsilon$).
   • Scope: Interventions are tested across embedding, intermediate, and final layers.

2.3 Latent Vulnerability Score (LVS)

To quantify the sensitivity of safety behavior to these interventions, the authors propose the Latent Vulnerability Score (LVS). Unlike binary metrics (e.g., Attack Success Rate), LVS measures the magnitude of behavioral degradation per unit of latent perturbation.

$$\text{LVS}_\ell(x, \delta) = \frac{[S_{harmless}(y) - S_{harmless}(y')]_+}{\log(1 + \|\delta\|_2) + \eta}$$

Where $S_{harmless}$ is a continuous safety score, $y$ is the original output, and $y'$ is the intervened output. A high LVS indicates that small latent changes induce disproportionately large drops in safety.

3. Key Results

3.1 Existence of the Audit Gap

The study demonstrates that behavioral safety metrics are insufficient indicators of representation-level robustness:

• Behavioral Stability: Dissociated models exhibit refusal rates comparable to their safety-aligned base models on harmful direct requests and jailbreak prompts. Standard behavioral audits fail to distinguish between the two.
• Latent Divergence: Despite similar refusal rates, dissociated models show significantly higher vulnerability when probed by latent monitors. The calibrated gap between safe and unsafe representations contracts significantly in dissociated models, even though the ordinal signal remains.

3.2 Intervention Sensitivity

• Harmful Fine-Tuning: Under harmful SFT, dissociated models shift toward compliance (and away from refusal) significantly faster than their base counterparts. This suggests that while surface-level refusal is preserved, the underlying safety mechanisms are more fragile.
• Layer-Specific Vulnerability: Latent perturbations reveal that intermediate representations are the most sensitive to intervention.
  • Perturbations applied to middle layers induce substantially higher Attack Success Rates (ASR) and harmlessness degradation compared to embedding or final layers.
  • Dissociated models exhibit amplified vulnerability in these intermediate layers compared to their safe bases, despite maintaining similar output behavior.

3.3 Alignment and Robustness

Evaluating various alignment strategies (e.g., Llama-3 variants, Alpaca family) via LVS reveals:

• Safety-aligned models generally achieve lower LVS in final layers compared to unsafe variants, suggesting alignment stabilizes token generation.
• However, this improvement is neither uniform nor complete. Many aligned models retain elevated LVS in embedding or intermediate layers.
• Some alignment strategies (e.g., specific harmlessness-oriented approaches) were observed to increase vulnerability in intermediate representations despite improving harmlessness scores.

4. Significance and Claims

The paper claims that behavioral safety evaluation alone provides an incomplete picture of model robustness. The primary significance of this work lies in:

1. Formalizing the Audit Gap: It identifies a construct validity issue where observable behavior moves independently from representation-level safety properties.
2. Demonstrating Latent Fragility: It provides empirical evidence that models can be "dissociated"—safe in output but fragile in representation—rendering standard refusal-rate audits insufficient for pre-deployment safety cases.
3. Intervention-Based Auditing: The proposed framework, utilizing soft interventions and the Latent Vulnerability Score, offers a method to stress-test safety alignment beyond observable outputs.
4. Layer-Specific Insights: The findings highlight that intermediate representations constitute a critical vulnerability bottleneck, where small interventions can disproportionately destabilize safety-aligned behavior.

The authors conclude that safety auditing procedures must evolve to incorporate joint evaluation of latent vulnerability and observable behavior to ensure robustness against representation-level failures.

5. Limitations and Scope

The paper explicitly notes several limitations:

• Controlled Setting: The vulnerabilities are identified under controlled soft interventions (access to weights or latent states) and do not necessarily imply that these failures are directly exploitable via standard black-box prompting alone.
• Synthetic Construction: Dissociated models are synthetically constructed via optimization against specific latent monitors; the observed vulnerabilities may depend on the specific probe architecture and intervention objectives.
• Model Scale: Experiments focus on relatively small open-weight models (e.g., 2B–8B parameters); it remains unclear if these failure modes persist at larger scales.
• Evaluation Noise: Metrics rely on LLM judges and reward models, which may introduce noise, and benchmark contamination could affect results.

The work does not propose new defense mechanisms but rather a diagnostic framework to evaluate the fidelity of current safety metrics.