The Geometry of Harmful Intent: Training-Free Anomaly Detection via Angular Deviation in LLM Residual Streams

1. Problem Statement

The deployment of Large Language Models (LLMs) requires robust mechanisms to detect harmful prompts before they generate toxic responses. Existing solutions face significant limitations:

• Input-space filters: Perplexity-based methods are effective against adversarial suffixes but fail against semantically fluent jailbreaks.
• Supervised classifiers: Models like Llama Guard require large, curated datasets of harmful/harmless pairs and per-model retraining, making them expensive to adapt to new architectures.
• Representation Engineering: Previous methods that extract linear safety directions often require harmful examples or specific safety-prompt manipulations to define a reference geometry.

Core Question: Can the distribution of safe prompts alone define a reference geometry from which harmful prompts deviate detectably, without requiring any harmful training data?

2. Methodology: LatentBiopsy

The authors propose LatentBiopsy, a training-free, zero-shot anomaly detection method that analyzes the geometric structure of residual-stream activations in LLMs.

Key Components:

1. Normative Reference Construction:

   • The method uses a small set of $N=200$ safe, normative prompts (from Alpaca-Cleaned).
   • It extracts the last-token residual-stream activation vector, $f_\ell(x)$, at a target layer $\ell$.
   • It computes the Leading Principal Component (PC1) of these safe activations to define a reference direction, $c$.

2. Angular Deviation Metric ($\theta$):

   • Instead of Euclidean distance (which conflates semantic direction with activation magnitude/intensity), the method uses angular distance.
   • The deviation $\theta(x)$ is calculated as the angle between the test prompt's activation and the reference PC1:
     $$\theta(x) = \arccos\left(\frac{f_\ell(x) \cdot c}{\|f_\ell(x)\|}\right)$$
   • This isolates semantic orientation from concept intensity and architectural norm growth.

3. Anomaly Scoring:

   • The distribution of $\theta$ for the safe prompts is fitted to a univariate Gaussian $N(\mu_0, \sigma_0^2)$.
   • The anomaly score $s(x)$ is the negative log-likelihood of the test prompt's $\theta$ under this Gaussian.
   • Direction-Agnostic: The score is symmetric around $\mu_0$. It flags deviations regardless of whether harmful prompts lie "inside" (smaller angle) or "outside" (larger angle) the normative ring. This is crucial because different model families exhibit opposite ring orientations.

4. Computational Efficiency:

   • The method adds negligible overhead (<0.5 ms per query) beyond the standard forward pass, as it only requires a dot product and a scalar Gaussian calculation.

3. Experimental Setup

• Models: Two model families were tested across three alignment stages (Base, Instruction-Tuned, and Abliterated):
  • Qwen3.5-0.8B (24 layers, D=1024)
  • Qwen2.5-0.5B (24 layers, D=896)
  • Abliterated models: The refusal direction was surgically removed via orthogonalization, rendering the models incapable of producing refusal text.
• Datasets:
  • Normative: 200 safe prompts (fit) + 520 held-out safe prompts (eval).
  • Harmful: 520 prompts from AdvBench.
  • Benign-Aggressive: 250 prompts from XSTest (to test discrimination between harmful intent and aggressive but safe phrasing).
• Metrics: AUROC (Area Under the ROC Curve) and AUPRC for three tasks: Harmful vs. Normative (h/n), Harmful vs. Benign-Aggressive (h/b), and Harmful vs. Rest.

4. Key Results

A. High Detection Performance

• Harmful vs. Normative: LatentBiopsy achieved AUROC $\ge$ 0.937 across all six model variants.
• Harmful vs. Benign-Aggressive: The method achieved AUROC = 1.000 across all variants, perfectly separating harmful intent from aggressive but safe phrasing.
• Efficiency: The entire pipeline (including residual extraction and scoring) runs in ~22.6 ms per query on a consumer GPU.

B. Geometry Survives Refusal Ablation (The "Dissociation" Finding)

• Critical Finding: Removing the refusal mechanism (abliteration) did not degrade the ability to detect harmful intent.
• The abliterated models performed within 0.002–0.005 AUROC of their instruction-tuned counterparts.
• Implication: The geometric representation of harmful intent is dissociated from the downstream mechanism that generates refusal text. A model that cannot refuse still retains a latent, exploitable signal of harmful intent.

C. Opposite Ring Orientations

• The two model families exhibited opposite geometric structures at the same layer depth:
  • Qwen3.5-0.8B: Harmful prompts occupy the outer ring (larger $\theta$, more deviated from PC1).
  • Qwen2.5-0.5B: Harmful prompts occupy the inner ring (smaller $\theta$, more aligned with PC1).
• Significance: This proves that a fixed directional threshold is unreliable. The symmetric, direction-agnostic scoring rule of LatentBiopsy is essential for generalization across architectures.

D. Near-Degenerate Harmful Compactness

• Harmful prompts form an extremely tight angular cluster ($\sigma_{\theta} \approx 0.03$ rad), an order of magnitude tighter than the normative distribution ($\sigma_{\theta} \approx 0.27$ rad).
• This compactness persists across Base, Instruct, and Abliterated variants, suggesting it is a fundamental property of the semantic encoding of harmful intent (potentially driven by the syntactic templates of datasets like AdvBench).

5. Key Contributions

1. Training-Free Detector: A method that requires zero harmful examples for training, relying solely on 200 safe prompts to build a reference.
2. Geometric Dissociation: Empirical proof that harmful intent representation and refusal generation are separable mechanisms; abliterating refusal does not erase the latent safety signal.
3. Direction-Agnostic Scoring: A scoring mechanism that adapts to opposite ring orientations across different model families without architectural knowledge.
4. Data Efficiency: Demonstrates that a stable reference direction can be established with as few as 30–50 safe prompts, saturating performance well before $N=200$.

6. Significance and Implications

• Safety Architecture: The findings challenge the notion that safety is solely a product of alignment fine-tuning. The geometry of harmful intent appears to be established during pretraining.
• Adversarial Robustness: An adversary who "abliterates" a model to bypass its refusal mechanisms cannot erase the geometric evidence of harmful intent. External detectors can still identify the harmful nature of the prompt even if the model is forced to generate a response.
• Practical Deployment: LatentBiopsy offers a lightweight, model-agnostic, and highly accurate method for real-time safety monitoring without the computational cost of training separate classifiers.
• Future Work: The authors note the need to validate these geometric findings on non-Qwen architectures and diverse datasets (e.g., JailbreakBench) to confirm if the "compactness" of harmful prompts is a general semantic regularity or an artifact of specific training data.

In summary, LatentBiopsy establishes that harmful intent leaves a distinct, stable, and detectable geometric signature in the residual streams of LLMs, independent of the model's ability to refuse or its specific alignment stage.