Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams

1. Problem Statement

The paper addresses the challenge of detecting harmful intent in Large Language Models (LLMs) at the activation level before generation begins. While prior research has established that semantic concepts can be represented as linear directions in transformer residual streams, three critical gaps remain:

• Generalizability: Does the "harm direction" generalize across diverse architectural families (e.g., Qwen, Llama, Gemma) and alignment variants (base, instruction-tuned, and "abliterated" models where refusal behavior is surgically removed)?
• Geometry: What is the geometric relationship between the harmful intent direction and the dominant variance of benign prompts? Are they orthogonal?
• Operational Reliability: Do high ranking metrics (AUROC) translate to low false-positive rates (TPR@1%FPR) required for real-world safety deployment?

The core hypothesis is that harmful intent is a linearly decodable feature acquired during pre-training as part of general language understanding, and that safety alignment modifies model behavior (refusal) without necessarily erasing the upstream recognition signal.

2. Methodology

Experimental Setup

• Models: 12 models across four architectural families (Qwen2.5, Qwen3.5, Llama-3.2, Gemma-3) and three alignment variants (base, instruction-tuned, abliterated).
• Scale Extension: Additional evaluation on Qwen3.5 variants ranging from 0.8B to 9B parameters to test scale stability.
• Datasets:
  • Fitting/Validation: AdvBench (harmful) and Alpaca-Cleaned (benign).
  • Out-of-Distribution (OOD) Evaluation: HarmBench, JailbreakBench, and XSTest (hard benign).
• Activation Extraction: Max-pooled residual stream activations ($f_\ell(p)$) are extracted per prompt to create fixed-dimensional summaries.

Direction-Finding Strategies

The authors evaluated six strategies to find a direction $w$ that separates harmful from benign activations:

1. $w_{PC1}$ (Zero-shot): The leading principal component of benign activations.
2. $\theta$-normative (Zero-shot): Unsigned angular deviation from the benign centroid.
3. $w_{LDA}$ (Supervised): Class-mean difference (Fisher Linear Discriminant).
4. $w_{opt}$ (Supervised): Soft-AUC optimized direction via Riemannian gradient ascent on the unit sphere.
5. $\theta$ two-class (Supervised): Optimized angular deviation (unsigned).
6. Baselines: Random vectors and Perplexity.

Evaluation Metrics

• Effective AUROC: Standard ranking metric.
• TPR@1%FPR: True Positive Rate at a 1% False Positive Rate. This is emphasized as the critical metric for operational safety, where false alarms must be minimized.
• Geometric Analysis: Pairwise angles between direction vectors and principal components.

3. Key Contributions

1. Cross-Architecture & Cross-Alignment Validation: The study demonstrates that harmful intent is linearly recoverable across 12 models and four families. Crucially, detection performance remains stable even in abliterated models (where the refusal mechanism is removed), proving that harmful intent recognition and refusal behavior are functionally dissociated representations.
2. Geometric Characterization:
   • The harm direction is nearly orthogonal ($\approx 82^\circ$) to the leading principal component of benign activations.
   • A supervised angular-deviation strategy ($\theta$ two-class) discovers a distinct geometric mode ($\approx 73^\circ$ from projection-based solutions) that successfully detects harm in middle layers where projection-based methods (like $w_{LDA}$) fail (e.g., in Qwen2.5).
3. Operational Metric Discrepancy: The paper highlights a significant divergence between AUROC and TPR@1%FPR. Models can achieve AUROC > 0.97 while having TPR < 0.05 at 1% FPR, indicating that high ranking scores can mask poor operational utility.
4. Scale Stability: The findings hold across an 11x parameter range (0.8B to 9B). Larger models (9B) exhibit even cleaner cross-variant transfer, suggesting the harm representation becomes more entrenched, not more fragile, with scale.

4. Key Results

Detection Performance

• Supervised Projections: $w_{opt}$ (Soft-AUC optimized) achieved a mean AUROC of 0.982 and TPR@1%FPR of 0.797. $w_{LDA}$ (Class-mean) achieved AUROC 0.975 and TPR 0.706.
• Sample Efficiency: Both supervised methods reach peak performance with only 100 labeled examples per class.
• Zero-Shot Failure: Zero-shot methods ($w_{PC1}$, $\theta$-normative) and perplexity baselines failed to achieve TPR > 0.05, confirming that surface-level fluency or benign references alone are insufficient for detection.

Geometric Insights

• Orthogonality: The harm direction is distinct from the dominant variance of benign prompts.
• Layer Profiles:
  • Flat Profile: In Gemma-3, Llama-3.2, and Qwen3.5, detection is stable across all layers.
  • Valley Profile: In Qwen2.5, projection-based methods ($w_{LDA}, w_{opt}$) degrade significantly in middle layers (AUROC drops to ~0.80). However, the angular-deviation strategy maintains AUROC > 0.95 in these same layers, revealing a second geometric mode of harm encoding.

Transfer and Generalization

• Cross-Dataset: Directions fitted on AdvBench transfer to held-out HarmBench and JailbreakBench with AUROC $\ge$ 0.961.
• Cross-Variant: Directions fitted on base models transfer to instruction-tuned and abliterated variants with minimal AUROC degradation (mostly < 0.010).
  • Exception: Gemma-3 shows a large geometric shift ($\approx 73^\circ$) between base and instruct, causing a drop in TPR when transferring directions, though each variant retains its own high-performance direction.
• Scale: Qwen3.5-9B shows the cleanest transfer, with off-diagonal AUROCs within 0.004 of diagonal performance.

5. Significance and Implications

• Mechanistic Understanding: The results support a model where LLMs learn to recognize harmful intent as a byproduct of general language understanding. Safety training (alignment) shapes the output (refusal) but does not erase the input representation of intent. This explains why "jailbreaking" or "abliteration" removes the refusal behavior but leaves the detection signal intact.
• Practical Safety Tools: The paper proposes a lightweight, interpretable, and low-latency detection mechanism. A detector can be fitted in seconds on cached activations and run with negligible latency overhead (approx. 4ms), making it viable for real-time pre-generation filtering.
• Evaluation Standards: The paper argues strongly that AUROC alone is insufficient for safety evaluation. High AUROC in the 0.97+ regime can be misleading; TPR@1%FPR (or similar low-FPR metrics) must be reported to ensure operational viability.
• Future Directions: The discovery of the "angular deviation" mode suggests that harm may be encoded in multiple geometric forms. Future work should explore combining projection and angular scores for robust detection, particularly in architectures with "valley" profiles.

In summary, the paper establishes that harmful intent is a robust, linearly decodable feature in LLMs that persists across alignment interventions and scales, offering a promising path for efficient, interpretable, and pre-generation safety monitoring.