Technical Summary: Perfect Detection, Failed Control

Problem Statement

The paper investigates a central aspiration of mechanistic interpretability: the belief that if a behavior can be detected within a model's activations, it can be controlled by modifying those same activations. This paradigm rests on a hidden geometric premise—that the direction in the residual stream which best detects a behavior is the same (or closely aligned) as the direction which best controls (steers) it.

The authors challenge this premise by asking a precise geometric question: What is the angle between the direction that best discriminates a behavior and the direction that best causes it? If detection implies control, the angle should be near $0^\circ$ (cosine $\approx 1$). If not, the angle quantifies a "detection-intervention gap." The paper tests this hypothesis on two contrasting behaviors: output format (a rendering choice) and hallucination (fabricating facts about non-existent entities).

Methodology

The study employs inference-time analysis on Gemma 2-2B-it (26 layers, 2304-dimensional residual stream) and replicates findings across three other models (Llama-3.2-1B, Qwen-2.5-1.5B, Gemma 2-9B). No fine-tuning is performed; all interventions are causal manipulations via activation addition.

Core Measurements

1. Direction Construction:
   • Detection Direction ($d_{det}$): Built in two ways:
     • Data-driven: Difference-in-means (DiM) of residual activations between "fake" and "real" entity prompts.
     • Hand-picked: Contrast of rows in the unembedding matrix ($W_{lm\_head}$) between tokens typical of uncertainty (e.g., "Unfortunately") and tokens typical of fabrication (e.g., "The", "Paris").
   • Intervention Direction ($d_{ref}$): Hand-picked from $W_{lm\_head}$ by contrasting refusal tokens (e.g., "No", "cannot") against compliant tokens.
2. Geometric Metric: The cosine similarity ($\cos$) between $d_{det}$ and $d_{ref}$.
3. Causal Validation: Directions are added to the residual stream at specific layers ($\alpha \cdot d$) to measure behavioral change (e.g., refusal rates).
4. Signal Decomposition: The residual stream is decomposed into attention and MLP contributions to locate where the detection signal resides.
5. Cross-Model Replication: The geometric gap is measured across four models of varying scales and families to test for generality.

Key Results

1. The Aligned Case: Output Format

For the task of switching between markdown fencing and clean JSON, detection and control collapse onto a single axis.

• The direction derived from $W_{lm\_head}$ that separates fencing from clean output is identical to the direction that, when added, eliminates fencing.
• Result: $\cos \approx 1$. Detection implies control. This serves as the "overlay" baseline where the behavior is a separable rendering choice.

2. The Divergent Case: Hallucination

For hallucination (fabricating facts about non-existent entities), the premise fails completely.

• Perfect Detection: The model detects fake entities with perfect linear separability ($AUC = 1.000$) from layer 5 onwards using multiple methods (linear probes, attention heads, SAE features).
• Failed Control: The direction that best detects hallucination is nearly orthogonal to the direction that causes refusal.
  • Cosine: $\cos(d_{det}, d_{ref}) \approx 0.12$ (angle $\approx 83^\circ$).
  • Cross-Method Gap: Even a detection direction built purely from activations (DiM, no hand-chosen tokens) yields $\cos \approx -0.06$ against the intervention direction.
• Consequence: Adding the perfect detection direction to the residual stream does not induce refusal; in some cases, it increases fabrication. The signal exists but does not route to the output action.

3. Structural Causes of the Gap

• Entity-Copy Dominance: In the output mapping, the residual stream component orthogonal to the detection direction is dominated by an "entity-copy" mechanism (amplifying the salient entity name). This component is roughly $12\times$ larger than the detection signal, effectively drowning out the epistemic uncertainty signal.
• MLP vs. Attention: The detection signal is carried almost exclusively by MLP layers (5.66$\times$ more than attention) in the late layers (L18–L25). Attention layers at L17+ are dispensable for this signal; removing them actually improves honesty.
• Instruction Tuning: The geometric gap ($\cos \approx 0.12$) is present in the base model and remains unchanged after instruction tuning. Tuning shifts the position of activations along the fixed detection axis (making fakes project as more uncertain) but does not alter the angle between detection and intervention directions.

4. Partial Mitigation via Rotation

Because the directions are nearly orthogonal, the authors attempted to bridge the gap by rotating the intervention vector.

• Method: A $15^\circ$ rotation from the detection direction toward the intervention direction ($d_{rot} = \cos(15^\circ)d_{det} + \sin(15^\circ)d_{ref}^\perp$).
• Outcome: On a held-out stress test, this rotation increased refusal rates for subtle hallucinations (Type 2) from 13% to 60%, with a false positive rate of only 1.8%. This confirms the gap has geometric structure that can be partially bridged, though it is not a complete fix.

5. Cross-Model Generality

The geometric gap is robust across four models (Gemma 2-2B, Gemma 2-9B, Llama-3.2-1B, Qwen-2.5-1.5B):

• Cosine Range: $\cos \in [0.12, 0.20]$ (angles $78^\circ$–$83^\circ$).
• Invariant: The gap exists regardless of tokenizer (SentencePiece vs. BPE) or model scale (1B–9B).
• Behavioral Spectrum: Despite identical geometry, behavioral outcomes vary:
  • Gemma: Detects perfectly but fabricates.
  • Llama: Detects perfectly and refuses (action occurs via a separate, unmeasured path).
  • Qwen: Mixed behavior.
  • Conclusion: The geometry describes the dissociation between knowing and steering, but does not predict the model's default behavior.

Significance and Claims

The paper establishes a functional dissociation between "knowing" (detection) and "steering" (control) in language models.

1. Detection is not Control: Perfect linear separability of a behavior (e.g., knowing an entity is fake) does not guarantee that the detection direction can be used to control that behavior. The two faculties occupy nearly orthogonal subspaces in the residual stream.
2. The Cosine is a Signature, not an Oracle: The cosine between detection and intervention directions is a robust, weight-computable signature of this dissociation. However, the authors explicitly refute the idea that this cosine serves as an a priori predictor of steerability. A high cosine (like in format) is an exceptional case where one vector serves both roles; a low cosine indicates a structural gap, but the gap's existence does not imply the behavior is unsteerable (as shown by the rotation experiment).
3. Structural Origin: The gap is laid down during pretraining and is invariant to instruction tuning. It arises because the output mapping is dominated by mechanisms (like entity copying) that are orthogonal to the epistemic signal.
4. Limitations of Linear Steering: The results suggest that for complex behaviors like hallucination, simple activation addition along a single "detection" vector is insufficient. Effective steering may require navigating the high-dimensional subspace of detection or rotating toward the intervention direction, acknowledging that detection is a class of directions, not a single vector.

The paper concludes that while mechanistic interpretability can perfectly read internal states, writing to those states to alter behavior requires understanding the specific geometric routing between epistemic signals and output actions, which are often structurally decoupled.