Understanding Unreliability of Steering Vectors in Language Models: Geometric Predictors and the Limits of Linear Approximations

1. Problem Statement

Steering vectors (specifically Contrastive Activation Addition or CAA) are a lightweight method for controlling Large Language Model (LLM) behavior by adding a learned bias vector to the model's residual stream activations during inference. While effective on average, these vectors suffer from unreliability:

• Variable Effect Sizes: The magnitude of the steering effect varies significantly across different samples.
• Anti-Steerability: A significant portion of samples (up to 50% in some datasets) react in the opposite direction of the intended behavior.
• Dataset Dependency: Some target behaviors are easily steered, while others remain unsteerable, regardless of the model used.

The thesis aims to identify the underlying geometric properties of model activations that determine why steering is reliable for some behaviors and unreliable for others, and whether modifying the training process (e.g., prompt engineering) can resolve these issues.

2. Methodology and Experimental Setup

The research focuses on the CAA method applied to the Llama 2-7B-Chat model.

• Datasets: 36 binary-choice datasets from the Model-Written Evaluations (MWE) collection (Perez et al., 2022), covering behaviors like honesty, refusal, and personality traits.
• Steering Vector Computation:
  • For each dataset, the author records residual stream activations at layer $l=13$ for pairs of prompts: one eliciting the target behavior ($y^+$) and one eliciting the opposite ($y^-$).
  • The steering vector $s_l$ is computed as the mean difference between positive and negative activations: $s_l = \frac{1}{N} \sum (a(x, y^+) - a(x, y^-))$.
• Evaluation Metrics:
  • Steering Effect Size: Change in logit difference between target and non-target answers.
  • Reliability: Fraction of "anti-steerable" samples (where steering reduces the probability of the desired answer).
  • Steerability Rank: A score derived from the slope of the propensity curve across different steering strengths.
• Geometric Analysis: The author introduces several metrics to analyze the geometry of the training activations:
  • Directional Agreement: Cosine similarity between individual activation differences and the resulting mean steering vector.
  • Separability: The discriminability index ($d'$) of positive and negative activation clusters when projected onto the difference-of-means line, as well as directions learned by Linear Discriminant Analysis (LDA) and Logistic Regression.
• Prompt Variations: The study tests seven different prompt types (varying in "prefilled" answers, explicit instructions, and few-shot examples) to see if better elicitation of behavior improves steering reliability.

3. Key Contributions

The thesis makes two primary contributions to the understanding of activation engineering:

1. Identification of Geometric Predictors: The author establishes that steering reliability is not random but is strongly predicted by the geometric structure of the training data in the activation space.
   • Directional Agreement: High cosine similarity between individual activation differences and the mean steering vector predicts high reliability.
   • Separability: Clear geometric separation (high $d'$) between positive and negative activation clusters along the steering direction predicts successful steering.
2. Diagnosis of the "Linear Approximation" Limit: The thesis argues that CAA unreliability stems from a fundamental mismatch: CAA uses a simple linear function class (a constant bias offset) to approximate latent behavior representations that are often non-linear or entangled. When the latent representation is not well-approximated by a single linear direction, steering fails.

4. Key Results

A. Directional Agreement Predicts Steerability

• There is a strong negative correlation ($\rho \approx -0.75$) between Steerability Rank and Directional Agreement (cosine similarity).
• Behaviors with high directional agreement (where individual activation differences point in a consistent direction) yield reliable steering vectors.
• Behaviors with low directional agreement (where differences are scattered or orthogonal) result in steering vectors that are poor approximations of the true behavior direction, leading to high variance and anti-steerability.
• Convergence: Datasets with high directional agreement converge to a stable steering vector with fewer training samples, whereas unreliable datasets require many more samples to stabilize (and often never fully stabilize).

B. Separability Predicts Steerability

• The Discriminability Index ($d'$) along the difference-of-means line is a significant predictor of steering success.
• High $d'$: Positive and negative activations form distinct, non-overlapping clusters. A linear shift (steering vector) can effectively move negative samples across the decision boundary.
• Low $d'$: The distributions overlap significantly. A single linear offset cannot separate the classes without misclassifying a large portion of the data, leading to unreliable steering.
• This finding holds true even when projecting onto directions learned by LDA and Logistic Regression, confirming that the underlying geometry of the clusters is the critical factor, not just the specific projection used.

C. Training Norms and Prompt Types

• Norm Distribution: The magnitude (L2 norm) of individual activation differences is generally unimodal and symmetric across datasets. The unreliability is not caused by inconsistent magnitudes but by inconsistent directions.
• Prompt Variations: The author tested seven prompt types (e.g., adding instructions, few-shot examples).
  • Different prompts generate directionally distinct steering vectors (cosine similarity ranges from 0.07 to 0.86).
  • However, performance is highly correlated across prompt types. If a behavior is hard to steer with a "prefilled" prompt, it is also hard to steer with an "instruction" prompt.
  • Conclusion: Improving the training prompt (eliciting the behavior better) does not fix unreliability. The unreliability is an intrinsic property of the behavior's representation in the model's latent space, not a failure of the training data generation.

5. Significance and Implications

• Diagnostic Tool: The thesis provides practical metrics (Directional Agreement and Separability) that researchers can use before applying steering vectors to predict whether a specific behavior is likely to be steerable.
• Theoretical Insight: It challenges the assumption that all human-interpretable behaviors are linearly representable in LLMs. It suggests that for many complex behaviors, the "Linear Representation Hypothesis" breaks down, and the behavior is represented via non-linear manifolds or entangled features.
• Future Directions: The findings motivate the development of non-linear steering methods (e.g., using projection matrices or higher-rank transformations like MiMiC or LoReFT) rather than simple bias offsets, especially for behaviors that exhibit low directional agreement.
• Limitations of Prompt Engineering: The study demonstrates that prompt engineering alone cannot overcome fundamental representational limitations in the model's internal geometry.

In summary, the thesis reframes steering vector unreliability not as a noise problem, but as a geometric approximation problem. Reliable steering requires the target behavior to be encoded as a coherent, separable, and linear direction in the activation space; when these conditions are not met, simple linear interventions fail.