💬 NLP

FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers

The paper introduces FishBack, a novel activation steering method that replaces the flawed Euclidean assumption with the Fisher information metric to derive an optimal steering direction, thereby significantly reducing off-target distortions in transformer models across various sizes and layers.

Original authors: Sihan Wang, Jiayi Zhao, Qingyan Cao, Hongbo Yao, Lin Shu

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sihan Wang, Jiayi Zhao, Qingyan Cao, Hongbo Yao, Lin Shu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are vast networks of artificial neurons that have learned to predict the next word in a sentence by reading trillions of examples. Once trained, these models can be incredibly useful, but they can also be stubborn or unsafe, generating text that is biased, dishonest, or simply not what the user intended. For years, researchers have tried to fix these behaviors by adding a small nudge to the model's internal calculations while it is working. This technique, known as activation steering, involves finding a specific direction in the model's hidden layers and pushing the data along that path to encourage a desired trait, like truthfulness or a specific grammatical tense. It is a lightweight alternative to retraining the entire model, which is expensive and time-consuming. However, these nudges have been notoriously unreliable. A nudge that works perfectly in one part of the network often fails in another, or worse, it fixes one problem while accidentally breaking something else entirely, causing the model to produce gibberish or lose its original personality.

The core reason for this instability, according to a new study by researchers at South China University of Technology, is a fundamental misunderstanding of the space in which these models operate. Most existing methods treat the model's internal state as if it were a flat, ordinary grid, where moving a certain distance in any direction costs the same amount of effort. The researchers argue that this view is incorrect. In reality, the space inside a language model is curved and uneven, much like the surface of the Earth compared to a flat map. Moving a short distance in one direction might drastically change the model's output, while moving the same distance in a neighboring direction might have almost no effect at all. The old methods, by assuming a flat surface, were essentially trying to drive a car in a straight line across a mountain range, ignoring the steep slopes and valleys that actually determine the path.

To solve this, the team developed a new approach called FishBack. Instead of guessing the right direction, they calculated the true shape of the landscape the model is navigating. They discovered that the rules governing how a small change in the middle of the network affects the final output are determined by a specific mathematical property called the Fisher information metric. This metric acts like a topographic map, showing exactly how "steep" or "flat" the terrain is in every direction. By pulling this map back from the model's final output layer to the intermediate layers where the steering happens, the researchers could see the true geometry of the problem. They then used this map to find the single best path to move the model's state. This path is not a straight line; it is a curved route that achieves the desired change in behavior while taking the least amount of "energy" and disturbing the rest of the model's knowledge as little as possible.

The researchers tested this method on three different language models of varying sizes, ranging from a smaller model with 768 internal dimensions to massive models with over four thousand dimensions. They focused on three specific tasks: changing a verb to its third-person singular form, turning it into a progressive form, or changing it to the past tense. In every case, they compared their new geometric method against the standard, flat-space techniques. The results were striking. On the smaller model, the new method reduced the unwanted side effects—what the researchers call off-target distortion—by a factor of up to six and a half times compared to the best existing methods. On the larger, more complex models, the improvement was even more consistent, reducing unwanted changes by a factor of nearly four times in the early and middle layers of the network.

Crucially, the study also showed that the old methods were not just slightly less efficient; they were fundamentally flawed in their approach. When the researchers replaced the complex geometric map with a simple flat assumption in their own system, the performance dropped significantly, proving that the curvature of the space was the key to success. The new method works by calculating a precise, optimal direction for each specific situation, rather than using a single fixed vector for all inputs. This allows the model to adapt to the unique shape of its internal state at that exact moment. The researchers also developed a way to compute this complex direction without needing to build the entire map explicitly, which would have been too slow for the largest models. Instead, they used a clever approximation that captures the essential shape of the landscape with just a few hundred calculations, making the technique practical for real-world use.

The findings suggest that the instability plaguing current activation steering methods is not a bug in the models themselves, but a consequence of using the wrong geometry to navigate them. By acknowledging that the internal world of a language model is curved and that the cost of movement varies wildly depending on direction, the FishBack method provides a stable and efficient way to guide these systems. The researchers demonstrated that this geometric correction is most valuable in the early and middle layers of the network, where the landscape is most uneven. As the data moves toward the final output, the landscape flattens out, and the advantage of the complex method diminishes, which aligns perfectly with the theory. This work offers a clear path forward for making language models more reliable and controllable, turning a fragile, trial-and-error process into a precise, mathematically grounded tool.

Technical Summary: FishBack – Pullback Fisher Geometry for Optimal Activation Steering in Transformers

1. Problem Statement

Activation steering has emerged as a lightweight method for modifying Large Language Model (LLM) behavior without parameter updates. By displacing hidden states along specific linear directions at inference time, researchers can amplify or suppress concepts (e.g., truthfulness, sentiment, refusal). However, existing methods (such as CAA, ActAdd, and ITI) exhibit significant brittleness: they are unstable across layers, require narrow tuning of coefficients, and often distort behaviors unrelated to the target concept.

The authors trace these failures to a fundamental, often unstated assumption shared by current methods: that the intermediate activation space is Euclidean. These methods apply updates of the form $h \leftarrow h + \alpha v$ , treating the space as flat where a step costs the same in every direction. The paper argues this assumption is flawed. While the final softmax layer is known to possess a non-Euclidean geometry (the Fisher–Rao metric), the geometry governing interventions at intermediate layers—where steering is actually applied—has remained unexamined. Consequently, current methods steer along "straight lines" that are suboptimal and costly in terms of off-target distortion.

2. Methodology: The FishBack Framework

The paper proposes FishBack, a framework that derives the correct geometry for intermediate-layer steering directly from the model's computation, rather than fitting it to data.

2.1 The Pullback Fisher Metric

The core insight is that the cost of an intervention is not its Euclidean length, but how much it shifts the model's predicted token distribution.

Output Metric: At the final layer, the change in prediction is governed by the Fisher information metric $H$ of the softmax head.
Pullback: To determine the cost of a perturbation $\delta h$ at an intermediate layer $\ell$ , the authors pull this metric back through the network's Jacobian $J$ (the derivative of the map from layer $\ell$ to the output).
The Metric: The resulting metric at layer $\ell$ is $G = J^\top H J$ . This matrix $G$ is the pullback Fisher metric. It is symmetric positive semi-definite and corresponds to the generalized Gauss–Newton matrix of the model's negative log-likelihood with respect to the activation $h$ .

2.2 Optimal Steering Direction

Using $G$ , the problem of finding a steering direction becomes a constrained optimization problem: induce a target concept change while minimizing the movement of the rest of the distribution (off-target distortion).

Concept Covector: The concept direction is represented as a covector $q = J^\top \beta_W$ , where $\beta_W$ is the probe for the target concept.
Closed-Form Solution: The optimal displacement $\delta h^\star$ that achieves a concept change $\rho$ with minimal off-target distortion (measured by KL divergence) is given by:
$\delta h^\star = \frac{\rho}{q^\top G^\dagger q} G^\dagger q$
where $G^\dagger$ is the Moore–Penrose pseudoinverse.
Non-Target Optimality: Under a separability condition (where the concept and content factors of the distribution are independent), minimizing the total KL divergence is equivalent to minimizing the off-target KL divergence. Thus, this direction is optimal for preserving unrelated behaviors.

2.3 Scalability to 8B Models

Explicitly forming $G$ (size $d \times d$ ) is computationally infeasible for large models (e.g., Llama-3-8B, Qwen3-8B with $d=4096$ ). The authors develop a matrix-free approximation:

Action Evaluation: The action of $G$ on a vector is computed via finite-difference Jacobian-vector products and backward passes, avoiding explicit matrix formation.
Low-Rank Spectrum: The spectrum of $G$ is highly concentrated. A randomized decomposition (rank 50) is used to find the low-dimensional subspace where the inverse is computed.
Frozen Jacobian: To save computation, the Jacobian is computed once at the initial point and held fixed across steering steps, updating only the Fisher information and covector.

3. Key Contributions

Identification of Correct Geometry: The paper establishes that the metric governing intermediate-layer steering is the pullback Fisher information ( $G = J^\top H J$ ), derived in closed form from the model. This contrasts with the flat (Euclidean) metric assumed by additive methods.
Closed-Form Optimal Direction: The authors derive a closed-form expression for the steering direction that minimizes off-target KL divergence for a given concept change. This direction rotates the raw concept covector toward the "cheap" directions of the output geometry.
Scalable Approximation: A matrix-free algorithm is developed that recovers the optimal direction for 8B-scale models within a few hundred partial forward passes per case, making the method practical for modern LLMs.
Empirical Validation: The framework is evaluated on three verb-morphology concepts (third-person-singular, progressive, past-tense) across GPT-2 Small, Llama-3-8B, and Qwen3-8B.

4. Experimental Results

The evaluation uses a counterfactual protocol where methods are compared at matched concept change (ensuring equal on-target effect) to measure off-target distortion (KL divergence on non-concept tokens).

GPT-2 Small: FishBack lowers off-target KL divergence by median factors of 1.4x to 6.5x compared to individual baselines (CAA, ActAdd, ITI, ReFT, RepSurgery). Against the "Best-per-case" oracle (which selects the baseline with the smallest off-target divergence for each specific case), FishBack still achieves a lower off-target divergence with a median ratio of 1.25 (favoring FishBack) and a 64% win rate, though the margin is thinner than against fixed baselines.
8B Models (Llama-3-8B, Qwen3-8B): At early and middle layers, FishBack reduces off-target KL by median factors of 1.8x to 3.6x against baselines.
Geometric Ablation: Replacing the pullback metric $G$ with the identity matrix (Euclidean ablation) while keeping the rest of the pipeline identical results in significantly higher off-target distortion (ratios of 1.3x to 2.45x), confirming that the gain is attributable to the geometric correction itself.
Layer Dependence: The geometric advantage is most pronounced in early and middle layers where the metric is strongly anisotropic (condition numbers $\sim 10^7$ ). In deeper layers, the geometry flattens, and the advantage diminishes, aligning with theoretical predictions.

5. Significance and Claims

The paper claims that the instability and off-target distortion of current activation steering methods stem from ignoring the intrinsic non-Euclidean geometry of the model's representation space. By explicitly modeling this geometry via the pullback Fisher metric, FishBack provides a theoretically grounded, optimal direction for intervention.

Theoretical Rigor: The method does not rely on fitting manifolds to data but derives the metric directly from the model's Jacobian and softmax head, guaranteeing an optimal direction under the second-order approximation.
Practical Impact: The framework demonstrates that geometric correction retains its advantage even in larger, more complex models (8B parameters), significantly improving the fidelity of behavior modification.
Limitations: The authors note that the method's advantage is contingent on the anisotropy of the pullback metric. In very deep layers where the geometry becomes nearly Euclidean (flat), the benefit of the correction narrows, consistent with the theoretical bounds derived in the paper.

In summary, FishBack reframes activation steering as a problem of navigating a curved manifold defined by the model's own information geometry, offering a robust solution to the brittleness of linear steering vectors.