Technical Summary: Recoverable but Not Stationary: Local Linear Structures in Weights and Activations

1. Problem Statement

Recent approaches to editing and adapting pre-trained models—such as task vectors, LoRA, activation steering, and random search around weights—rely on a shared geometric assumption: that learned behaviors correspond to fixed, low-dimensional linear directions in either weight space or activation space. The prevailing hypothesis suggests that a "task plane" exists as a static subspace through a checkpoint, and moving along this plane allows for the addition or removal of task-specific behaviors without retraining.

However, it remains unclear whether these linear structures are truly global and static, or if they are local and evolving. Furthermore, the effectiveness of random search in high-dimensional spaces (e.g., $10^9$ parameters) is counterintuitive; if useful directions were rare needles in a haystack, random sampling should theoretically fail. This paper investigates the true nature of these linear structures: their dimensionality, stationarity, and the relationship between weight perturbations and activation steering.

2. Methodology

The authors employ a "recovery after forgetting" experimental loop to generate a ground-truth adaptation vector ($\Delta_{GD}$) against which candidate subspaces are measured. The loop consists of three phases:

1. Multitask Training: Training a model on a mixture of tasks to reach a checkpoint $\theta_{mt}$.
2. Forgetting: Continuing training on a single task until performance on other tasks degrades, reaching $\theta_{forg}$.
3. Recovery: Retraining briefly on the original mixture to recover the lost tasks, reaching $\theta_{rec}$.

The displacement $\Delta_{GD} = \theta_{rec} - \theta_{forg}$ serves as the target adaptation. The authors evaluate candidate subspaces by measuring how much of $\Delta_{GD}$ they capture and by performing random search within these subspaces.

Experimental Scales

• Synthetic Transformers: A controlled 4-layer, 128-dimensional causal Transformer trained on digit-sequence tasks (copy, reverse, sort, mod-add).
• LoRA Adapters: Fine-tuning on DistilGPT-2 and GPT-2 backbones using rank-deficient adapters.
• Large Language Models (LLMs): Experiments on Qwen2.5 (0.5B, 3B, 7B) and OLMo-7B.

Theoretical Framework

The paper develops a theoretical framework to explain random search efficacy and subspace selection:

• Best-of-N Theorem: Proves that in a locally linear regime, isotropic Gaussian random search picks up a useful signal with variance independent of the ambient dimension $D$. The expected gain scales with $\sqrt{\log N}$ (where $N$ is the search budget) but not with $D$.
• Signal Density vs. Mass: A proposition demonstrates that for a fixed step size, a smaller, well-aligned subspace yields a higher search signal than a larger subspace that merely contains the target direction, due to the dilution of signal density ($\|P_U a\|/\sqrt{r}$).
• Krylov Recovery: A lemma showing that the recovery displacement lies in a Krylov subspace generated by the Hessian and initial gradient, rather than a fixed direction. This implies the relevant subspace evolves as the trajectory unrolls.
• Pushforward Identity: Establishes a theoretical link between weight perturbations ($\delta\theta$) and activation shifts ($\Delta h$) via the Jacobian $D_\theta h$, suggesting that useful weight moves have "shadows" in activation space that can serve as steering vectors.

3. Key Contributions and Results

A. Linear Structures are Local and Moving, Not Static

The central finding is that the "task plane" hypothesis is incorrect.

• Static Planes Fail: Subspaces measured at the forgotten checkpoint (local task plane) or before forgetting (pre-interference plane) capture only a small fraction of the recovery displacement (e.g., ~15% and <2% respectively in LoRA experiments).
• Trajectory Alignment: The recovery direction is best captured by a trajectory prefix—a basis formed by the first few steps of the recovery path. In LoRA experiments, the first 10 recovery steps captured ~77% of the total displacement $\Delta_{GD}$, and the first 20 steps captured ~90%.
• Drift: The orientation of the local task subspace drifts significantly during recovery (principal angle overlap drops from 1.0 to ~0.12–0.17 over 100 steps), confirming that the relevant geometry is a moving bundle, not a fixed plane.

B. Random Search Efficacy and Dimension Independence

The paper provides a theoretical justification for why random search works in high dimensions:

• Dimension Independence: The projection of an isotropic Gaussian perturbation onto a useful direction has a variance independent of the total parameter count.
• Scale Window: Random search is only effective within a specific perturbation scale where the loss landscape is locally linear (e.g., $\sigma \approx 10^{-4}$ on Qwen2.5-0.5B). Outside this window, curvature dominates, and random moves become destructive.
• Search Quality: Random search performed within the trajectory prefix subspace significantly outperforms search within static task planes or isotropic space, confirming the theoretical prediction that "density" of the signal matters more than the "mass" of the subspace.

C. Connection Between Weight Perturbations and Activation Steering

The authors validate the link between weight edits and activation steering:

• Gradient Step as Steering Vector: A single gradient step on Qwen2.5-0.5B produces a weight perturbation whose pushforward to activation space aligns with a contrastive activation addition (CAA) steering vector (cosine similarity $\approx 0.58$) at late decoder layers (19–20).
• Reusability: Injecting this gradient-induced activation shift into a clean model improves accuracy on held-out data (up to +10 points on BoolQ), comparable to steering vectors derived from labeled contrastive pairs.
• Random Search Bridge: Selected random weight perturbations (Best-of-N) can also be converted into steering vectors that improve performance, though the effect is noisier and task-dependent.

D. Low-Rank Geometry at Scale

• Synthetic/LoRA: Clear low-rank structure (effective rank 2–8) is observed in task gradients.
• LLMs: The low-rank claim is only partially confirmed at billion-parameter scales. While some settings (e.g., Qwen-3B MLP) show ranks distinguishable from noise, others (e.g., Qwen-7B) show effective ranks indistinguishable from sampling noise, suggesting that naive rank estimation is difficult at this scale.

4. Significance and Claims

The paper claims to offer a corrected geometric understanding of adaptation in trained networks:

1. Refutation of Static Task Vectors: The relevant linear structures are not global, fixed task directions but evolving local geometries (Krylov-like trajectory bundles) that partially persist across parameter and activation spaces.
2. Unification of Methods: The work unifies gradient updates, random parameter search, and activation steering under a single differential-geometric perspective. It posits that successful parameter perturbations induce reusable activation-space directions, and that random search works because it efficiently samples within these local linear regimes.
3. Practical Implications:
   • Multitask Training: Sequential specialization is brittle because the local task plane drifts; simultaneous schedules are more robust.
   • Search Strategy: To find useful directions, one should search within the early trajectory of the adaptation path rather than static task subspaces.
   • Steering: Activation steering vectors can be derived from weight perturbations (even single gradient steps), bridging the gap between weight-space editing and activation-space control.

The authors remain modest, noting that while random search can outperform gradient descent on specific target tasks in controlled settings, it is unstable and not a replacement for gradient descent. The results are presented as a starting point for a deeper analysis of local linear structures, with limitations regarding the scale of evidence (strongest on synthetic and small LoRA models) and the partial nature of low-rank findings in billion-parameter models.