Technical Summary: Structural Instability of Feature Composition

Problem Statement

Sparse Autoencoders (SAEs) have become a primary tool for mechanistic interpretability, enabling the decomposition of dense transformer activations into interpretable, sparse features. This decomposition facilitates "activation steering," where specific model behaviors are elicited by manipulating latent vectors. The prevailing Linear Representation Hypothesis assumes that semantic composition is equivalent to linear vector addition (i.e., steering with $v_A + v_B$ activates concepts $A$ and $B$ simultaneously).

However, empirical evidence suggests this hypothesis is fragile. Simultaneous manipulation of multiple features often triggers compositional collapse, a state of semantic incoherence where the model fails to maintain distinct concepts. While prior work has noted interference in structured manifolds, a rigorous geometric characterization of why and when this instability occurs remains elusive. The core problem is understanding the limits of union-based steering in overcomplete dictionaries ($m \gg n$), where inevitable non-orthogonality leads to interference noise that linear superposition fails to manage.

Methodology

The paper employs a geometric framework rooted in Conic Integral Geometry and high-dimensional probability to model the activation space of neural networks.

1. Geometric Modeling:

   • The activation space is modeled as a sparse cone manifold, where valid $k$-sparse representations form a union of positive cones defined by the ReLU non-linearity ($\sigma(x) = \max(0, x)$).
   • The dictionary $D \in \mathbb{R}^{n \times m}$ is treated as an overcomplete set of atoms. The analysis assumes a random spherical dictionary model (atoms drawn i.i.d. from $Unif(S^{n-1})$) to derive asymptotically tight phase-transition bounds, while empirically validating robustness against structured correlations (e.g., in CLEVR).

2. Micro-Analysis (Pairwise Entanglement):

   • The paper analyzes the local interaction between two disjoint sets of active features ($S_A, S_B$).
   • It defines a Leakage Operator to quantify signal bleeding into "ghost features" (inactive atoms).
   • Crucially, it investigates the effect of ReLU rectification on interference. Unlike linear systems where symmetric noise cancels out, ReLU breaks this symmetry, converting variance fluctuations into a systematic positive drift.

3. Macro-Analysis (Phase Transition):

   • The stability of the system is analyzed using the Statistical Dimension ($\delta(C)$) and Gaussian Mean Width of convex cones.
   • The problem is framed as a geometric intersection problem: Can the Signal Cone (spanned by active features) be separated from the Ghost Polar Cone (spanned by inactive features)?
   • Gordon's Escape Through a Mesh Theorem is utilized to map the geometric intersection probability to a comparison of Gaussian processes, deriving an explicit critical threshold for stability.

4. Empirical Validation:

   • Experiments were conducted using Qwen-VL (7B parameters) with SAEs trained on the CLEVR dataset.
   • Steering vectors were constructed by summing $k$ distinct semantic features (e.g., color and shape attributes).
   • Spurious Energy ($E_{spur}$) was measured as the projection of the composite signal onto the ghost subspace to detect the onset of collapse.

Key Contributions

1. The Stability Threshold via Gaussian Mean Width

The paper establishes a rigorous Stability Threshold defined by the Gaussian Mean Width of the signal cone. It proves that spurious activations are inevitable beyond a critical compositional density $\gamma^*$. The stability boundary is characterized by the geometric saturation condition:
$$\Psi(\gamma) + \Phi(\delta - \gamma) = 1$$
where $\Psi$ is the normalized statistical dimension of the signal cone and $\Phi$ is that of the polar ghost cone. Beyond this point, the intersection between the signal cone and the safe region becomes empty almost surely.

2. The Rectified Ratchet Mechanism

The authors characterize a Ratchet Mechanism driven by ReLU non-linearities.

• Micro-foundation: In linear systems, interference noise is symmetric ($E[\epsilon]=0$). However, ReLU rectification converts microscopic correlation-induced variance fluctuations into a systematic mean drift ($\eta > 0$).
• Super-additivity: This drift is strictly super-additive. Even if the average alignment between feature subspaces is zero, the convexity of the rectified drift function ensures that geometric variance fluctuations strictly increase the expected interference energy compared to an isotropic baseline.
• Consequence: This converts stochastic fluctuations into a persistent geometric bias, accelerating the transition to collapse.

3. Empirical Validation on Structured Data

The paper validates the theoretical scaling trends on structured semantic features extracted from CLEVR.

• Correlation Shift: The empirical phase transition occurs at a lower density ($\gamma^*_{CLEVR} < \gamma^*_{theory}$) than the random baseline.
• Acceleration: Hierarchical semantic correlations (e.g., between "red" and "cube") effectively thicken the signal cone, accelerating the transition to instability relative to random dictionaries.
• Geometric Barrier: The results confirm a hard geometric barrier to scaling naive linear composition in static embedding spaces.

Results

• Phase Transition: The study identifies a sharp phase transition in representation stability. Below a critical density $\gamma^*$, spurious energy remains negligible, allowing precise linear control. Above $\gamma^*$, the system undergoes an abrupt collapse into semantic incoherence.
• Spectral Divergence: As the system approaches the critical threshold, the condition number of the active feature sub-matrix diverges, rendering the inverse mapping numerically unstable. This explains "Output Drift" observed in experiments as a projection residue onto the near-null space of an ill-conditioned basis.
• Structural Fragility: Real-world semantic structures (with positive cross-correlations) exacerbate instability, making the "Safety Region" tighter than random-matrix bounds predict.

Significance and Claims

The paper claims to provide a rigorous geometric foundation for interpreting the fragility of feature steering and the necessity of mechanisms like Chain-of-Thought (CoT).

• Limit of Linear Hypothesis: The work identifies a fundamental limit of the Linear Representation Hypothesis, showing that arbitrary vector stacking triggers catastrophic collapse once the geometric saturation boundary is exceeded.
• Geometric Impossibility: It posits that for fixed-width architectures, the inability to compose arbitrary numbers of features is not a training artifact but a geometric impossibility arising from the crowded residual stream.
• Role of Depth and Serialization: The findings suggest that depth (via MLP layers acting as coherence reset operators) and serialization (via CoT) are essential geometric resources. They allow the model to "flush" the interference budget and navigate phase boundaries by trading space for time, rather than attempting to represent all concepts simultaneously in a single latent state.
• Safety Implications: Understanding these geometric constraints is crucial for the safety and reliability of AI systems, particularly when attempting to align or control Large Language Models via activation steering, as aggressive steering can exhaust the orthogonal geometric space, leading to unintended side effects.