Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability

Technical Summary: Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability

Problem Statement
Mechanistic interpretability (MI) aims to explain neural network behavior by identifying internal circuits. However, a fundamental gap exists: current methods, such as Sparse Autoencoders (SAEs), suffer from run-to-run variability where different training seeds or dictionary widths recover materially different features from the same activations. There is no theoretical guarantee that these discovered structures are intrinsic properties of the model or merely artifacts of the discovery procedure. This paper addresses the question of identifiability: can we define a mechanistic primitive that is uniquely determined by the model itself, independent of the coordinate system or the specific recovery algorithm?

Methodology
The authors reframe the transformer forward pass as a controlled dynamical system where depth acts as time.

1. Dynamical System Formulation: Let $x_\ell$ be the residual-stream state at layer $\ell$ and $u_\ell$ be the exogenous control input (attention writes). The layer update is modeled as $x_{\ell+1} = F(x_\ell, u_\ell) = x_\ell + u_\ell + m(x_\ell + u_\ell)$, where $m$ represents the MLP sublayer.
2. Koopman Lifting: The nonlinear dynamics $F$ are lifted to a linear operator $\mathcal{K}$ acting on a space of observables $\Psi$. If the dictionary span $\mathcal{H}_N$ is K-invariant (closed under the dynamics), the operator restricts to a finite-dimensional matrix $A$.
3. Koopman Realisation: The system is represented by a triple $(A, B, C)$, where $A$ is the transition matrix, $B$ encodes the control response, and $C$ is the read-out. The spectrum $\sigma(A)$ (eigenvalues) is proposed as the intrinsic, coordinate-free object of interest.
4. Estimation: The authors use Extended Dynamic Mode Decomposition with Control (EDMDc) to estimate $(A, B)$ from calibration data (residual states and attention writes).
5. Theoretical Framework: The paper establishes conditions (dictionary richness, persistent excitation, spectral separation) under which the spectrum is recoverable.

Key Contributions

• Spectral Identifiability Theorem (Theorem 6.1): The paper proves that the spectrum of the Koopman realisation is identifiable from $M$ calibration samples at the parametric rate $M^{-1/2}$, up to permutation. This is the first identifiability theorem for a mechanistic-interpretability primitive.
  • The eigenvalues are unique up to permutation and invariant under any change of dictionary basis.
  • A matching minimax lower bound (Theorem 7.1) confirms that no estimator can achieve a better rate in $M$.
  • A gap-free eigenvalue guarantee (Theorem 6.8) is derived using optimal matching (Wasserstein-$\infty$), removing the need for strict spectral separation for eigenvalue recovery, though separation remains necessary for eigenvector recovery.
• Robustness to Heavy Tails (Theorem 7.2): The authors extend the identifiability guarantee to heavy-tailed activations using a median-of-means estimator, showing the $M^{-1/2}$ rate holds under fourth-moment conditions.
• Modal-Principal Dissociation (Theorem 8.1): The paper proves a structural dissociation: if the realisation is non-normal (which empirical measurements confirm), the principal directions of the activations (variance-maximizing) cannot coincide with the Koopman modes (dynamics-carrying). This implies that the "identifiable object" (spectrum) and the "legible object" (principal components) are distinct.
• Algebraic Completeness of Intervention Calculus (Theorem 9.2): The spectral projectors of $A$ generate a commutative algebra that is complete for first-order interventions (activation patching, path patching, etc.), providing a unique closed-form representative for any intervention on the lifted dynamics.
• Structural Origin of SAE Variability (Theorem 9.5): The paper demonstrates that the run-to-run variability of SAEs is a structural consequence of their objective function, which fails to enforce K-invariance (DR2). An explicit invariance penalty is proposed as a remedy.
• Instance-Dependent Reduction (Theorem 10.1): A certified model reduction bound is derived that exploits the low effective rank of attention writes, offering a strictly sharper error certificate than classical Enns–Glover bounds in the transformer regime.

Experimental Results
The theory was evaluated on three pretrained transformers: GPT-2 small, Gemma-2-2B, and Qwen3-8B-Base.

• Convergence: The spectral distance between split-half estimates converges as $M$ increases. On Qwen3-8B-Base, the spectral dictionary achieved the predicted exponent of $-0.506 \pm 0.031$, matching the theoretical $M^{-1/2}$ rate.
• SAE Invariance Gap: SAE dictionaries exhibited significantly larger projection residuals (invariance violations) compared to spectral and random dictionaries, confirming Theorem 9.5. This gap was shown to be sensitive to feature selection rules (e.g., variance vs. frequency).
• Intervention Performance: When tested on the Indirect Object Identification (IOI) circuit, Koopman modes outperformed random directions but underperformed Principal Component Analysis (PCA) in ablating the circuit's behavior. The performance gap decayed by $4.1\times$ as the depth distance increased, consistent with the dissociation theorem.
• Universality Criterion: A spectral test for cross-model universality failed to distinguish between seed replicas of the same architecture, indicating that dictionary mismatch bias (not just sampling error) limits current universality claims.
• Heavy Tails: Contrary to expectations, the "lifting" process (random Fourier features) effectively removed heavy tails from the residual stream, rendering the robust median-of-means estimator unnecessary for the specific dictionaries tested (except in one specific layer of Qwen3).

Significance and Claims
The paper claims to place mechanistic interpretability on a rigorous identifiability footing.

1. Intrinsic Fingerprint: The Koopman spectrum is an intrinsic, model-intrinsic fingerprint that is recoverable with a stated error bar, distinguishing it from artifacts of the discovery method.
2. Limits of Legibility: The paper explicitly claims that identifiability does not imply legibility. Because transformer realizations are non-normal, the mathematically identifiable modes (Koopman spectrum) are structurally distinct from the variance-based directions (PCA) that humans often find interpretable.
3. Structural vs. Algorithmic: The variability observed in SAEs is not a tuning problem but a structural failure of the objective to enforce dynamical invariance.
4. Safety Implications: For safety arguments requiring the identification of a mechanism $Y$ responsible for behavior $X$, this work provides a certificate that $Y$ is a property of the model rather than the procedure. However, it cautions that this certificate does not guarantee behavioral coverage or human legibility.

The authors conclude that while the Koopman spectrum provides a mathematically sound, identifiable decomposition of transformer depth dynamics, it is not a direct substitute for human-legible circuit discovery, and the two objects (identifiable vs. legible) are fundamentally different in the non-normal regime occupied by these models.