Technical Summary: Transformers Converge to Invariant Algorithmic Cores

1. Problem Statement

The central obstacle to mechanistic interpretability in neural networks is underdetermination. While training constrains a model's behavior (the mapping from inputs to outputs), it does not constrain the internal realization of that behavior. Independently trained models often exhibit functional equivalence—performing the same task with vastly different weight configurations and internal circuitry. This phenomenon, observed in loss landscape geometry, model merging, and biological systems (e.g., degeneracy and system drift), poses a fundamental challenge: if mechanisms do not generalize across realizations, which explanations are "real"?

Current interpretability efforts often focus on specific implementation details (e.g., specific attention heads or sparse features), which may be idiosyncratic to a single training run. This paper argues that to achieve generalizable mechanistic understanding, research must shift focus from realization-specific circuitry to invariants—stable computational structures shared across independent training runs.

2. Methodology: Algorithmic Core Extraction (ACE)

The paper introduces Algorithmic Core Extraction (ACE), a framework designed to isolate low-dimensional subspaces that are both necessary and sufficient for a task and invariant across model realizations. The method operationalizes principles from linear system theory (specifically the Kalman decomposition and balanced truncation) for transformers.

The ACE pipeline consists of three stages:

1. Extract: For a given transformer layer with hidden dimension $D$ and $N$ inputs, the method computes the interaction between mean-centered activations ($H$) and the Jacobians of task-relevant outputs ($J$). By performing Singular Value Decomposition (SVD) on $HJ^\top$, ACE identifies directions that are jointly active and relevant. The leading singular vectors define a low-dimensional subspace (the "core") mapped back into activation space.
2. Validate: The extracted core is causally validated via ablation. A core is deemed sufficient if projecting activations onto it preserves task performance, and necessary if removing it (projecting onto the complement) reduces performance to chance levels.
3. Identify: The computational structure within the core is recovered by fitting linear operators (e.g., $z_{t+1} \approx Az_t$) to the core coordinates and inspecting their spectral properties (eigenvalues) to determine the underlying dynamics.

3. Key Results

A. Synthetic Markov Chain Tasks

In single-layer transformers trained on a four-state Markov chain:

• Divergent Parameters, Convergent Cores: Models trained with different random seeds achieved near-optimal accuracy but had near-zero cosine similarity in their weights.
• Invariant 3D Subspaces: ACE successfully isolated a 3-dimensional core in each model. These cores were necessary and sufficient for the task.
• Geometric vs. Statistical Alignment: While the cores resided in nearly orthogonal subspaces (projector overlap $\approx 0.02$), they exhibited near-perfect statistical alignment (Canonical Correlation Analysis $\approx 0.99$).
• Dynamics Recovery: Linear operators fitted within these cores recovered the exact non-trivial eigenvalues of the ground-truth Markov transition matrix, demonstrating that the models encode the same transition dynamics in different coordinate systems.

B. Modular Addition and Grokking

In two-layer transformers trained on modular addition ($a + b \equiv c \mod p$):

• Emergence at Grokking: Algorithmic cores crystallized (condensed into low-dimensional, necessary/sufficient subspaces) coincident with the "grokking" phenomenon (the delayed transition from memorization to generalization).
• Rotational Dynamics: Automated operator fitting revealed that at grokking, the core operators' eigenvalues snapped onto the unit circle, indicating the emergence of cyclic/rotational dynamics required for modular arithmetic.
• Core Inflation under Regularization: Under continued weight decay after grokking, cores "inflated" from ~15 to ~60 dimensions. This inflation was driven by redundancy: while the number of dimensions sufficient for performance remained stable, the number of dimensions necessary to prevent chance performance expanded. Weight decay acted as a redistribution force, spreading weights across all functionally equivalent modes (up to the theoretical maximum of $\lfloor p/2 \rfloor$).
• Inverse Scaling Law: The paper derives a theoretical link between functional redundancy, regularization, and grokking time. It posits that functional equivalence accelerates the transition to generalization, yielding an inverse scaling law for grokking time: $\tau_{grok} \propto 1/(\omega p)$, where $\omega$ is the weight decay and $p$ is the redundancy. Empirical sweeps of $\omega$ and $p$ validated this prediction.

C. Large Language Models (LLMs)

ACE was applied to six pretrained models spanning four families (GPT-2 Small/Medium/Large, LLaMA-3.1, Gemma-2, Qwen2.5) and over two orders of magnitude in scale (117M to 32B parameters) on the task of subject–verb number agreement:

• Universal 1D Core: A single, one-dimensional causal axis was identified in the late layers of all six models.
• Causal Control: This axis is necessary and sufficient for agreement. Projecting activations onto it predicts the singular-plural logit margin. Crucially, flipping the sign of this axis inverts grammatical number preferences throughout open-ended generation (e.g., changing "The key... are" to "The key... is").
• Cross-Architecture Alignment: Despite differences in architecture, tokenization, and training data, the coordinates of these 1D cores aligned significantly across models (Spearman correlations up to 0.93). This suggests a shared, invariant mechanism for grammatical number that is independent of implementation details.

4. Significance and Claims

The paper claims that beneath the apparent complexity of trained transformers lies a simpler, shared computational structure. The primary contributions are:

1. A Shift in Interpretability Paradigm: Moving from studying specific circuit realizations to studying invariants. The authors argue that mechanistic explanations should be tethered to stable, implementation-invariant quantities rather than idiosyncratic details.
2. Operationalization of Invariants: The ACE framework provides a method to isolate these invariants, causally validate them, and recover the algorithms they implement without presupposing their form.
3. Mechanistic Explanation of Grokking: The paper provides a theory linking functional equivalence, regularization, and grokking time, suggesting that redundancy allows weight decay to accelerate the transition to generalization by distributing weights across valid modes.
4. Evidence of Internal World Models: The recovery of ground-truth task structures (Markov spectra, cyclic operators) suggests that these cores encode internal representations of the data-generating process, aligning with the "internal model principle" from control theory.
5. Implications for Model Merging: The observation that functionally equivalent models embed cores in nearly orthogonal subspaces explains the difficulty of model merging (naive interpolation fails). It suggests that aligning algorithmic cores, rather than weights, may be a principled approach to model recombination.

The paper concludes that targeting invariants offers a more tractable path to mechanistic understanding and control, noting that while the dimensionality of cores for complex reasoning tasks remains an open question, the 1D agreement core's stability across massive scales suggests core dimensionality may not strictly depend on model size.