🤖 machine learning

Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure

This paper introduces "causal dimensionality" (kappa) as a model-intrinsic metric quantifying the effective rank of a transformer layer's causal influence, revealing that while representational capacity scales linearly with sparse autoencoder width, causal capacity saturates at a sub-linear level that remains invariant to model size and exhibits distinct structural patterns across network depth.

Original authors: Nilesh Sarkar, Dawar Jyoti Deka

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nilesh Sarkar, Dawar Jyoti Deka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, complex factory (the AI model) that takes in raw materials (text) and produces a finished product (an answer). Inside this factory, there are thousands of workers (neurons) passing notes to each other. For a long time, researchers have tried to understand this factory by giving each worker a name tag, hoping to see who does what.

This paper introduces a new way to count how many workers are actually essential to the factory's output, revealing a surprising gap between how many workers exist and how many matter.

Here is the breakdown of their findings using simple analogies:

1. The "Name Tag" Problem (SAEs)

Researchers use a tool called a Sparse Autoencoder (SAE). Think of this as a giant dictionary that tries to give a unique name tag to every distinct "thought" or "feature" the AI has.

The Old Assumption: If you make the dictionary bigger (add more name tags), you assume you are finding more important things that actually change the AI's answer.
The Reality: The authors tested this by making the dictionary 64 times bigger. They found that while the number of name tags (features that light up) exploded by 15.6 times, the number of actually important features (those that, if removed, would change the AI's answer) only grew by 4.35 times.

The Analogy: Imagine a library. If you double the size of the library, you double the number of books (representational capacity). But the authors found that even if you make the library 64 times bigger, the number of books that actually contain the "secret code" to solve a specific puzzle only grows a little bit. Most of the new books are just variations of the same old stories.

2. The "Causal Wedge"

The authors call this gap the Representational-Causal Wedge.

Representational: How many different "ideas" the AI can hold in its head.
Causal: How many of those ideas actually drive the final decision.
The Finding: The AI can hold a massive number of ideas, but only a tiny fraction of them are the "bosses" that actually pull the levers to make the final decision. The rest are just "background noise" or redundant copies.

3. The "Magic Number" (Causal Dimensionality)

The paper defines a specific number, $\kappa$ (kappa), which represents the true number of "boss" ideas in a specific layer of the AI.

The Discovery: They calculated this number for a 2-billion-parameter AI and found it to be around 1,990.
The Surprise: This number is intrinsic. It doesn't matter if you make the AI 3.5 times bigger (from 2B to 9B parameters). The number of "boss" ideas stays exactly the same.
The Analogy: Imagine a 2-person band and a 100-person orchestra playing the same song. The 100-person orchestra has more instruments (parameters), but they are just playing the same 3 core melodies louder and with more harmony. The number of unique melodies (causal dimensionality) hasn't changed; they just have more musicians playing them.

4. The "Deepening" Effect

As you go deeper into the AI (from the first layer to the last), the "volume" of the signal drops.

The Finding: The signal required to identify an important feature gets 20 times weaker as you go deeper into the network.
The Analogy: Imagine a game of "Telephone." The first person speaks loudly. By the time the message reaches the last person, it's a whisper. If you use a microphone set to "loud" to listen at the end, you'll think no one is speaking. You have to turn the sensitivity way up to hear the important parts in the deep layers.

5. The "Filter" vs. The "Selector" (A Major Twist)

The paper tested what happens if you swap out the AI's "encoder" (the part that decides which features to activate) with a random, messy one.

The Result: When they used a random encoder, the number of "important" features exploded by 9 times.
The Conclusion: The trained encoder isn't a "selector" picking the best features. It is a filter (or a suppressor). Its main job is to turn off the background noise.
The Analogy: Think of the encoder as a bouncer at a club. You might think the bouncer is there to pick the "cool" people (features) to let in. But this paper shows the bouncer is actually there to keep the crowd out. If you remove the bouncer (use a random one), the club gets chaotic and overcrowded with people who don't belong. The "cool" people were always there; the bouncer just made sure the noise didn't drown them out.

Summary of What This Means

More size $\neq$ More complexity: Making AI models bigger doesn't necessarily give them more "types" of thinking; it just gives them more ways to say the same things.
The "Boss" count is fixed: There is a hard limit to how many independent "boss" concepts a layer can handle, regardless of how big the model is.
Interpretability needs a new focus: If you want to understand how an AI thinks, don't just look for the "loudest" features. You need to look at how the system filters out the noise, because that's where the real structure lies.

The paper concludes that we have a new, measurable way to count the true "thinking capacity" of an AI layer, and it turns out that capacity is much smaller and more stable than we previously thought.

Technical Summary: Causal Dimensionality of Transformer Representations

Problem Statement

Sparse Autoencoders (SAEs) have become the standard tool for decomposing transformer residual streams into interpretable, monosemantic feature dictionaries. A prevailing assumption in the field is that representational capacity (the number of features that fire on any input) scales linearly with causal capacity (the number of features whose perturbation actually changes model behavior). If true, simply increasing SAE width would proportionally expand the set of causally relevant features available for mechanistic interpretability.

This paper challenges that assumption, arguing that representational and causal capacities diverge significantly as SAEs scale. The authors posit that the field lacks a systematic characterization of the relationship between SAE width and causal influence, necessitating a new metric to quantify the intrinsic causal structure of transformer layers.

Methodology

1. Defining Causal Dimensionality ( $\kappa$ )

The authors introduce causal dimensionality, denoted as $\kappa(L, M, T)$ , defined as the effective rank of the expected Jacobian outer product at layer $L$ :
$\kappa(L, M, T) = \text{eff-rank}(\Sigma_L), \quad \text{where } \Sigma_L = \mathbb{E}_{x \sim T} [J_L(x)^\top J_L(x)]$
Here, $J_L(x)$ is the Jacobian of the model output with respect to the residual stream at layer $L$ . $\kappa$ represents the effective number of directions in the residual stream that causally influence the output. The authors prove that $\kappa \le \min(d, |V|)$ , where $d$ is the model dimension and $|V|$ is the vocabulary size.

2. Estimation via SAE Width Sweep and Attribution Patching

Since computing $\Sigma_L$ exactly is intractable (requiring $d$ backward passes per input), the authors propose a consistent estimator using Attribution Patching (AtP) paired with an SAE width sweep:

Attribution Patching: Used as a first-order proxy for causal influence. The score for a feature $i$ is $AtP_i = \mathbb{E} [|\frac{\partial \mathcal{L}}{\partial f_i} \cdot f_i(x)|]$ , measuring the impact of the feature on the loss.
Width Sweep: The authors train SAEs on Gemma-2-2B (Layer 12) across seven widths ranging from 16,384 to 1,048,576 features.
Metric: They count the number of features $N_{causal}(m)$ that exceed a calibrated absolute threshold $\epsilon$ (set at the 98th percentile of AtP scores at the reference width).
Consistency: Under assumptions of spanning, faithfulness, and no dead features, $N_{causal}(m)$ converges to a spectral count that lower-bounds $\kappa$ .

3. Experimental Controls

The study employs rigorous controls to isolate the causal signal:

Matched- $\ell_0$ : Controlling for the average number of active features across widths to prevent confounding by sparsity drift.
Scale Invariance: Comparing Gemma-2-2B and Gemma-2-9B at matched relative depths.
Architecture Invariance: Testing TopK, JumpReLU, and standard ReLU SAEs.
Ablations: A four-cell encoder/decoder ablation to distinguish between feature selection and sparsity filtering.
Synthetic Ground Truth: Testing recovery of known causal directions in a toy model.

Key Results

1. The Representational–Causal Wedge

On Gemma-2-2B Layer 12, increasing SAE width by 64× (16k to 1M) resulted in:

Representational Growth: The count of firing features ( $N_{repr}$ ) increased by 15.6×.
Causal Growth: The count of causally active features ( $N_{causal}$ ) increased by only 4.35×.
This divergence, termed the representational–causal wedge, demonstrates that representational capacity grows linearly while causal capacity saturates sub-linearly.

2. Estimation of $\kappa$

Fitting a saturating curve to the $N_{causal}(m)$ trajectory yields an estimated causal dimensionality:

$\hat{\kappa} \approx 1,990$ (with a wide confidence interval due to hardware limits on SAE width, but a robust ratio).
$\hat{\kappa} / d_{model} \approx 0.86$ : The model utilizes nearly all its representational dimensions causally on generic text, but this number is bounded and does not grow with SAE width.
Participation Ratio: A lower bound $\kappa_{PR} \approx 280$ indicates a heavy-tailed spectrum of causal influence.

3. Invariance and Structure

Scale Invariance: Despite a 3.46× increase in parameters (Gemma-2-2B vs. 9B), the causal count at matched relative depth remained identical ( $N_{causal} = 328$ ). This suggests $\kappa$ is determined by task diversity, not parameter count.
Depth Attenuation: Across eight layers, the absolute attribution threshold required to identify the top 2% of features dropped 20× from Layer 1 to Layer 23, indicating that causal signal strength attenuates with depth.
Architecture Invariance: The sub-linear growth of $N_{causal}$ held across TopK, JumpReLU, and standard ReLU architectures.

4. Mechanistic Insight: The Encoder as a Filter

A critical finding from the four-cell ablation is that replacing the trained encoder with a random orthonormal projection inflated $N_{causal}$ by 9.27×.

Implication: The trained encoder acts primarily as a sparsity filter (suppressing background features) rather than a feature selector.
Consequence: Interpretability work focusing on encoder feature labels may be misdirected; the decoder structure and routing of the suppressed features are likely more critical.

5. Validation and Controls

Geometric Privilege: Causal features showed no geometric distinction (magnitude or cosine similarity) from non-causal features in decoder space.
Synthetic Recovery: AtP failed to recover ground-truth causal directions in a toy model (Recall = 0), confirming that AtP measures activation patterns routed through the decoder, not a direct recovery of a causal basis.
Task-Specific Recovery: On specific task circuits (Marks et al., 2025), the top 5% of AtP features recovered 98.2% of the ground truth, supporting a "layered structure" where a high-AtP core is shared across tasks, surrounded by a task-specific halo.

Significance and Claims

The paper establishes $\kappa$ as a measurable, model-intrinsic property of transformer layers. Its primary contributions are:

Theoretical Definition: Defining causal dimensionality as the effective rank of the causal Jacobian and proving its upper bounds.
Empirical Characterization: Demonstrating that causal capacity is sub-linearly recoverable by SAE width and invariant to model scaling, contradicting the assumption that wider SAEs linearly expand causal interpretability.
Mechanistic Reframing: Identifying the trained encoder as a sparsity filter, suggesting that the "causal set" is defined by the encoder's suppression of noise rather than the selection of specific concepts.
Practical Implication: Current SAE widths (e.g., 1M) capture only ~72% of the estimated causal dimension ( $\hat{\kappa}$ ), implying that for a complete causal enumeration, significantly wider SAEs (approx. 2.6M) would be required, which is currently computationally prohibitive.

The authors conclude that the "wedge" is a fundamental property of transformer representations, driven by the saturation of causal subspace dimensions, and that interpretability efforts must account for this non-linear scaling and the specific role of the encoder in filtering background features.