🤖 machine learning

A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation

This paper establishes an idealized theoretical framework proving that activation patching and weight-space ablation yield divergent causal interpretations of model components, deriving exact conditions for their agreement and disagreement while validating these predictions through synthetic experiments and real-world transformer models.

Original authors: Abdallah Khemais

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Abdallah Khemais

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Dilemma: How We Try to Understand AI Brains

Imagine you have a giant, complex machine that can tell stories, solve math problems, or write poems. You want to know how it works. Is it the gears in the middle? The springs at the top? To find out, scientists use two main tricks. The first is like Activation Patching: imagine the machine is running a story, and you pause it, swap out a specific gear with a gear from a different story, and see if the plot changes. If the story changes, that gear was important. The second trick is Weight-Space Ablation: instead of pausing the machine, you take a screwdriver and permanently remove or "zero out" that gear. Then you run the machine again. If the story breaks, that gear was important.

For a long time, scientists hoped these two tricks would tell the same story. They assumed that if a gear mattered for a specific moment (patching), it would also matter for the machine's overall design (ablation). But in the world of Artificial Intelligence, things are rarely that simple. Sometimes, a machine is so full of backup gears that removing one doesn't break anything, even if swapping it in the middle of a story causes a disaster. This paper asks a crucial question: When do these two detective methods agree, and when do they lie to us? The author doesn't just guess; they build a mathematical model to prove exactly why these methods can disagree, and they test it on tiny, made-up AI brains to see if the math holds up in the real world.

The Paper: Why Two Detective Methods Can Disagree

This paper, titled A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations, is a deep dive into the mechanics of how AI models make decisions. The author, Abdallah Khemais, treats the AI not as a mysterious black box, but as a set of mathematical ingredients being mixed together. They focus on a specific type of AI behavior called "conditional computation"—where the AI decides to do one thing if it sees a "red" token and another thing if it sees a "blue" token.

The Three Big Discoveries

The paper proves three main things using a simplified mathematical model, and then checks if those things happen in actual, trained AI models.

1. The "Perfect Collapse" (When Removing Parts Makes the AI Forget)
Imagine the AI is deciding between two paths: Path A (the "Inverted" route) and Path B (the "Literal" route). The author proves that if you remove a specific set of "carriers" (the parts of the AI that carry the decision), the AI might suddenly forget the difference between Path A and Path B. It collapses into a single, unconditional answer.

The Catch: This only happens if the removal is perfectly balanced. If you remove a part that pushes the AI toward Path A, you must also remove a part that pushes it toward Path B with the exact same strength. If the balance is off, the AI doesn't collapse; it just gets confused.
The Real-World Test: The author trained tiny AI models on a game where they had to remember keys and values. They found that sometimes, removing one part made the AI choose the wrong path, but removing a different combination of parts made it choose the opposite wrong path. This "polarity reversal" proved that the AI wasn't just failing randomly; it was collapsing in a very specific, predictable way that the math predicted.

2. The "Patch vs. Ablate" Mismatch (The Redundancy Trap)
This is the paper's most playful and surprising finding. The author shows that Activation Patching and Weight Ablation measure two completely different things.

The Analogy: Imagine a team of five people carrying a heavy box. If you ask, "Who is strong enough to carry the box alone?" and you swap one person with a super-strong giant (Patching), the box flies across the room. That person is "sufficient." But if you ask, "Who is necessary?" and you quietly remove one person from the team (Ablation), the other four might just pick up the slack, and the box doesn't drop.
The Finding: The paper proves mathematically that you can have a situation where every single person is strong enough to carry the box on their own (so swapping any one of them changes the outcome), but no single person is necessary (so removing any one of them does nothing).
The Result: In their experiments, they found AI components that, when swapped, completely flipped the AI's decision (a "recovery ratio" greater than 1, meaning it overshot the goal). Yet, when they removed those same components, the AI kept working perfectly fine. This explains why some AI parts look like heroes in one test and like useless decorations in another.

3. The "Hidden Interaction" (When Parts Talk to Each Other)
The simplified math model assumes that every part of the AI works independently, like separate ingredients in a salad. But in real AI models, parts often talk to each other. Specifically, the author looked at how an "Attention Head" (a part that looks at other words) interacts with the "MLP" (a part that processes the result) in the same layer.

The Math: They derived a precise formula showing that if you remove the Attention Head, it changes the input for the MLP, creating a "ripple effect" or "interaction term" that the simple model ignores.
The Proof: They showed that if you remove only the MLP, this ripple effect vanishes (the math is exact). But if you remove the Attention Head, the ripple effect is real and measurable.
The Experiment: They measured this "interaction magnitude" in their AI models. They found a strong, negative correlation (a Spearman rank of -0.83): the bigger the hidden interaction, the less accurate the simple model's predictions were.

The "Oops" Moment: No Magic Threshold

Here is where the paper gets very honest and scientific. In the beginning, the author thought they might find a "magic number" or a clear line that separates when the simple model works and when it fails. They looked at 14 specific setups and saw a clean gap: some had zero interaction, others had high interaction, and the model worked perfectly for the zero ones.

But then, they tested 25 more setups (out-of-sample). The clean gap disappeared. The setups that looked like they should fail sometimes worked, and vice versa.

The Conclusion: The author admits that while the size of the interaction is a great predictor of how wrong the simple model will be, there is no single "cutoff number" you can use to say "This is safe, that is not." The relationship is a smooth curve, not a switch. They explicitly state that the "clean separation" they first saw was likely a fluke of the small sample size, and the real world is messier.

Why This Matters

This paper doesn't just say "AI is hard." It gives us a precise map of why our tools for understanding AI can be misleading.

It tells us that if an AI part looks "dispensable" when we remove it, it might just be because the AI is redundant, not because the part is unimportant.
It tells us that if we want to understand AI, we can't just rely on one method (like patching or ablation); we have to understand the math of how they differ.
Most importantly, it shows that while we can predict how much a model will fail when we simplify it, we can't yet predict exactly when it will fail with a simple yes/no rule.

The author concludes that the "idealized model" (the simple math) is a powerful tool, but it has a known error term. By measuring that error term, we can get a much clearer picture of what the AI is actually doing, rather than being fooled by the redundancy of its own design. It's a reminder that in the quest to understand AI, the truth is often a gradient, not a switch.

=== SUMMARY ===
This paper investigates the conditions under which two dominant families of mechanistic interpretability interventions—activation patching and weight-space ablation—agree or diverge. While both methods are used to argue that a specific network component is causally responsible for a behavior, they operate on different objects: patching alters a single realized computation (activations), whereas ablation alters the parameters (weights) that generate all computations. The paper demonstrates that these interventions measure fundamentally different quantities and can lead to contradictory conclusions about a component's importance, even within an idealized mathematical model.

Problem Formulation

The author studies a conditional computation carried additively through a residual stream, modeled as:
$F(x) = F_0(x) + \sum_{i=1}^k \alpha_i(x) v_i$
where $F_0$ is the unconditional part, $v_i$ are fixed feature directions (carriers), and $\alpha_i(x)$ are scalar selector coefficients. The network's decision is determined by a linear readout $s(x) = \psi(F(x)) + b$ . The study focuses on matched pairs of inputs $(x_A, x_B)$ that require the same output but are processed through opposite conditional branches (i.e., $s(x_A) > 0 > s(x_B)$ ).

The core question is: When does deleting a subset of carriers (ablation) force both inputs to the same unconditional output (collapse), and how does this relate to the effect of patching a single carrier?

Methodology

The paper employs a theoretical analysis of an Abstract Conditional Model followed by synthetic validation on small transformers trained on conditional tasks.

Theoretical Derivation: The author derives exact identities for the effects of ablation and patching. They define "carriers" as the pairs $(v_i, \alpha_i)$ and analyze the residual stream after removing a subset of these terms.
Idealized Assumptions: The theory assumes residual additivity, low-rank support for conditional computations, stable base computation, and a linear readout.
Synthetic Validation: Small transformers (4-layer, width 64) are trained on a synthetic associative recall task involving a "marker" token that switches between two formats (direct lookup vs. permuted lookup). The author performs greedy activation patching to identify "carriers" (attention heads or MLPs) and then performs low-rank weight-space ablations by projecting donor-receiver activation deltas out of the weight matrices.
Interaction Analysis: The paper derives a first-order formula for the interaction term that arises when an attention head and its layer's MLP are composed, showing how the idealized additive model fails in this specific architectural configuration.

Key Contributions and Results

1. Conditional Collapse (Theorem 1)
The paper proves that deleting a subset of carriers $S$ collapses a matched pair onto a single unconditional output if and only if two exact conditions are met:

Symmetry: The removed mass is symmetric on the pair ( $\bar{q}_S = 0$ ).
No Surviving Contrast: No contrast remains outside the removed subset ( $\Psi_S = 0$ ).
When these hold, the resulting error is deterministic, and the polarity of the collapse is determined by the sign of the pair's mean margin. The author provides an exact error identity (Proposition 1) showing that if these conditions hold only approximately, the error is bounded by the deviation from these equalities, allowing the theory to apply to real networks where exact equality is impossible.

2. Patching–Ablation Dissociation (Theorem 2)
The author proves that patching and ablation measure distinct quantities:

Patching moves the readout by the carrier's contrast ( $\beta_i \delta_i$ , where $\delta_i$ is the difference in activation between donor and receiver).
Ablation moves the readout by the carrier's absolute level at the receiver ( $-\beta_i \alpha_i(x_B)$ ).
The author constructs matched pairs where every single-carrier patch flips the decision (sufficiency), while no single-carrier ablation does (lack of necessity). This "redundancy regime" explains why patching can overstate importance (overshooting the decision boundary) while ablation can understate it (components appearing dispensable due to redundancy).

3. Nonlinear Interaction (Theorem 3)
The idealized model assumes carriers are independent. The author derives an exact formula for the interaction term $\Delta(x)$ that arises when an attention head is ablated but its layer's MLP is not.

The interaction is second-order in the size of the perturbation.
It vanishes identically if only the MLP is ablated ( $\eta(x) \equiv 0$ ).
It is generically non-zero if the head is ablated, meaning the head's ablation perturbs the MLP's output even if the MLP is untouched.
This interaction term explains why the idealized collapse predictions may fail in real networks when heads are involved.

4. Synthetic Validation
Experiments on five trained transformer instances confirm the theoretical predictions:

Collapse: Nested ablation subsets can collapse the network onto opposite unconditional branches (e.g., one subset forces an "inverted" reading, a larger subset forces a "literal" reading), a phenomenon impossible in the idealized model but explained by the interaction term.
Dissociation: A single-site patch showed a recovery ratio $\kappa > 1$ (overshoot), while the corresponding weight ablation left the majority of inputs correct, confirming the sufficiency-without-necessity regime.
Interaction vs. Fidelity: Across 39 ablation configurations, the magnitude of the interaction term (the gap between the idealized model and the true edited network) is strongly monotonically correlated (Spearman $\rho \approx -0.83$ ) with the model's prediction accuracy.
Negative Result: A "clean separation" (a threshold that perfectly distinguished idealized from non-idealized configurations) observed in a subset of 14 configurations did not survive when tested on 25 additional configurations. The author concludes that while the interaction magnitude is the correct quantity to track, a universal numerical threshold for decision-making is not established by these experiments.

Significance and Claims

The paper claims to establish that activation patching and weight-space ablation are not two measurements of the same causal quantity. Instead, they are operators on different objects that interrogate different mathematical properties (contrast vs. absolute level).

Interpretability Implications: A negative ablation result does not prove a component is unimportant, nor does a positive patching result prove it is necessary. These discrepancies are structural features of redundant, superposed codes, not merely measurement noise.
Theoretical Precision: The paper provides exact identities for the errors introduced by the idealized additive model, specifically bounding the interaction term for head-MLP compositions. It shows that the failure of the idealized model is not random noise but a predictable, second-order effect.
Modesty on Thresholds: The author explicitly states that while the theory identifies the right quantity (interaction magnitude) to predict model behavior, it does not yet provide a calibrated threshold that transfers across networks or configurations. The significance lies in the qualitative understanding of when and why the interventions diverge, rather than in a new binary decision rule for interpretability.

The work is framed as a "single-block theory" with synthetic validation, with extensions to multi-layer interactions and real pretrained models reserved for a companion analysis.