🤖 AI

Finding Usable Weight Mechanisms with Tiled SVD

This paper proposes a method for extracting interpretable weight mechanisms directly from neural network linear layers using column-tiled SVD to define features as trigger-write-strength triples, demonstrating perfect performance across all tested sites on a Gemma-2-2B model while releasing the associated code and evaluation suite.

Original authors: Ash Manvi, Samreena Tajreen

Published 2026-08-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ash Manvi, Samreena Tajreen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a giant, super-smart robot brain works. For a long time, scientists have been like detectives trying to figure out what the robot is thinking by looking at its "notes" (activations) while it solves problems. They built special dictionaries to label these notes, giving names like "cat" or "justice" to specific patterns of activity. It's a bit like having a translator that tells you, "Ah, right now the robot is thinking about a cat!" But there's a catch: this translator is a separate tool, not part of the robot's actual brain. It's like having a manual that explains the machine, rather than seeing the gears and levers moving inside.

The big question this paper tackles is: Can we find the actual "gears" inside the robot's brain that do the thinking, without needing a separate manual? The authors are looking at the "wires" (weights) that connect different parts of the brain. They want to find specific, usable switches that turn ideas on and off. If we can find these switches directly in the wiring, we won't just be guessing what the robot is thinking; we'll be able to see exactly how it builds its thoughts, step by step. This is crucial because if we understand the real mechanics, we might be able to fix mistakes or steer the robot in safer directions, rather than just guessing what it means.

The Paper's Story: Finding the Real Gears

The authors, Ash Manvi and Samreena Tajreen from Aquin Labs, decided to stop looking at the robot's "notes" and start looking directly at its "wiring diagrams." They focused on a specific type of robot brain called Gemma-2-2B and asked: "If we cut the wiring into small, manageable chunks, can we find the exact switches that control the robot's thoughts?"

To do this, they used a mathematical tool called SVD (Singular Value Decomposition). Think of SVD as a super-powerful way to take a messy, tangled ball of yarn and untangle it into neat, straight strings. Usually, scientists untangle the whole ball of yarn at once to find the biggest strings. But the authors had a hunch that the most useful strings might be hidden in smaller, specific sections. So, they tried a new trick: Tiled SVD.

Imagine the robot's wiring as a giant wall of switches. Instead of trying to untangle the whole wall at once, they chopped the wall into small, square tiles (like a mosaic). They untangled each little tile separately. From each tile, they pulled out a "mechanism mount," which is just a fancy name for a triple package:

The Trigger (v): The specific signal that flips the switch.
The Write (u): The direction the switch pushes the robot's thoughts.
The Strength (σ): How hard that switch pushes.

The authors didn't just guess if these switches worked; they built a rigorous test to see if they were real "mechanisms" or just random noise. They checked three main things:

Energy Lift: Does flipping this switch actually make the robot's brain do more work in a useful way? They found that their "tiled" approach (cutting the wall into small pieces) was much better at finding these powerful switches than looking at the whole wall at once.
Coverage: Do these switches cover enough of the robot's brain? They found that just a few switches per tile were enough to explain almost everything the robot was doing.
Causal Steering: If they manually flipped these switches, did the robot actually change its mind in a predictable way? They tested this by injecting signals into the robot's "residual stream" (the main highway where thoughts travel) and checking if the robot's final answers changed as expected.

What They Found

The results were surprisingly successful. They tested seven different types of wiring in every single layer of the robot's brain (26 layers total). That's 182 different spots they checked.

The "Big" Switches: For the two main types of switches that actually write new thoughts into the robot's memory (called mlp.down and attn.o), they used a full test that included checking if the robot changed its mind correctly. All 52 of these spots passed.
The "Helper" Switches: For the other five types of switches that help process information but don't write directly to memory, they used a slightly simpler test. All 130 of these spots passed.

In total, 182 out of 182 spots passed the test. This means they successfully found usable, real mechanisms in every single part of the robot's brain they looked at.

What This Means (and What It Doesn't)

The authors are very clear about what they didn't do. They didn't find a dictionary of words like "cat" or "love." They didn't replace the old "note-taking" methods (like sparse autoencoders) for discovering new concepts. Instead, they proved that if you look at the wiring in small, tiled chunks, you can find the actual physical rules the robot uses to think.

They also discovered that for some parts of the brain, you can't just look at the raw wiring; you have to look at the "effective path" (how the signal actually flows when the robot is working). Once they adjusted for this, the switches worked perfectly.

The paper suggests that this "tiled" approach is a better way to find the robot's internal gears than looking at the whole system at once. It's a reproducible, honest test that shows we can find the "weight rules" (the actual math inside the brain) that drive the robot's behavior, without needing a separate translator to tell us what's happening. While they only tested this on one specific robot model and one type of text, the method they built is now available for anyone to use to explore other brains.

Technical Summary: Finding Usable Weight Mechanisms with Tiled SVD

Problem Statement
Current dominant approaches in mechanistic interpretability rely on training proxy dictionaries, such as sparse autoencoders (SAEs), to label features based on max-activating text. While these atlases successfully identify concepts, the "identity" of a feature resides in the learned dictionary rather than the network weights themselves. Consequently, the specific weight rules that actually write into the residual stream remain implicit. Furthermore, existing methods often use Singular Value Decomposition (SVD) merely as a lens or circuit primitive, rather than as a rigorous test to determine which specific chunking of a weight matrix ( $W$ ) yields usable, on-distribution mechanisms.

Methodology
The authors propose extracting "mechanism mounts" directly from linear sites within the network using column-tiled SVD.

Mechanism Mounts: A mount is defined as a triple $(v, u, \sigma)$ representing a trigger, a write direction, and a strength. The identity of a mechanism is the weight rule itself, not a verbal label.
Tiling Strategy: Instead of performing SVD on the entire weight matrix, the input columns of $W$ are partitioned into tiles. Each tile is factored independently ( $B_t = U_t \Sigma_t V_t^\top$ ). The top- $k$ modes from each tile are retained to form the mounts.
Evaluation Metrics: The authors introduce a measurement stack judged on full-write energy lift rather than tile-local lift.
- Full-Write Energy Lift ( $L_{full}$ ): Measures how well a direction $u$ explains the energy of the actual write tensor $\Delta h_t$ across the dataset. This metric avoids the tautological advantage of one-column tiles that inflate local metrics.
- Coverage Saturation: Evaluates the fraction of the Frobenius norm of $W$ recovered by per-tile rank- $k$ reconstructions. It requires the lift to saturate early (within 1–2 modes) and not collapse significantly.
- Causal Steer Check (Experiment C): For residual writes, the authors inject the steer direction after post-sublayer RMSNorm and measure the Spearman correlation ( $\rho$ ) between the resulting logit changes and the unembedding of the write direction.
Effective-Path Mounts: For non-residual sites like mlp.up and attn.v, raw module weights often fail causal checks. The method constructs "effective maps" (e.g., weighting mlp.up by mean gate activations or using ridge least-squares for attn.v) to capture the actual on-distribution usage.

Experimental Setup

Model: Google Gemma-2-2B (26 layers).
Data: WikiText-2 raw training data, subsampled to 16,384 tokens.
Sites: All seven linear maps per layer are evaluated:
- Residual Writes (A/B/C checks): mlp.down, attn.o.
- Non-Residual (A/B checks only): mlp.gate, mlp.up, attn.q, attn.k, attn.v.
Baselines: The tiled SVD approach is compared against whole-matrix SVD, column sampling, and random baselines.

Key Results
The proposed suite was applied to all 182 site-layers (7 sites $\times$ 26 layers).

Experiment A (Chunking): Tiled SVD consistently achieved higher full-write energy lift than whole-matrix SVD, column sampling, and random baselines across all depths for residual writes. This indicates that local column structure in $W$ is not well summarized by a single global SVD.
Experiment B (Coverage): Coverage lift saturated rapidly, reaching peak performance with only 1 or 2 modes per tile. Additional modes added mostly redundancy.
Experiment C (Causal Depth): For residual writes, the alignment between steered logits and the unembedding of the write direction increased with depth. Early layers (0–5) showed weak alignment (justifying a waiver for the C1 check), while final layers reached high correlations ( $\rho \approx 0.91$ for mlp.down, $\rho \approx 0.75$ for attn.o).
Aggregate Verdict: The suite passed 182/182 site-layers.
- Residual sites (mlp.down, attn.o) passed A, B, and C checks (52/52).
- Non-residual sites passed A and B checks (130/130), provided mlp.up and attn.v used effective-path mounts.

Significance and Claims
The authors position this work not as a replacement for sparse autoencoders in concept discovery, nor as a method for generating human-readable concept names. Instead, the significance lies in:

Fair Chunking: Demonstrating that tiled SVD is a superior method for extracting usable weight mechanisms compared to global SVD or column sampling under a matched budget.
Negative Result on Local Metrics: Highlighting that tile-local lift metrics are tautological and that full-write energy lift is a necessary criterion for usability.
Reproducible Protocol: Providing a "honest multi-site protocol" that validates mechanism extraction across all linear maps in a transformer block, including a depth-conditioned causal check.
Identity Definition: Establishing that the "identity" of a mechanism is the weight rule itself, independent of external dictionaries.

The paper concludes that with effective-path mounts for specific layers, a simple, reproducible suite of checks can validate usable mechanisms across an entire model architecture. The authors release the library code, corpus builder, and experiment entrypoints to support this protocol.

The Paper's Story: Finding the Real Gears

What They Found

What This Means (and What It Doesn't)

Technical Summary: Finding Usable Weight Mechanisms with Tiled SVD

More like this