💻 computer science

Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers

This paper introduces the multi-level IMPACT framework to demonstrate that while structural sparsity in Vision Transformers reduces circuit size, it does not systematically improve semantic interpretability, as pruning merely redistributes computation rather than isolating simpler functional modules.

Original authors: Siyu Zhang

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Does Making a Brain "Leaner" Make it Easier to Understand?

Imagine you have a giant, chaotic library (a Vision Transformer, or AI model that looks at pictures). This library has millions of books, and every book is connected to every other book by a tangled web of red strings. When the AI tries to identify a picture of a "cat," it pulls on a massive, complex web of strings to find the answer.

The Hypothesis:
Researchers thought: "If we cut away most of those red strings (make the model sparse), the library will be smaller, the web will be simpler, and it will be much easier to see exactly how the AI finds the 'cat'."

It's like taking a messy, cluttered workshop and throwing away 70% of the tools. You'd expect the remaining tools to be the "best" ones, making the work easier to understand.

The Reality Check:
This paper says: "Not so fast."

While cutting the strings did make the physical web smaller, it didn't make the logic inside any clearer. The AI didn't become a "simpler" thinker; it just became a "sparser" one.

The Experiment: The "IMPACT" Framework

To test this, the researchers built a multi-level testing kit called IMPACT. Think of it as a set of four different flashlights, each shining on a different part of the AI's brain to see if it got "cleaner" after the pruning.

1. The Neuron Level (The Individual Workers)

The Metaphor: Imagine the AI is a factory with thousands of workers (neurons). In a "dense" factory, every worker talks to everyone else. In a "sparse" factory, we fired 70% of the connections.
The Question: Do the remaining workers now specialize? (e.g., Does one worker only think about "ears" and another only think about "tails"?)
The Result: No. The workers are still generalists. They are still juggling multiple concepts at once (polysemanticity). Cutting the connections didn't force them to specialize; it just made them work harder with fewer handshakes.

2. The Layer Level (The Team Huddles)

The Metaphor: The AI processes images in layers, like a relay race. The researchers used a tool (Sparse Autoencoders) to try to untangle the messages being passed between runners.
The Question: Did the messages become clearer and less mixed up?
The Result: No. The messages were still a jumbled mix. The "sparse" team didn't communicate more clearly than the "dense" team.

3. The Circuit Level (The Blueprint)

The Metaphor: This is where the researchers found the only real change. They traced the exact path the AI took to solve a problem.
The Result: Yes, but with a catch. The "sparse" model used a much shorter path (fewer red strings). It was about 2.5 times smaller.
- However, because the factory was so small to begin with, the AI had to use a higher percentage of its remaining workers to do the job. It's like a small kitchen where the chef has to use 90% of the available knives to chop a carrot, whereas in a big kitchen, they only used 30% of the knives. The path is shorter, but the workload distribution is just as messy.

4. The Model Level (The Final Explanation)

The Metaphor: When the AI says "That's a cat," it highlights the part of the picture that matters (the cat's face). We want to know if the highlight is accurate.
The Result: Mixed. At low levels of pruning, the highlights got slightly better. But at high levels (70% cut), the highlights didn't get much better. The AI still sometimes pointed to the background or the wrong part of the cat, just like the dense model did.

The Big Conclusion: "Sparse but Not Simpler"

The paper's title sums it up perfectly: Sparse but not Simpler.

What happened: The AI became physically smaller and used fewer connections.
What didn't happen: The AI didn't become more logical, more specialized, or easier for humans to understand.

The Takeaway:
Simply cutting out connections (pruning) is like taking a messy room and throwing away 70% of the furniture. The room is less cluttered, but the way the person lives in the room hasn't changed. They are still sleeping in a pile of clothes and cooking in the bathroom; they just have fewer clothes and less kitchenware to do it with.

To make AI truly understandable, we can't just rely on cutting things out. We need to teach the AI to organize its thoughts differently from the start, rather than just hoping that removing connections will magically reveal the "truth" inside.

1. Problem Statement

The paper addresses a central hypothesis in mechanistic interpretability: Does structural sparsity (fewer active weights) lead to improved semantic interpretability?

Context: While sparse neural networks are often assumed to be more interpretable because they theoretically produce "compact circuits" (similar to findings in language models), it is unclear if this holds true for Vision Transformers (ViTs).
The Gap: Existing evidence relies heavily on structural metrics (e.g., circuit size). However, a smaller circuit does not guarantee semantic clarity. Neurons may remain "polysemantic" (responding to multiple concepts), features may not align with human concepts, and attribution maps may lack faithfulness.
Objective: To systematically evaluate whether weight sparsity in ViTs actually yields more interpretable models across multiple levels of analysis, or if it merely redistributes computation without simplifying the underlying logic.

2. Methodology

The authors introduce IMPACT (Interpretability Multi-level Pipeline for Assessing Computational Transparency), a framework that evaluates interpretability across four complementary levels.

A. Experimental Setup

Base Model: DeiT-III B/16 (a strong supervised Vision Transformer).
Sparsification: Models are pruned using Wanda (a weight-pruning method based on magnitude $\times$ input activation norm) and compared against magnitude pruning.
Sparsity Levels: Models are pruned to 10%, 30%, 50%, and 70% sparsity, followed by fine-tuning on ImageNet-1K to recover performance. The 70% sparsity level is selected as the primary operating point as it retains >96% of the dense model's accuracy.

B. The IMPACT Framework (Four Levels of Analysis)

Neuron Level: Evaluates individual unit behavior using four metrics:
- Ablation Impact: How much the model's output changes when top neurons are removed.
- Selectivity: How well a neuron correlates with a specific class (AUROC).
- Class Variance: Consistency of response within a class.
- Label Entropy: Monosemanticity (low entropy implies the neuron responds to only one class).
Layer Level: Uses BatchTopK Sparse Autoencoders (SAEs) to decompose dense residual stream activations into sparse latent features.
- Metrics: Reconstruction quality (NMSE, dead latents) and feature interpretability (applying the same neuron-level metrics to SAE features).
Circuit Level: Extracts minimal functional circuits responsible for single-category predictions (100 ImageNet classes).
- Method: Learnable node masking via sigmoid straight-through estimators to identify necessary nodes.
- Metric: Normalized Circuit Quality (NCQ), which balances fidelity (accuracy retained) against compactness (fraction of nodes removed).
Model Level: Evaluates the faithfulness of input attribution maps (explanations of why the model made a prediction).
- Method: Transformer Attribution (gradient-weighted attention + LRP).
- Metrics: Insertion and Deletion AUC (measuring how quickly model confidence changes as relevant pixels are added/removed).

3. Key Contributions

Systematic Comparison: The first extensive comparison of interpretability between dense and sparse Vision Transformers, extending circuit analysis from language models to the vision domain.
IMPACT Framework: A novel multi-level evaluation pipeline that moves beyond simple circuit size to assess neuron selectivity, feature disentanglement, and attribution faithfulness.
Empirical Evidence: Demonstrates that while sparsity reduces structural complexity (edge count), it fails to improve semantic interpretability, challenging the assumption that "sparse = simpler."

4. Key Results

A. Circuit Level: Structural Reduction vs. Functional Spreading

Edge Reduction: Sparse models produce significantly smaller circuits. At 70% sparsity, circuits have ~2.5× fewer edges than dense models (12.94% vs. 32.42% of total edges).
Node Usage: Despite fewer edges, sparse circuits activate a larger fraction of available nodes (41.60% for 70% sparse vs. 29.03% for dense).
Conclusion: Pruning does not isolate computation into fewer functional modules. Instead, it causes "representation spreading," where the model distributes computation across more surviving components to compensate for reduced connectivity.

B. Neuron & Layer Levels: No Gain in Selectivity

Polysemanticity Persists: Sparse models show no systematic improvement in neuron selectivity, label entropy, or class variance compared to dense models.
SAE Features: Features extracted via SAEs from sparse models are not more disentangled or monosemantic than those from dense models.
Implication: Weight sparsity alone does not reduce feature superposition or multiplexing.

C. Model Level: Limited Faithfulness Gains

Attribution: Light pruning (10–30%) shows modest improvements in attribution faithfulness (Insertion/Deletion metrics), likely due to a regularization effect.
High Sparsity: At 70% sparsity, faithfulness gains plateau or decline. Qualitative visualizations show no consistent improvement in spatial localization or semantic grounding.

5. Significance and Conclusion

Main Finding: Structural sparsity does not guarantee semantic interpretability. While sparse models are "simpler" in terms of parameter count and edge density, they are not "simpler" in terms of how they represent concepts or explain their decisions.
Mechanism: Pruning redistributes computation rather than isolating it. The model compensates for lost connections by engaging more nodes, maintaining a high degree of polysemanticity and entanglement.
Implications for Future Work:
- Relying solely on post-hoc pruning (like Wanda) is insufficient for creating interpretable models.
- Achieving true mechanistic transparency likely requires training objectives or architectural priors that explicitly encourage semantic structure and disentanglement, rather than just sparsity.
- Evaluation frameworks must look beyond circuit compactness (size) to assess the actual semantic quality of the representations.

In summary, the paper concludes that "Sparse but not Simpler" is an accurate description of pruned Vision Transformers: they are structurally smaller but semantically just as complex and difficult to interpret as their dense counterparts.