💻 computer science

From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition

This paper introduces SITH, a novel data-free and training-free framework that decomposes CLIP's attention head weights into singular vectors and interprets them as sparse, coherent concepts via a new algorithm, enabling faithful intra-head explanations and precise model edits without retraining.

Original authors: Francesco Gentile, Nicola Dall'Asen, Francesco Tonini, Massimiliano Mancini, Lorenzo Vaquero, Elisa Ricci

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Francesco Gentile, Nicola Dall'Asen, Francesco Tonini, Massimiliano Mancini, Lorenzo Vaquero, Elisa Ricci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot artist named CLIP. This robot can look at a picture and describe it perfectly, or read a description and find the matching picture. But here's the problem: no one really knows how it does it. It's like a "black box." You put a picture in, and a description comes out, but the gears inside are hidden.

Usually, to understand how a robot thinks, scientists have to feed it thousands of pictures and watch what happens inside while it works. This is like trying to understand a chef by watching them cook a meal; you see the ingredients they grab, but you don't know the secret recipe written on the wall. Also, if the chef only cooks with ingredients from a specific grocery store, you might think that's all they know, which isn't true.

The paper introduces a new method called SITH (which sounds like a spooky ghost, but stands for Semantic Inspection of Transformer Heads). SITH is like a magic X-ray that lets us see the robot's "brain" without ever showing it a single picture.

Here is how SITH works, broken down into simple steps:

1. The "Recipe Book" vs. The "Cooking Show"

Most previous methods were like watching the "Cooking Show" (looking at activations). They needed a lot of data (ingredients) to guess what the robot was thinking. If the robot saw too many pictures of cats, it might think it only knows about cats.

SITH is different. It looks directly at the Recipe Book (the weights). It doesn't care what the robot is currently cooking; it just reads the instructions written in its brain. This means it's data-free. It doesn't need to see a single image to understand how the robot works.

2. The "Swiss Army Knife" Analogy

Inside the robot's brain, there are many little tools called Attention Heads. Think of one of these heads as a Swiss Army Knife.

Old way: Scientists would say, "This knife is good for cutting." That's a bit vague. Is it good for cutting paper? Steak? Hair?
SITH's way: SITH takes the knife apart. It uses a mathematical tool called SVD (Singular Value Decomposition) to separate the knife into its individual blades: a tiny screwdriver, a small scissors, and a bottle opener.

3. The "Translator" (COMP)

Once SITH separates the Swiss Army Knife into individual blades, it still needs to know what each blade does. This is where a new algorithm called COMP comes in.

Imagine you have a weird, abstract shape (a blade). You want to describe it using words.

The Problem: If you just pick the closest word, you might say "red" for a shape that is actually "a red, spiky, angry star." You miss the nuance.
The SITH Solution (COMP): COMP is like a smart translator that builds a description using a sparse, coherent combination. It says, "Okay, this blade isn't just 'red.' It's a mix of 'crimson,' 'sharp,' and 'danger.'" It picks a few words that fit together perfectly to describe that specific blade.

4. What Did They Find?

When they used SITH on the CLIP robot, they found some amazing things:

Specialized Blades: They discovered that inside one "knife" (attention head), there are specific blades dedicated to colors (one blade for "red," another for "blue"), locations (one for "kitchens," another for "beaches"), and even letters (some blades can actually "read" text inside an image!).
No Bias: Because they looked at the recipe book and not the cooking show, they found these features even if the robot had never seen a picture of a "pink flamingo" during its training.

5. The Superpower: "Brain Surgery" Without Anesthesia

The coolest part is what you can do with this knowledge. Since SITH understands the robot's brain in terms of concepts (like "red" or "background"), we can perform surgery on the robot without retraining it.

Fixing Bad Habits: Imagine the robot thinks a "bird" is only a bird if it's standing on a "tree." If you show it a bird on a "rock," it gets confused. SITH can find the specific "tree" blade and turn it down. Now the robot can see birds on rocks too!
Safety: If the robot is too sensitive to "scary" or "unsafe" images, SITH can find the "scary" blades and mute them, making the robot safer to use.
Better Performance: If you want the robot to be better at identifying flowers, SITH can turn up the volume on the "flower" blades and turn down the "cars" blades.

Summary

SITH is a tool that lets us peek inside a vision AI's brain by reading its instruction manual (weights) instead of watching it work. It breaks down complex brain parts into simple, human-readable concepts (like "green" or "beach"). This allows us to fix the robot's bad habits, make it safer, and improve its skills—all without needing to feed it more data or retrain it from scratch.

It's like taking a black box, opening it up, finding the specific wires responsible for "red" and "blue," and being able to tweak them with a screwdriver, all while the robot is still running.

1. Problem Statement

As Vision-Language Models (VLMs) like CLIP are deployed at scale, understanding their internal mechanisms is critical. Existing interpretability methods face three major limitations:

Data Dependence: Most methods (e.g., TextSpan, SAEs) rely on model activations derived from large datasets (e.g., ImageNet). This makes them vulnerable to dataset biases and limits their applicability to scenarios where data is unavailable or sensitive.
Coarse Granularity: Activation-based methods often provide explanations at the attention head level (e.g., "Head 1 focuses on colors"). They fail to dissect the internal structure of a head, missing the specific sub-components responsible for distinct concepts.
Instability: Methods like Sparse Autoencoders (SAEs) can be unstable, producing different feature dictionaries even when trained on similar data, and often struggle with the "superposition" problem where neurons encode multiple unrelated concepts.

The paper asks: Can we interpret the internal mechanisms of CLIP directly from its weights, without any data or training, to achieve fine-grained, human-interpretable explanations?

2. Methodology: SITH and COMP

The authors propose SITH (Semantic Inspection of Transformer Heads), a fully data-free and training-free framework.

A. Weight-Based Analysis (SITH)

Instead of analyzing activations, SITH analyzes the Value-Output (VO) weight matrix ( $W_{VO}$ ) of each attention head in CLIP's Vision Transformer.

Isolation: The authors mathematically isolate the $W_{VO}$ matrix (combining Value and Output projections) from the Multi-Head Attention mechanism. This matrix determines what information a head writes to the residual stream.
Singular Value Decomposition (SVD): They decompose $W_{VO}$ $W_{V O}$ using SVD:
$W_{VO} = U \Sigma V^T$
- Right Singular Vectors ( $V$ ): Represent the dominant output directions (writing directions) of the head.
- Left Singular Vectors ( $U$ ): Represent the dominant input directions (reading directions).
- Singular Values ( $\Sigma$ ): Quantify the strength of these directions.
- Insight: Each singular vector corresponds to a specific computational direction within the head, allowing for a decomposition of the head's functionality into distinct sub-spaces.

B. Semantic Interpretation (COMP)

Since singular vectors exist in a high-dimensional embedding space, they must be mapped to human-interpretable concepts. The authors introduce COMP (Coherent Orthogonal Matching Pursuit).

Concept Pool: They use a large, overcomplete dictionary of text concepts (e.g., ConceptNet 5.5).
Sparse Decomposition: COMP attempts to represent a singular vector $\hat{v}$ as a sparse, non-negative linear combination of concept embeddings: $\hat{v} \approx \sum c_i \hat{\gamma}_i$ .
Coherence Regularization: Standard greedy algorithms (like NNOMP) optimize only for reconstruction error, often leading to semantically disjoint concepts (e.g., "red" + "toaster"). COMP modifies the selection criterion to include a coherence term:
$\text{score}(\hat{\gamma}_i) = \langle r_{k-1}, \hat{\gamma}_i \rangle + \frac{\lambda}{|S_{k-1}|} \sum_{j \in S_{k-1}} \langle \hat{\gamma}_i, \hat{\gamma}_j \rangle$
This ensures that selected concepts are not only good for reconstruction but also semantically related to previously selected concepts, yielding a coherent explanation (e.g., "mint green" + "grass green").

3. Key Contributions

Data-Free Interpretability: SITH is the first framework to provide fine-grained, intra-head explanations for VLMs purely from weights, eliminating dataset bias and the need for activation collection.
Fine-Grained Decomposition: By analyzing singular vectors rather than entire heads, SITH reveals that a single attention head often contains multiple distinct functional sub-components (e.g., one vector for "red," another for "green" within the same "color" head).
COMP Algorithm: A novel sparse coding algorithm that balances reconstruction fidelity with semantic coherence, producing human-readable concept sets.
Model Editing without Retraining: The framework enables precise "weight-space surgery." By amplifying or suppressing specific singular values, the authors can modify model behavior without gradient updates or training data.

4. Results and Experiments

A. Interpretability and Fidelity

Qualitative: SITH successfully maps singular vectors to specific concepts like textures, locations, colors, and even letters (e.g., a vector dedicated to the letter "C").
Quantitative: COMP achieves a superior trade-off between fidelity (how well the concepts reconstruct the vector) and interpretability (coherence of the concept set) compared to baselines like Top-k and NNOMP.
Grounding: Images retrieved based on similarity to specific singular vectors strongly align with the textual explanations generated by COMP.

B. Model Editing Applications

Suppressing Spurious Correlations: On the Waterbirds dataset (birds on backgrounds), SITH identified and suppressed singular vectors encoding "background" concepts. This improved worst-group accuracy significantly (from 47.9% to 70.6%) without retraining.
Safety (NSFW Removal): SITH identified vectors encoding unsafe concepts (nudity, violence) and suppressed them. The modified model outperformed Safe-CLIP (a model specifically trained for safety) on safe queries and showed improved robustness on unsafe queries.
Performance Improvement: By amplifying singular vectors relevant to specific downstream tasks (e.g., Flowers 102), the authors improved zero-shot classification accuracy by up to 1.0 point.

C. Model Adaptation Analysis

Fine-Tuning Mechanism: The authors analyzed how CLIP adapts to new tasks (Flowers, Pets, Birds) via fine-tuning.
Stable Basis: They found that fine-tuning does not learn entirely new features. Instead, it induces a subtle shift in the existing semantic basis. The singular vectors remain geometrically stable, but their weights (singular values) are reoriented to align with the target task.
Task Alignment: The "delta" vectors (difference between pre-trained and fine-tuned weights) were found to be semantically aligned with the specific domain (e.g., flower-related concepts for the Flowers dataset).

5. Significance and Impact

Paradigm Shift: Moves interpretability from "activation-based" (data-dependent) to "weight-based" (data-independent), offering a more robust and universal view of model internals.
Precision Control: Demonstrates that VLMs can be surgically edited at the level of individual semantic subspaces, offering a lightweight alternative to fine-tuning for safety and bias mitigation.
Universality: The findings suggest that different VLM architectures converge on similar functional circuits (e.g., specific heads for colors, locations) regardless of scale or training data, supporting the "Universality Hypothesis" in mechanistic interpretability.
Efficiency: Provides a method to understand and modify models without the computational cost of training or the logistical burden of curating large datasets.

In summary, SITH provides a powerful lens into the "black box" of CLIP, revealing that its knowledge is organized into stable, interpretable, and editable semantic subspaces defined by its weight matrices.