💻 computer science

Capability $\neq$ Interpretability: Human Interpretability of Vision Foundation Models

This paper introduces a psychophysics-based framework demonstrating that vision foundation models are consistently less human-interpretable than supervised counterparts, revealing that interpretability is an independent dimension driven by feature locality and coarse-grained semantic alignment rather than overall model capability.

Original authors: Julien Colin, Lore Goetschalckx, Nuria Oliver, Thomas Serre

Published 2026-05-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Julien Colin, Lore Goetschalckx, Nuria Oliver, Thomas Serre

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Can We Understand the "Brain" of AI?

Imagine you have a super-smart robot that can identify a cat, a car, or a traffic light with incredible accuracy. It's so good at its job that it's being used to drive cars and help doctors diagnose diseases. But here's the problem: How does it actually "see" the world?

If you ask the robot, "What part of this picture made you think it was a cat?" it doesn't have a voice. It just has internal signals (features) that light up. The big question this paper asks is: Can a regular human look at those signals and understand what they mean?

The authors call this Interpretability. It's not about how smart the robot is (its capability); it's about how clear its thinking is to us.

The Old Way vs. The New Way

The Old Way (The "Multiple Choice" Trap):
Previous tests tried to see if humans understood AI features by showing them pictures and asking, "Which of these two pictures matches the AI's signal?"

The Flaw: The authors found a trick in this method. Humans could often get the right answer just by knowing what the feature wasn't.
- Analogy: Imagine I show you a picture of a cornfield and a picture of a beach, and I say, "This AI signal is NOT about corn." You can easily pick the beach, even if you have no idea what the signal actually is. You're just eliminating the wrong answer, not truly understanding the signal. This made the test unfair and unreliable.

The New Way (The "Detective" Framework):
The authors built a new, fairer test with two parts, like a detective trying to solve a mystery:

Localizability (The "Where" Game):
- The Setup: You are shown a "mystery signal" along with a few examples of what triggers it (e.g., a picture of a tire, a heatmap showing where the signal is bright).
- The Task: You are shown a new picture and asked to click exactly where you think that signal would light up.
- The Goal: Can you point to the specific part of the image (like the tire) that matches the signal?
Nameability (The "What" Game):
- The Setup: Same visual clues as above.
- The Task: You have to write a short description of what the signal represents.
- The Goal: Can you describe it in words? (e.g., "It's a car wheel" or "It's a red traffic light").

To make sure they were testing pure "features" and not just messy, mixed-up neurons, they used a special tool (a Sparse Autoencoder) to isolate single, clear concepts, like separating individual ingredients from a smoothie.

The Shocking Discovery: Smarter Doesn't Mean Clearer

The researchers tested six different AI models:

2 Older Models: Trained in a traditional, supervised way (like a student with a teacher).
4 "Foundation" Models: The new, massive, super-powerful models (like DINO, CLIP) that are trained on the whole internet and can do almost anything.

The Result:
The "Foundation" models were less interpretable than the older, simpler models.

The Analogy: Imagine the older models are like a student who learned by memorizing a textbook. You can easily ask them, "Why did you pick that answer?" and they give a clear reason.
The new Foundation models are like a genius who learned by reading the entire library of human knowledge. They are smarter and can solve harder problems, but if you ask them to explain their reasoning, they give you a vague, confusing answer. Their internal "thoughts" are harder for humans to pin down.

Crucially, being "smarter" didn't help. The paper found zero connection between how well the model performed on tasks (like identifying objects) and how easy it was for humans to understand its features. Being a better driver doesn't mean your brain is easier to read.

What Makes a Feature Easy to Understand?

The authors found two specific things that make an AI's "thoughts" easier for humans to grasp:

Locality (The "Spotlight" Effect):
- If a feature lights up in a tight, specific spot (like just the tire of a car), humans can understand it easily.
- If a feature lights up everywhere at once (like the whole car and the road and the sky), humans get confused. The new Foundation models tend to have these "diffuse" signals that cover too much ground.
- Analogy: It's easier to find a specific person in a crowd if they are wearing a bright red hat (local) than if they are just "part of the general vibe" of the crowd (diffuse).
Coarse-Grained Alignment (The "Big Picture" Match):
- Humans understand the world in broad categories (e.g., "Animals," "Vehicles"). If the AI organizes its features to match these big categories, humans understand it better.
- If the AI focuses on tiny, hyper-specific details (like the exact texture of a butterfly's wing compared to another butterfly), humans actually find it harder to understand.
- Analogy: It's easier to explain a map if you show me "North America" and "South America" (coarse) rather than trying to explain the specific shape of every single street in a city (fine-grained).

The One Bright Spot: DINOv3

There was one exception. A model called DINOv3 was both very capable and very interpretable. Why? Because the people who built it specifically programmed it to look for "local" features (things that light up in specific spots). This proves that we can build smart models that are also easy to understand, but we have to design them with that goal in mind.

Summary

Capability $\neq$ Interpretability: Just because an AI is super smart doesn't mean we can understand how it thinks. In fact, the newest, smartest models are often the hardest to understand.
The Gap: There is a gap between "doing the job well" and "explaining the job clearly."
The Fix: To make AI more transparent, we need to train them to focus on specific, local details and organize their knowledge in broad, human-like categories, rather than just making them bigger and more powerful.

The paper concludes that we cannot assume that as AI gets smarter, it will naturally become more understandable. We have to build that clarity in from the start.

Technical Summary: Capability $\neq$ Interpretability: Human Interpretability of Vision Foundation Models

Problem Statement

As vision foundation models (e.g., CLIP, DINOv2, SigLIP) transition from research benchmarks to high-stakes deployments (clinical decision support, autonomous driving), understanding their internal representations becomes critical. However, existing evaluation protocols fail to reliably measure and compare the human interpretability of features across different models. Prior methods suffer from two main limitations:

Polysemanticity: They often rely on individual neurons, which encode multiple concepts (superposition), complicating the isolation of single features.
Methodological Artifacts: Previous psychophysics protocols (e.g., forced-choice tasks) allow participants to solve tasks by identifying what a feature is not (process of elimination) rather than understanding what it is. This leads to model-specific chance-level baselines that make raw accuracy scores incomparable across architectures.

Consequently, it remains an open question whether the superior capability of foundation models correlates with their features being more interpretable to humans.

Methodology

The authors introduce a framework to measure human interpretability using two complementary psychophysics protocols, applied to six vision transformers (two supervised ViTs and four foundation models: DINOv2, DINOv3, CLIP, SigLIP).

1. Feature Extraction & Selection:

Monosemantic Basis: Instead of analyzing raw neurons, the authors use Sparse Autoencoders (SAEs) trained on ImageNet activations to recover a dictionary of monosemantic directions. This ensures each tested unit corresponds to a single concept.
Relevance Filtering: Features are selected based on their contribution to a linear probe trained on ImageNet (using Gradient $\times$ Input). Dense features (activating on most inputs) are excluded.
Standardized Anchoring: To ensure fair comparison, feature selection is anchored to a shared set of input images rather than random sampling, and scores are calibrated against a common chance level.

2. Psychophysics Protocols:
Participants (377 valid subjects, 13,400 responses) were presented with visual explanations of a feature (synthetic visualization, highly activating natural images, and RISE heatmaps) and asked to perform one of two tasks:

Localizability (Where): Participants click on a novel image where they believe the feature fires. Scores are mapped to a chance-anchored scale ( $s=0.5$ ) based on the empirical cumulative distribution of the heatmap, normalizing for sparse vs. diffuse activations.
Nameability (What): Participants provide a free-text description of the feature. Descriptions are scored by computing the mean cosine similarity between the text embedding and the CLIP image embeddings of crops centered on the feature's peak activation.

3. Analysis:
The study analyzes the relationship between interpretability scores and:

Downstream task performance (ImageNet classification, semantic segmentation, perceptual grouping).
Representational properties: Locality (Hoyer sparsity of activations) and Human Alignment (visual strategies, coarse-grained vs. fine-grained semantic similarity judgments).

Key Results

1. Foundation Models are Less Interpretable:
Contrary to the assumption that larger, more capable models are more transparent, every foundation model tested produced features that were less interpretable than their supervised counterparts (ViT-S and ViT-B). This gap holds for both vision-only self-supervised models (DINOv2, DINOv3) and vision-language models (CLIP, SigLIP).

Note: DINOv2 elicited the highest participant confidence but ranked among the least interpretable, highlighting that "feeling" interpretable does not equate to being interpretable.

2. Interpretability is Orthogonal to Capability:
There is no significant correlation between a model's downstream task performance (classification, segmentation, grouping) and its human interpretability scores (localizability or nameability). High capability does not guarantee high interpretability, nor does it necessarily trade off against it; they appear to be independent dimensions of representation quality.

3. Predictors of Interpretability:
Two representational properties strongly predict interpretability:

Locality: Features with focal, spatially confined activations (high Hoyer sparsity) are significantly more interpretable than diffuse features. DINOv3, which explicitly promotes local features in its training objective, serves as a counterexample, showing higher interpretability among foundation models.
Coarse-Grained Semantic Alignment: Alignment with coarse-grained human similarity judgments (broad categorical structure) correlates with interpretability. In contrast, fine-grained perceptual alignment (within-category distinctions) and alignment with human visual strategies (attention maps) do not predict interpretability.

Significance and Claims

The paper establishes that capability alone cannot close the interpretability gap between foundation models and supervised baselines. The authors claim:

Interpretability is a distinct, measurable dimension: It is not a byproduct of scale or downstream performance.
The "Foundation Model Gap" is real: Despite surpassing supervised models on benchmarks, foundation models learn representations that are less legible to humans.
Design Implications: To build more interpretable models, training objectives should explicitly promote local feature activations and align representations with coarse-grained human semantic structures, rather than relying on scale or fine-grained perceptual alignment.

The study concludes that while foundation models offer superior generalization, their internal features are often "superhuman" in a way that obscures human understanding, necessitating explicit design choices to recover interpretability in high-stakes applications.