🤖 AI

Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations

This paper introduces a human-grounded evaluation framework featuring the Fully-Binary Matching Pursuit (FBMP) algorithm and the Targeted Attribute Perturbation Alignment Score (TAPAScore), along with synthetic benchmarks (synCUB and synCOCO), to rigorously quantify the semantic alignment and interpretability of sparse autoencoders, revealing that moderate dictionary sizes offer the optimal trade-off for concept extraction.

Original authors: Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Begüm Demir

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Begüm Demir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Trying to Read a "Black Box"

Imagine you have a super-smart robot (a Vision Model) that can look at a photo and tell you what's in it. But inside the robot's brain, the information isn't stored in neat, labeled folders like "Dog," "Tree," or "Blue Sky." Instead, it's stored as a massive, chaotic cloud of numbers.

To understand how the robot thinks, scientists use a tool called a Sparse Autoencoder (SAE). Think of the SAE as a translator or a decoder ring. Its job is to take that chaotic cloud of numbers and break it down into simple, human-readable "concepts."

For example, instead of a messy mix of numbers, the SAE might say: "Ah, this specific number is on when there is a 'striped belly,' and this other one is on when there is a 'solid belly'."

The Problem: Is the Translator Lying?

The problem is that we don't always know if the translator is doing a good job.

The "Feature Splitting" Problem: Imagine the concept of "striped belly" is so complex that the translator splits it into three different numbers. One number means "stripes," another means "belly," and a third means "brown." If you look at just one number, you think it means nothing. The concept is fragmented.
The "Feature Absorption" Problem: Imagine the translator has a number for "bird," but it's so busy that it ignores specific details like "red wings" because it thinks the "bird" number covers everything.
The "Feature Composition" Problem: Sometimes, two different things (like "yellow head" and "yellow beak") get mashed into a single number, making it hard to tell them apart.

Until now, scientists mostly guessed if these translators were working by looking at how well they could rebuild the original image (like checking if a puzzle was put back together correctly). But a puzzle can look perfect even if the pieces are in the wrong order. We needed a way to check if the meaning was actually right.

The Solution: A New "Truth Test"

The authors built a new framework to test these translators using three main steps:

1. The "Match-Up" Game (Latent-Concept Matching)

First, they compared the translator's output against a list of human-annotated facts (e.g., "This bird has a solid belly").

Old Way: They tried to match one human fact to exactly one robot number.
New Way (FBMP): They realized that sometimes one human fact needs several robot numbers to be explained. They invented a method called Fully-Binary Matching Pursuit (FBMP).
- Analogy: Imagine you are trying to describe a "red car." The old way tried to find one magic word that meant "red car." The new way says, "Okay, let's use the word 'red' AND the word 'car' together." This allows the system to handle complex ideas that are split across multiple numbers.

2. The "Magic Edit" Benchmarks (Synthetic Datasets)

To really test the translator, you need to change the picture and see if the robot's brain changes in the right way. But in the real world, if you change a bird's belly pattern, you might accidentally change its pose or the background too.

The Fix: The authors created two fake (synthetic) datasets:
- synCUB: A collection of bird photos where they used AI to change only one thing (e.g., turning a "solid" belly into a "spotted" belly) while keeping everything else exactly the same.
- synCOCO: A collection of busy street scenes where they used AI to remove one object (like a car) without messing up the rest of the scene.
- Analogy: It's like having a magic editing tool that can swap a shirt on a person without changing their face, hair, or the background. This lets scientists isolate exactly what the robot is reacting to.

3. The "Direction Check" (TAPAScore)

This is the most important part. Just because a robot number lights up when a "striped belly" is present doesn't mean it causes the concept. It might just be a coincidence.

The Test: They take a picture, edit it to add a feature (like a spotted belly), and see if the specific robot number for "spotted" goes UP. Then they edit a picture to remove that feature and see if the number goes DOWN.
The Score: They call this TAPAScore. If the number goes up when the feature appears and down when it disappears, the translator is trustworthy. If the number stays the same or goes the wrong way, the translator is confused.

Key Findings: Less is More

The paper tested these tools on different types of translators with different "dictionary sizes" (how many numbers they are allowed to use).

Bigger isn't always better: They found that giving the translator a huge dictionary (thousands of numbers) actually made it worse at being interpretable. It started creating "ghost" concepts that didn't match reality.
The Sweet Spot: A moderate dictionary size provided the best balance. It was big enough to understand complex ideas but small enough to stay focused and clear.
The Winner: The combination of their new matching method (FBMP) and their new truth test (TAPAScore) was the only way to reliably tell the difference between a smart, trained translator and a random, untrained one.

Summary

The authors built a new "report card" for AI translators. Instead of just checking if the AI can rebuild an image, they check if the AI's internal thoughts actually match human concepts by:

Allowing multiple numbers to explain one idea (FBMP).
Using magic-edited photos to test cause-and-effect (synCUB/synCOCO).
Scoring whether the AI reacts in the right direction when the image changes (TAPAScore).

They discovered that for these AI brains to be truly understandable, they shouldn't be too big; a moderate size works best.

Technical Summary: Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations

Problem Statement

Sparse Autoencoders (SAEs) are increasingly employed to decompose high-dimensional internal representations of vision and vision-language models (e.g., CLIP, DINOv2) into sparse, interpretable latents. However, existing evaluation methods for vision-based SAEs rely heavily on structural proxies (e.g., reconstruction fidelity, sparsity) or qualitative inspection. These methods fail to measure semantic correspondence—the degree to which learned latents align with human-understandable concepts.

Furthermore, vision models exhibit systematic failure modes such as feature splitting (one concept fragmented across multiple latents), feature absorption (general features developing blind spots), and feature composition (co-occurring concepts merging). Current metrics, often adapted from natural language processing, struggle to validate whether a latent truly encodes a specific semantic attribute or merely correlates with it due to confounding factors. Crucially, functional evaluation in vision is difficult because isolating a single semantic change in natural images is non-trivial without controlled interventions.

Methodology

The authors propose a human-grounded evaluation framework that quantifies the alignment between SAE latents and human-annotated concepts without requiring user studies. The framework consists of three core components:

1. Latent-Concept Matching

To measure statistical alignment, the authors compare binarized SAE activations against binary ground-truth attribute vectors.

One-to-One Matching: Selects the single latent maximizing the F1 score for a given attribute.
Fully-Binary Matching Pursuit (FBMP): Recognizing that features often split across multiple latents, the authors introduce a coalition-based matching procedure. FBMP is a greedy, sequential selection algorithm tailored for the binary domain. Unlike standard Matching Pursuit, it operates entirely in the binary space:
- It uses binary similarity metrics (e.g., $F_\beta$ ) instead of inner products.
- It uses logical OR ( $\lor$ ) to sum contributions and logical AND/NOT ( $\land, \neg$ ) to update residuals.
- It builds a small subset (coalition) of latents that collectively reconstruct the attribute annotation, supporting many-to-one mappings.
$\Delta$ MATCHScore: To account for the inflation of matching scores in larger dictionaries (where random correlations increase), the authors subtract the matching score of an untrained (random weight) SAE from the trained SAE's score.

2. Synthetic Intervention Benchmarks

To enable functional validation, the authors construct two synthetic datasets where image pairs differ in exactly one semantic attribute:

synCUB: Derived from CUB-200-2011, this dataset targets fine-grained attribute perturbations (e.g., changing "solid" belly pattern to "spotted") in single-object bird images. It uses Flux2 to edit base images guided by reference images, preserving identity, pose, and background.
synCOCO: Derived from MS-COCO, this dataset targets object removal in complex multi-object scenes. It removes specific objects while preserving the rest of the scene, addressing the challenge of controlled interventions in naturalistic settings.

3. Targeted Attribute Perturbation Alignment Score (TAPAScore)

This metric tests whether matched latents respond selectively and in the expected direction under targeted perturbations.

For a pair of images $(x, \hat{x})$ differing in one attribute, the authors compute the signed response of the latent coalition matched to the added or removed attribute.
TAPAScore is defined as $\Delta_{add} - \Delta_{rem}$ , where $\Delta$ represents the average change in the maximum activation of the matched latent set. A positive score indicates that the latents increase when the attribute is added and decrease when removed, confirming causal encoding.

Key Contributions

FBMP Algorithm: A novel coalition-based matching procedure that supports many-to-one mappings between SAE latents and concepts. The authors demonstrate that FBMP consistently outperforms one-to-one baselines and passes sanity checks that existing metrics (like Feature Monosemanticity Score or CKNNA) fail.
Synthetic Benchmarks (synCUB & synCOCO): The first intervention-style benchmarks for SAE evaluation in naturalistic images, enabling controlled attribute and object perturbations.
TAPAScore: A functional metric that validates whether statistical matches correspond to causal semantic encodings by testing response directionality under perturbation.
Empirical Guidelines: The study provides actionable guidance for practitioners, recommending moderate dictionary sizes and the combination of FBMP with TAPAScore for optimal interpretability evaluation.

Experimental Results

The framework was evaluated on SAEs trained on CLIP and DINOv2 embeddings across varying dictionary sizes ($128$ to $4096$) and SAE variants (TopK, BatchTopK, Matryoshka, JumpReLU).

Sanity Checks: TAPAScore and $\Delta$ MATCHScore (with FBMP) were the only metrics capable of reliably distinguishing trained SAEs from untrained or random baselines. Metrics like CKNNA and Feature Monosemanticity Score (FMS) showed limited sensitivity or inconsistent behavior (e.g., CKNNA increasing for untrained SAEs).
Matching Performance: FBMP consistently outperformed one-to-one matching across all variants. The $F_{0.5}$ variant of FBMP (favoring precision) generally yielded the highest scores.
Dictionary Size Trade-offs:
- Increasing dictionary size (overcompleteness) improved statistical matching scores ( $\Delta$ MATCHScore) up to a point but often degraded perturbation alignment (TAPAScore).
- For TopK SAEs on CUB, TAPAScore peaked early and then sharply declined as dictionary size increased, indicating that larger dictionaries achieved better statistical alignment without encoding causally valid concepts.
- Moderate dictionary sizes (e.g., 256–1024 depending on the variant) offered the best trade-off between semantic coverage and functional selectivity.
Correlation: While matching scores and TAPAScore were generally positively correlated, the divergence at large dictionary sizes (where matching improved but TAPAScore dropped) highlighted that statistical alignment is insufficient for guaranteeing interpretability.

Significance and Claims

The paper claims that interpretability cannot be assumed from architectural design or reconstruction quality alone; it must be explicitly evaluated against human-understandable concepts. The proposed framework operationalizes interpretability as the alignment between learned features and annotated concepts, validated through controlled interventions.

The authors conclude that moderate dictionary sizes provide the optimal balance, yielding the most interpretable SAEs. They recommend FBMP ( $F_{0.5}$ , $k=3$ ) paired with TAPAScore as the default evaluation protocol. The work highlights that increased overcompleteness can reduce perturbation alignment, suggesting a loss of interpretability despite improved statistical metrics.

The authors acknowledge limitations, noting that the framework's quality is bounded by the granularity and accuracy of human annotations. They suggest future work could reduce reliance on manual annotations by leveraging vision-language models for automatic attribute generation and explore using TAPAScore as a tool for active model steering.