📊 statistics

CB-SLICE: Concept-Based Interpretable Error Slice Discovery

The paper introduces CB-SLICE, a concept-based error slice discovery method that leverages Concept Bottleneck Models to identify systematic model failures by grouping samples with shared concept mispredictions, thereby providing more faithful and interpretable explanations than existing approaches.

Original authors: Yael Konforti, Mateo Espinosa Zarlenga, Elaf Almahmoud, Mateja Jamnik

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yael Konforti, Mateo Espinosa Zarlenga, Elaf Almahmoud, Mateja Jamnik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that is great at most jobs, but occasionally it makes strange, specific mistakes. Maybe it thinks a picture of a bird on a lake is a water bird, even if it's a land bird, or it confuses a specific type of dog for a cat.

Usually, when these robots fail, we can only say, "It got it wrong." We don't know why it got it wrong. Did it look at the background? Did it miss a specific feature? Traditional methods of finding these mistakes are like hiring a detective who looks at the robot's output and guesses what went wrong based on general knowledge, not the robot's actual brain. This detective might be right, but they are often just guessing and might miss the real cause.

Enter CB-SLICE: The "Internal Logic" Detective

The paper introduces a new tool called CB-SLICE. Think of this tool as a detective that doesn't just look at the robot's final answer; it actually speaks the robot's native language.

Here is how it works, broken down into three simple steps:

1. The Robot's "Thought Process" (Concept Bottleneck Models)

To use CB-SLICE, the robot must be built in a special way called a Concept Bottleneck Model (CBM).

Normal Robot: Looks at a picture and instantly guesses "Dog" or "Cat." It's a black box; we don't know its steps.
CBM Robot: Looks at a picture and first lists its "thoughts" or concepts out loud. For example: "I see fur," "I see pointy ears," "I see a tail." Then, it uses those thoughts to guess "Dog."

Because the robot has to say its thoughts first, we can listen to them. CB-SLICE uses this feature to its advantage.

2. Finding the "Bad Thoughts" (Error-Prone Concepts)

When the CBM Robot makes a mistake, CB-SLICE asks: "Which of your thoughts led you astray?"

It looks at the robot's "thoughts" (concepts) for the pictures it got wrong.
It uses a special test to see which thoughts, if changed, would fix the mistake.
It filters out the "good thoughts" and focuses only on the "error-prone concepts"—the specific ideas (like "medium size" or "red color") that are consistently confusing the robot.

3. Grouping the Mistakes (Slicing)

Now that we know which thoughts are causing trouble, CB-SLICE groups the mistakes together.

Imagine you have a pile of wrong answers. CB-SLICE sorts them into buckets based on why they were wrong.
Bucket A: All the mistakes where the robot thought the bird was "medium size" but it was actually small.
Bucket B: All the mistakes where the robot thought the background was "ocean" but it was actually a "lake."

These buckets are called Error Slices. Instead of a messy pile of errors, you now have neat, organized groups of specific failure types.

4. The "Why" Explanation (Keyword Discovery)

For each bucket, CB-SLICE generates a simple explanation using keywords.

Instead of saying "The model failed on 500 images," it says: "The model fails when it sees a Landbird on a Lake background because it incorrectly thinks the bird is Medium Size."
This is powerful because it tells you exactly what the robot is misinterpreting. It's like the robot admitting, "I got confused because I thought that small bird was actually a medium one."

Why is this better than the old way?

Old Way (The External Detective): Uses a separate AI to guess why the robot failed. It might say, "Oh, the robot failed on dark skin tones because it looks like night." But that's just a guess from a different AI. It might be wrong or miss the real reason.
CB-SLICE (The Internal Detective): Looks directly at the robot's own "thoughts." If the robot thought "dark skin" was "night," CB-SLICE sees that specific thought and reports it. It is faithful to the robot's actual logic.

Real-World Examples from the Paper

The researchers tested this on several datasets:

Birds: They found that the robot often confused land birds on water backgrounds with water birds. CB-SLICE explained that the robot was specifically confused by the concept of "medium size" when combined with a "lake" background.
Faces: It found that the robot struggled with men who had "blonde hair" and "no big nose," a specific combination the robot hadn't seen enough of during training.
Digits: It discovered that the robot was systematically confusing the number "3" with "2" in certain positions, leading to wrong sums.

The Bottom Line

CB-SLICE is a tool that helps developers debug their AI models by grouping mistakes based on the specific "thoughts" (concepts) the model got wrong. It turns a confusing pile of errors into clear, organized lists with simple explanations, helping humans fix the robot's brain exactly where it needs help.

Important Note: The paper emphasizes that this tool works best when the robot is already built to "think" in concepts (a CBM). If the robot is a standard "black box" that doesn't list its thoughts, this specific tool cannot be used directly.

Technical Summary: CB-SLICE: Concept-Based Interpretable Error Slice Discovery

Problem Statement

Deep learning models often achieve strong average-case performance but exhibit systematic errors on specific population groups, known as error slices. Identifying these groups and their root causes is critical for debugging and bias mitigation. However, existing Error Slice Discovery Methods (SDMs) face a fundamental limitation: they typically rely on auxiliary language-based models (e.g., CLIP) to generate explanations. Consequently, these explanations are disconnected from the analyzed model's internal inference process, approximating the error source rather than revealing it. This disconnect leads to explanations that may be inaccurate, misleading, or biased by the auxiliary model itself. Furthermore, existing methods often fail to explain why a failure occurs in terms of the specific concepts the model relies upon.

Methodology: CB-SLICE

The authors propose CB-SLICE, a concept-based SDM that leverages Concept Bottleneck Models (CBMs) to integrate error slice discovery and bias explanation into a single, model-aware process. CBMs are interpretable architectures that first predict human-understandable semantic concepts (e.g., "dark skin," "asymmetry") and then predict the final task label based on these concepts. CB-SLICE operates in three distinct steps:

1. Error-Prone Concept Filtering

The method first identifies a subset of concepts most likely responsible for downstream misclassifications.

Mechanism: It utilizes the Expected Change in Target Prediction (ECTP) score. For each concept $i$ , the ECTP score quantifies the expected change in the downstream task distribution when intervening on that concept (setting its value to 0 or 1).
Process: The ECTP score is computed class-wise to mitigate imbalance. The top- $t_e$ concepts with the highest scores for each class are selected to form the error-prone concept set ( $C_{err}$ ). This ensures that subsequent slice formation focuses only on concepts driving errors.

2. Error Slice Formation

Once the error-prone concepts are identified, the method clusters erroneous validation samples to form coherent error slices.

Representation: Instead of using probability spaces, CB-SLICE uses concept logits ( $H$ ) restricted to the error-prone set $C_{err}$ . These logits encode the model's confidence and are well-approximated by Gaussian distributions.
Clustering: A Gaussian Mixture Model (GMM) is trained to cluster these logits. The objective function combines:
- GMM Likelihood ( $L_{GMM}$ ): To ensure semantic coherence.
- Auxiliary Losses ( $L_{ctrue}$ and $L_{cpred}$ ): Two linear classifiers predict ground-truth concept labels and predicted concept labels, respectively, based on slice membership probabilities. These losses encourage slices to group samples that share the same true and predicted concept values, ensuring the slices capture coherent concept-level error patterns.
Output: A set of slices $S = \{S_1, \dots, S_{t_g}\}$ .

3. Failure Mode Explanation

For each discovered slice, the method identifies the specific concepts responsible for its formation.

Mechanism: The method extends the ECTP score to measure the Expected Change in Slice Assignment (ECSA). It quantifies how intervening on a specific concept changes the posterior probability of a sample belonging to a specific slice.
Output: The top- $t_k$ concepts with the highest ECSA scores are selected as keyword-concepts for the slice. The presence or absence of these concepts is determined by the weighted average of their predictions across slice members.

Slice Prioritization

To address the impracticality of analyzing all discovered slices, CB-SLICE introduces a prioritization strategy based on an Informativeness Score (SI). This score combines:

Misprediction Coherence (MC): The consistency of downstream task mispredictions within a slice.
Semantic Compactness (SC): The degree to which slice members are concentrated in the concept-logit space.
Penalty Factor: A factor that down-weights very small slices to avoid noise.

Key Contributions

CB-SLICE Framework: A novel SDM that links discovered explanatory keywords directly to the model's internal decision logic via CBMs, avoiding the disconnect inherent in auxiliary-model-based approaches.
Error-Prone Concept Identification: A principled method using ECTP scores to filter concepts for both slice formation and failure mode characterization.
Slice Prioritization: A strategy to guide practitioners toward the most informative slices for targeted analysis.
Empirical Validation: Demonstrated consistent improvements over state-of-the-art baselines across multiple benchmarks.

Experimental Results

The authors evaluated CB-SLICE on four datasets with known error slices: Waterbirds, CelebA, MetaShift, and MNIST-Sum. They compared CB-SLICE against baselines including Domino, GEORGE, HiBug2, Spotlight, and K-Means.

Recovery of Known Biases (Precision@10): CB-SLICE consistently outperformed all baselines in recovering ground-truth error slices. For instance, on the Waterbirds dataset with a jointly trained CBM, CB-SLICE achieved a Precision@10 of 0.83, significantly surpassing the next best method (Domino at 0.62).
Failure Mode Alignment (MGF): CB-SLICE produced slices with higher Matched Ground-Truth Group Frequency (MGF), indicating that the discovered slices were more homogeneous and faithfully captured specific underlying failure modes.
Explanation Fidelity: Qualitative analysis showed that CB-SLICE provided faithful, model-grounded explanations.
- Example: In Waterbirds, CB-SLICE identified that misclassifications of Landbirds on water backgrounds were driven by systematic mispredictions of size-related concepts (e.g., "medium size"), rather than just the background.
- Example: In MNIST-Sum, it uncovered a failure mode where the digit "3" was systematically misread as "2," a nuance missed by text-based baselines.
Slice Selection: The paper demonstrates that the convergence of the CB-SLICE loss function aligns with the stabilization of performance metrics, providing a principled criterion for selecting the number of slices without ground-truth labels.

Significance and Claims

The paper claims that CB-SLICE reframes error slice discovery as a model-aware process. By grounding explanations in the representations the model actually learns (concept logits), it avoids the inaccuracies and potential biases introduced by post-hoc auxiliary models.

The authors emphasize that CB-SLICE distinguishes between distinct failure types:

Slices driven by systematic concept mispredictions (where the concept encoder $g$ fails).
Slices arising from rare, correctly predicted concept combinations (where the label predictor $f$ fails due to lack of training data for specific concept intersections).

The work positions CBMs as a principled foundation for debugging, offering richer and more faithful explanations of error sources compared to existing methods. The authors note that the effectiveness of the method relies on the availability of complete and faithful concept annotations, a limitation inherent to the CBM architecture itself.