🤖 machine learning

Assessing Reliability of Symbol Detection in Concept Bottleneck Models

This paper reveals that high task accuracy in Concept Bottleneck Models does not guarantee reliable concept detection due to spurious shortcuts, and proposes a reliability-aware training strategy that optimizes shared concept detectors across multiple heads to mitigate this issue and significantly improve the interchangeability of detectors and classification heads.

Original authors: Javier Fumanal-Idocin, Javier Andreu-Perez

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Javier Fumanal-Idocin, Javier Andreu-Perez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of experts to identify birds. You want them to be explainable, meaning they shouldn't just say "That's a Robin!" and stop. Instead, they must first list the features they see: "It has a red breast," "It has a small beak," and "It has a grey leg." Only after listing these "concepts" do they make the final guess.

This is how Concept Bottleneck Models (CBMs) work. They are popular in AI because they seem transparent. However, this paper asks a scary question: Are these experts actually seeing the features, or are they just guessing the bird based on hidden shortcuts?

Here is a breakdown of the paper's findings and solutions, using simple analogies.

1. The Problem: The "Cheat Sheet" Effect

In a standard AI setup, the "feature detector" (the part that spots the red breast) and the "classifier" (the part that names the bird) are trained together at the same time.

The authors found that when you train them together, they often develop a secret language.

The Analogy: Imagine a student (the detector) and a teacher (the classifier) studying for a test together. The student doesn't actually learn what a "red breast" looks like. Instead, they notice that every time the teacher sees a red background in the photo, the answer is "Robin." So, the student starts shouting "Red Breast!" whenever they see a red background, even if the bird is actually a Blue Jay.
The Result: The AI gets the right answer (the bird name) 100% of the time, but its explanation is a lie. It thinks it's seeing a red breast, but it's actually just reacting to the background color. This is called information leakage.

2. The Stress Test: The "Swap" Experiment

How do you know if the AI is cheating? The authors came up with a clever test: The Swap.

The Analogy: Imagine you have five different students (detectors) and five different teachers (classifiers). You train them all separately.
- Scenario A (Good): If Student 1 is truly learning what a "red breast" looks like, they should be able to hand their notes to any of the other teachers, and the teacher should still get the bird right.
- Scenario B (Bad/Cheat): If Student 1 is just memorizing "Red Background = Robin" for their specific teacher, and you swap them with a new teacher who doesn't know that secret code, the system will fail miserably.
The Findings:
- On a real-world bird dataset (CUB-200), the models were surprisingly honest. Swapping students and teachers didn't break the system much. The "concepts" were real.
- On a fake, controlled dataset, the authors turned down the "honesty training" (concept supervision). Suddenly, the models kept getting the bird name right, but when you swapped the parts, the system collapsed to random guessing. The concepts were completely fake; they were just shortcuts.

3. The Solution: The "Group Project" Strategy

The authors propose a new way to train these models to stop them from cheating.

The Old Way: One student trains with one teacher. They get too comfortable and develop secret shortcuts.
The New Way (Reliability-Aware Training): One student is forced to work with five different teachers at the same time.
- The Analogy: Imagine a student trying to learn "Red Breast." If they try to cheat by saying "Red Background," Teacher A might say "No, that's wrong for this bird." Teacher B might say "Actually, that's a Blue Jay." Because the student has to satisfy all five teachers simultaneously, they can't rely on a shortcut that only works for one of them. They are forced to learn the actual feature (the red breast) because that's the only thing that satisfies everyone.
The Result: This method significantly reduced the cheating. Even when the training was tricky, the swapped models performed much better, proving the concepts were real and not just shortcuts.

4. Checking the "Confidence" Meter

The paper also looked at Uncertainty.

The Analogy: If five students look at a bird and four say "Grey Leg" and one says "No Leg," the group is confused. The paper found that when the AI is confused about a specific feature (like the leg color), it usually leads to a wrong final guess.
By measuring this "group disagreement," the system can flag when it is making a guess based on shaky evidence.

Summary

This paper is a "sanity check" for explainable AI. It shows that just because an AI gives you a list of reasons for its decision, doesn't mean those reasons are true. They might be clever lies (shortcuts) learned during training.

The authors proved that by forcing the AI to explain itself to multiple different "judges" at once, you can stop it from learning these lies and force it to learn the actual features, making the AI both accurate and truly trustworthy.

Technical Summary: Assessing Reliability of Symbol Detection in Concept Bottleneck Models

1. Problem Statement

Concept Bottleneck Models (CBMs) are a prominent class of neuro-symbolic methods designed for Explainable AI (XAI). They operate by first extracting human-interpretable concepts from input data and then using a lightweight, interpretable classifier to map these concepts to final predictions. While high task accuracy is often used as a proxy for model quality, this paper argues that accuracy does not guarantee faithful concept detection.

The core problem identified is information leakage (or "shortcuts") within the bottleneck. When concept detectors and classification heads are jointly trained, the detector may encode task-specific signals into the concept activations that are useful for classification but do not align with the intended semantic meaning of the concepts. Consequently, the model may achieve high accuracy by relying on spurious correlations rather than genuine concept understanding. This renders the model's explanations unreliable, as the intermediate symbols may not represent what they claim to represent.

2. Methodology

2.1 Measuring Misalignment

The authors propose a suite of diagnostics to assess the reliability of symbol detection, moving beyond standard accuracy metrics:

Global Symbol Reliability (Swap Test): The authors train multiple CBMs independently. They then swap the concept detector from one model with the classification head from another.
- If the swapped model maintains high accuracy, the concept detector and head share a robust, invariant symbolic representation.
- A significant drop in accuracy ("swap drop") indicates that the detector and head have co-adapted to spurious encodings specific to their original pairing.
Symbol Interchangeability: To test if the model relies on the identity of concepts or merely the count of fired concepts, the authors employ two null distributions:
- Concept-dimension permutation: Randomly shuffling concept labels before feeding them to the head.
- Within-concept sample-shuffle: Permuting concept values across samples.
- If accuracy remains high under these permutations, the symbols are not distinguishable as distinct concepts, indicating unreliable detection.
Individual Prediction Reliability: Using an ensemble of independently trained detectors, the authors decompose predictive uncertainty into aleatoric (data noise) and epistemic (model disagreement) components. They attribute label uncertainty to specific symbols by freezing individual concept predictions to the ensemble consensus and measuring the reduction in label uncertainty.

2.2 Proposed Solution: Reliability-Aware Training

To mitigate information leakage, the paper proposes a training strategy that reduces co-adaptation between the detector and the head:

Multi-Head Training: Instead of training one detector with one head, a shared concept detector $g_\theta$ $g_{θ}$ is trained against an ensemble of $R$ $R$ independently initialized classification heads $\{f^{(r)}_{\phi_r}\}$ ${f_{ϕ_{r}}^{(r)}}$ .
- The objective function averages the classification loss across all heads.
- Concept Dropout: To prevent the heads from converging to the same solution (which would negate the ensemble effect), each head receives concept activations masked by an independent random dropout mask. This forces the detector to learn representations that are robust to missing information and not reliant on single shortcuts.
Reliability-Aware Regularization:
- A global reliability score $r_k$ is computed for each concept based on its detection accuracy on a validation set.
- Instance-level epistemic uncertainty $U_{epi}(x_i, k)$ is estimated for each symbol.
- An unreliability weight $\omega_{ik}$ is calculated combining global reliability and instance uncertainty.
- A penalty term is added to the loss function that discourages the classifier from relying on concepts with low global reliability or high instance-level uncertainty.

3. Key Results

3.1 CUB-200-2011 Dataset (Full Supervision)

On the bird species classification task with full concept supervision ( $\lambda=1$ ):

High Interchangeability: Detectors and heads are almost freely interchangeable. The swap drop is minimal (below 1 accuracy point), and relative retention is above 99%.
Robustness: No concepts were detected below chance levels, and heads trained on one seed transferred almost losslessly to detectors from other seeds.
Uncertainty: Symbol-level epistemic uncertainty was low, but label-level uncertainty was significantly higher, indicating that small disagreements across many concepts compound at the classification head.

3.2 Synthetic Dataset (Controlled Leakage)

On a synthetic colored-shape dataset, the authors varied the concept-supervision weight $\lambda$ to simulate leakage:

Leakage Phenomenon: As $\lambda$ decreased (weakening concept supervision), matched accuracy remained near-perfect (~100%), but swapped accuracy collapsed to chance levels (20–30%). Agreement between detected concepts and ground truth also dropped to ~50%.
Mitigation: The proposed multi-head and reliability-aware strategies substantially mitigated this leakage. At low $\lambda$ (0.1), these methods roughly doubled the swap accuracy (from ~26% to ~50%) compared to standard joint training, confirming that the detector was learning more robust, less head-specific representations.

3.3 Diagnostic Findings

Spurious Firing: In the joint training regime, specific concepts (e.g., leg color) showed high disagreement among ensemble detectors. This disagreement was directly linked to prediction errors; freezing the contested symbol to the consensus often corrected the misclassification.
Null Tests: Heads trained on standard joint CBMs collapsed to near-chance accuracy when concept identities were permuted, confirming that the models genuinely relied on the specific concept detection rather than incidental correlations.

4. Significance and Claims

The paper claims to provide a rigorous framework for stress-testing the reliability of CBMs, moving beyond the assumption that high accuracy implies faithful explanations.

Validation of Leakage: The authors demonstrate that high task accuracy can coexist with symbols that "no longer mean what they claim," particularly when concept supervision is weak or when joint training encourages shortcuts.
Practical Mitigation: The proposed multi-head training with concept dropout and reliability-aware regularization offers a practical method to reduce information leakage without sacrificing task performance significantly.
Diagnostic Utility: The swap test and uncertainty decomposition provide actionable diagnostics to identify which specific concepts are prone to spurious firing, allowing for targeted intervention.

The authors conclude that while their reliability-aware training mitigates leakage, it involves a trade-off, sometimes slightly reducing overall accuracy. Harmonizing high performance with strict reliability remains an open challenge for future work. The paper does not claim to solve all XAI issues but specifically addresses the fidelity of the intermediate symbolic layer in CBMs.