🤖 machine learning

Graph Concept Bottleneck Models

This paper introduces GraphCBMs, a novel variant of Concept Bottleneck Models that constructs latent concept graphs to capture intrinsic relationships between concepts, thereby improving classification performance, enabling more effective interventions, and enhancing interpretability compared to traditional CBMs that assume concept independence.

Original authors: Haotian Xu, Tsui-Wei Weng, Lam M. Nguyen, Tengfei Ma

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haotian Xu, Tsui-Wei Weng, Lam M. Nguyen, Tengfei Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to recognize different types of birds.

The Old Way (The "Black Box" and the "Isolated Facts")
Traditionally, deep learning models are like "black boxes." You show them a picture, and they give you an answer ("That's a Robin!"), but you have no idea why they decided that.

To fix this, scientists created Concept Bottleneck Models (CBMs). Think of these as a model that doesn't just guess; it first lists the "clues" it sees.

Input: A picture of a bird.
Step 1 (The Clues): The model says, "I see red feathers, a small beak, and a short tail."
Step 2 (The Guess): Based only on those clues, it says, "That's a Robin."

This is great because a human can look at the clues and say, "Wait, that bird has a long tail, not a short one. Change your mind!" This is called intervention, and it makes the AI trustworthy.

The Problem: The "Silent Room"
However, the old CBMs had a weird flaw. They treated every clue as if it were in a silent room, completely alone.

If the model saw "feathers," it didn't realize that "feathers" usually go with "wings."
If it saw "a beak," it didn't know that "beaks" usually mean "wings" and "tails" are also likely.

In the real world, clues are rarely isolated. If you see a "beak," you almost certainly have a "head." If you see "fur," you probably have "paws." The old models ignored these natural connections, which sometimes made their guesses less accurate and their explanations a bit rigid.

The New Solution: The "Graph Concept Bottleneck" (Graph CBM)
This paper introduces Graph CBMs. Imagine the "clues" (concepts) are no longer in a silent room. Instead, they are in a busy town square connected by invisible roads.

The Map: The model learns a "map" (a graph) that connects related clues. "Beak" is connected to "Wings." "Fur" is connected to "Tail."
The Message Passing: When the model sees a picture, it doesn't just look at the clues in isolation. It lets the clues "talk" to each other. If the model is unsure about "Wings," it can ask the "Beak" clue for help. If "Beak" is strong, it sends a message to "Wings" saying, "Hey, I'm pretty sure we have a beak, so you should probably be active too."
The Result: The model makes better guesses because it understands the relationships between the clues, not just the clues themselves.

Why This Matters (The "Super-Intervention")
The paper shows that this new "town square" approach does three main things:

It's Smarter: By letting clues help each other, the model gets more accurate at identifying things (like birds, flowers, or skin conditions) without needing more data.
It's More Flexible: If a human wants to fix a mistake (intervention), the "town square" helps. If you tell the model, "Actually, this bird has a long tail," the model doesn't just change the "tail" clue. Because of the connections, it automatically updates related clues (like "body shape" or "wing span") to match the new story. It's like correcting one sentence in a story, and the whole paragraph automatically makes more sense.
It Works Everywhere: The researchers tested this on many different types of images (from common objects to medical X-rays) and found it worked well whether the model had a teacher (labeled data) or had to learn on its own (no labels).

The Bottom Line
The paper claims that by teaching AI models to understand that "clues" are connected to each other—just like how "rain" is connected to "umbrellas"—we can build systems that are not only more accurate but also easier for humans to understand and fix when they make mistakes. It's a simple "plug-in" upgrade that turns a list of isolated facts into a connected web of understanding.

1. Problem Statement

Concept Bottleneck Models (CBMs) are a framework for interpretable deep learning that maps input data to human-understandable intermediate concepts before making a final prediction. While CBMs allow for transparency and direct intervention (e.g., correcting a prediction by adjusting a concept score), existing CBMs suffer from a critical limitation:

Assumption of Independence: Standard CBMs assume that concepts are conditionally independent given the label.
Reality of Correlation: In practice, concepts are highly correlated (e.g., "fur" and "whiskers" often co-occur with "tail"). Ignoring these intrinsic dependencies leads to oversimplified representations, reduced predictive accuracy, and weaker interpretability.
Intervention Limitations: When users intervene on a single concept, standard CBMs fail to propagate the logical consequences of that change to related concepts, leading to suboptimal corrections.

2. Methodology: Graph CBMs (G-CBMs)

The authors propose Graph CBMs, a novel framework that explicitly models the relational structure among concepts using a latent concept graph. The architecture is designed as a lightweight, plug-and-play module compatible with existing CBM variants (both label-free and concept-supervised).

Core Components

The model consists of three main modules:

Module 1: Encoding & Parameterization
- Inputs: Images are encoded via a pretrained vision encoder ( $E_v$ ), and concepts are encoded via a pretrained language encoder ( $E_t$ ) (in label-free settings).
- Graph Structure: A latent graph $G = (V, A)$ is defined where nodes $V$ represent concepts and the adjacency matrix $A$ encodes the relational strength between them.
- Initialization: The graph is initialized (randomly or via embedding similarity) and shared across all data instances, but the activation of nodes is instance-dependent.
Module 2: GNN Message Passing
- To capture dependencies, the model employs a Graph Neural Network (GNN) with multiple layers.
- Dual Message Passing: At each layer $l$ $l$ , information is aggregated from neighboring nodes for both:
  - Embeddings ( $V_{emb}$ ): The semantic meaning of the concept.
  - Activations ( $V_{act}$ ): The extent to which a concept is activated for a specific image.
- The update rule combines the adjacency matrix with element-wise multiplication of activations and embeddings, allowing the model to propagate context from related concepts (e.g., if "wings" is active, the model boosts the probability of "feathers").
Module 3: Regularization & Prediction
- Contrastive Learning: Since ground-truth concept relationships are often unavailable, the model uses self-supervised contrastive learning to learn the graph structure.
  - Embedding-level Loss ( $L_{emb}$ ): Aligns the image embedding with the graph-level embedding (mean-pooled node embeddings).
  - Activation-level Loss ( $L_{act}$ ): Aligns the projected image embedding with the updated concept activation vector.
- Loss Function: The total loss combines the classification cross-entropy ($CE$), the contrastive losses ( $L_{emb}, L_{act}$ ), and an $L_1$ sparsity regularization term on the adjacency matrix to prevent overfitting and ensure interpretability.
- Prediction: The final label is predicted by passing the refined concept activations through a linear layer or MLP.

3. Key Contributions

Novel Framework: Introduction of Graph CBMs, the first CBM variant to explicitly model latent concept correlations via a learnable graph structure, enhancing both reasoning and interpretability.
Self-Supervised Graph Learning: Development of a framework that automatically constructs latent concept graphs without requiring ground-truth relational annotations, utilizing multi-level contrastive learning.
Plug-and-Play Integration: The method is orthogonal to existing CBM designs, seamlessly integrating into both Label-Free CBMs (e.g., LF-CBM, PCBM) and Concept-Supervised CBMs (e.g., standard CBM, CEM).
Robust Intervention: Demonstrated that modeling concept relations significantly improves the efficacy of post-hoc interventions, allowing corrections to propagate logically through the concept network.

4. Experimental Results

The authors evaluated Graph CBMs on multiple real-world datasets, including CUB-200-2011 (birds), Flower102, HAM10000 (skin lesions), CIFAR-10/100, and ChestXpert.

Predictive Accuracy:
- Graph CBMs consistently outperformed baselines across all datasets.
- Label-Free Setting: Graph-PCBM improved accuracy by ~1.5% to 3% over standard PCBM (e.g., 77.95% vs. 74.69% on CUB).
- Concept-Supervised Setting: Graph-CBM and Graph-CEM matched or surpassed state-of-the-art methods (e.g., ProbCBM, HardAR, E-CBM) while offering better interpretability.
Intervention Performance:
- In intervention experiments (modifying concept scores to correct errors), Graph CBMs showed significantly higher robustness.
- When concepts were masked or perturbed, Graph CBMs recovered performance better than baselines by aggregating signals from connected neighbors.
- Graph CBMs achieved high accuracy with single-step interventions, whereas energy-based models (like E-CBM) required iterative gradient calculations.
Ablation Studies:
- Removing the GNN message passing caused performance to drop to baseline levels, confirming that the gain comes from modeling interactions, not just increased parameters.
- Using both $L_{emb}$ and $L_{act}$ was crucial; using only one led to either overly dense or overly sparse graphs.
Graph Interpretability:
- The learned graphs revealed semantically meaningful connections (e.g., "loud, rasping voice" connected to "raucous call") that aligned with human intuition and even outperformed LLM-generated heuristics in capturing non-obvious correlations.

5. Significance and Impact

Bridging the Gap: Graph CBMs bridge the gap between the interpretability of CBMs and the relational reasoning capabilities of Graph Neural Networks.
Trustworthiness: By modeling how concepts influence one another, the model provides a more realistic and trustworthy decision-making process, crucial for safety-critical domains like healthcare (e.g., understanding how "pneumonia" relates to "pleural effusion").
Efficiency: The approach achieves state-of-the-art performance with minimal computational overhead (adding only $O(k^2)$ parameters for $k$ concepts) and does not require retraining the backbone image encoder.
Generalization: The framework is versatile, working effectively in both scenarios where concept labels are available and where they must be discovered automatically (label-free).

In conclusion, Graph CBMs represent a significant advancement in interpretable AI, moving beyond the assumption of independent concepts to capture the rich, interconnected nature of human reasoning, thereby improving both model accuracy and the reliability of human interventions.