🤖 machine learning

Spatially Grounded Concept Bottleneck Models via Part-Factorized Attention

This paper introduces a part-factorized Concept Bottleneck Model that leverages a frozen DINOv3 vision transformer, a foreground gate, and a learnable Gaussian spatial prior to enforce spatial grounding in concept attention, thereby significantly improving pointing accuracy while maintaining competitive classification performance on the CUB-200-2011 dataset with minimal or no per-image supervision.

Original authors: Dhanesh Ramachandram

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dhanesh Ramachandram

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a birdwatcher trying to identify a specific species of bird. You don’t just look at the whole bird and guess; you look for specific clues: Is the beak yellow? Is the throat white? Is the wing pattern striped?

This paper introduces a new way for Artificial Intelligence to think exactly like that careful birdwatcher. It’s called a Part-Factorized Concept Bottleneck Model (PF-CBM).

Here is the problem it solves, the solution it offers, and why it matters, explained with everyday analogies.

The Problem: The "Sloppy Student"

Traditional AI models are like students who are good at getting the right answer on a test but can’t explain how they got it. Worse, they often cheat.

Imagine a student asked to identify a bird by its throat color. To get the right answer, the student might ignore the throat entirely and instead look at the wing, because they’ve noticed that birds with white throats often have black wings. They get the right answer, but their reasoning is wrong. If you asked them to point to the throat, they’d point to the wing.

In AI terms, this is called being "spatially ungrounded." The model knows the concept ("white throat") exists, but it doesn’t know where in the image that concept lives. It’s guessing based on correlations, not actual evidence.

The Solution: The "Strict Librarian"

The authors created a model that acts like a strict librarian who forces the AI to look in the right place before it can answer. They built this on top of a powerful, pre-trained visual engine called DINOv3 (think of this as the AI’s eyes, which are already very good at seeing).

The PF-CBM adds four strict rules to make the AI honest:

1. The "Foreground Gate" (Ignoring the Clutter)

First, the AI must decide: Is this patch of the image the bird, or is it just a tree branch in the background?

Analogy: Imagine a spotlight that only shines on the bird and dims everything else. The AI is forced to ignore the background noise. This is done with a small "gate" that suppresses background pixels so the AI doesn’t get distracted by a fence or a leaf.

2. The "Fixed Map" (No Cheating Allowed)

The AI has 312 different "concept detectors" (like has_yellow_beak or has_striped_belly). In older models, these detectors could look anywhere. In this new model, each detector is hard-wired to look at only one specific body part.

Analogy: Imagine a library where every book is locked in a specific drawer. The "Beak Color" book can only be read from the "Beak" drawer. It is physically impossible for the "Beak Color" detector to look at the "Wing" drawer. This is called structural routing. If the AI wants to know about the beak, it must look at the beak.

3. The "Global Token" (The Big Picture)

Some questions aren’t about a specific body part. For example, "Is the bird large?" or "Is the shape round?"

Analogy: These questions are like asking about the whole library, not a specific book. The model has a special "Global Token" that looks at the entire bird (the foreground) to answer these general questions, without being tied to a specific anatomical spot.

4. The "Gaussian Prior" (The Rough Sketch)

Here is the tricky part: How does the AI know which "drawer" is the beak and which is the tail? If you just give it empty drawers, it might get confused and swap them (permutation symmetry).

Analogy: The authors give the AI a very rough, blurry sketch of a bird. They say, "The beak is usually around here, and the tail is usually over there." This is a Gaussian Prior—a statistical guess based on the average location of bird parts in the dataset.
The Magic: The AI doesn’t need to be told exactly where the beak is for every single photo. It just needs this rough starting hint. From there, it learns to refine its focus. The paper shows that you only need 0.5% of the training images (about 27 photos) to create this rough sketch. The rest of the learning happens without any human pointing out body parts.

The Results: Honest and Accurate

The team tested this on a dataset of 200 bird species (CUB-200-2011).

Accuracy: The model was just as good at identifying the bird species as the best fully supervised models (about 89% accuracy).
Honesty (Pointing Accuracy): This is where it shines. When asked to point to the evidence for its decision, the old models were only right 36% of the time. The new PF-CBM was right 69.5% of the time (and 52.2% even when using minimal supervision).
Efficiency: It achieved this high level of "honesty" without needing humans to label every single body part in every single image. It only needed a tiny fraction of labeled data to set up its initial "rough sketch."

Why This Matters

In high-stakes situations—like medical diagnosis or safety-critical engineering—you don’t just want an AI to say "This is a disease." You need to know "This is a disease because of this specific spot on the X-ray."

If the AI is looking at the wrong spot (the "sloppy student" problem), you can’t trust it. This paper shows that by building structural constraints into the AI’s brain—forcing it to look at the beak when it talks about the beak—we can make AI explanations auditable and trustworthy, without sacrificing accuracy. It turns the AI from a black box that guesses into a transparent expert that points to the evidence.

Technical Summary: Spatially Grounded Concept Bottleneck Models via Part-Factorized Attention

Problem Statement
Concept Bottleneck Models (CBMs) offer auditable decision-making by predicting human-named attributes before predicting a class. However, in fine-grained recognition tasks, standard CBMs suffer from a "spatial grounding gap." Because concept heads are free to attend to any region of the image, a head named for a specific body part (e.g., "throat color") may be satisfied by evidence from an unrelated region (e.g., a wing patch) if that patch correlates with the attribute. This renders the concept layer numerically interpretable but spatially ungrounded, preventing reliable test-time interventions based on anatomical evidence.

Methodology
The authors propose a Part-Factorized Concept Bottleneck Model (PF-CBM) built on a frozen DINOv3 vision transformer. The model enforces spatial grounding through four structural mechanisms:

Foreground Gating: A learned MLP acts as a foreground gate over DINOv3 patch features. It produces per-patch logits that suppress background regions. This gate is injected additively in log space into the attention mechanism, ensuring differentiability and avoiding the gradient discontinuities of hard masks.
Part-Factorized Attention with Gaussian Prior: The model utilizes $P=12$ learnable part queries that cross-attend to patch features. To prevent permutation symmetry (where queries collapse to a common location), a learnable 2D Gaussian spatial prior is added to the attention logits. The means of these Gaussians ( $\mu_p$ ) are initialized using dataset-average keypoint locations, requiring no per-image keypoint supervision during training or testing.
Slot-to-Part Routing: A fixed concept-to-part map routes each of the 293 part-specific attributes to read exclusively from the part token its name implies (e.g., "has_wing_color" reads only from the "wing" token). This structural constraint ensures that a concept head cannot attend to irrelevant anatomical regions.
Global Token Pathway: A separate global token attention pathway handles 19 whole-object attributes (e.g., size, shape). A single learnable global query cross-attends over all foreground patches without a spatial prior, as these attributes are not tied to specific anatomical regions.

Training Objective
The model is trained in two stages. Stage 1 pretrains the gate, part attention, and attribute heads using concept, foreground, alignment, and entropy losses. Stage 2 jointly trains the classifier and temperatures with a significantly lower learning rate for the grounding components to prevent classification pressure from overwriting the spatially aligned state. The foreground supervision can be derived from bounding boxes or, in a box-free setting, from a PCA-based foreground target derived from DINOv3 features.

Key Contributions

Structural Grounding: Unlike methods that use soft alignment or optimal transport plans (e.g., DOT-CBM), PF-CBM guarantees grounding by construction through fixed routing and spatial priors.
Supervision Efficiency: The model demonstrates that spatial grounding can be achieved with minimal supervision. Initializing the Gaussian prior requires only 0.5% of the training set (approximately 27 images) with keypoint annotations, with no measurable loss in performance.
Box-Free Operation: By combining a PCA foreground target with the Gaussian spatial prior, the model removes all per-image supervision (bounding boxes and keypoints) while maintaining high performance.

Results
Experiments on the CUB-200-2011 dataset show:

Classification Accuracy: The spatial-prior model achieves 89.01% top-1 accuracy, matching a fully supervised baseline (88.95%). The box-free variant (no bounding boxes, no per-image keypoints) achieves 88.77% top-1 accuracy.
Pointing Accuracy: The spatial-prior model raises pointing accuracy (the fraction of attention maps correctly localized to keypoints) from 36.4% (baseline) to 52.2%. The box-free variant further improves this to 69.5%.
Ablations: Removing the spatial prior entirely causes pointing accuracy to collapse to 2.9%, confirming the necessity of the prior for breaking permutation symmetry. Replacing ground-truth keypoints with pseudo-keypoints from k-means prototypes yields inferior results, indicating that PCA foreground discovery is sufficient for object localization but not for semantic part identification.
Comparison: PF-CBM outperforms DOT-CBM (85.39% top-1) while using a smaller backbone (ViT-B vs. ViT-L/14) and providing explicit per-part pointing metrics.

Significance and Limitations
The paper claims that PF-CBM provides a structurally guaranteed form of spatial grounding that is trivial to audit and supports predictable intervention behavior. It demonstrates that high-quality grounding can be achieved with sparse annotation, making it suitable for domains with limited labeling resources.

Limitations noted by the authors include:

The isotropic Gaussian prior does not model inter-part coupling, limiting performance on elongated parts or occlusion reasoning.
The method assumes a single object in a canonical pose, which may not generalize to scenes with arbitrary viewpoints or multiple objects without additional normalization.
Removing part identity entirely without any dataset-level spatial signal remains an unsolved challenge, as pseudo-keypoints are not reliably semantic.