💻 computer science

Concepts in Motion: Temporal Concept Bottleneck Model for Interpretable Video Classification

This paper introduces MoTIF, a transformer-based framework that enhances interpretable video classification by leveraging a class-conditioned VLM to automatically discover temporal concepts and modeling their dynamic patterns through per-concept self-attention, thereby bridging the performance gap between interpretable models and black-box video baselines.

Original authors: Patrick Knab, Sascha Marton, Philipp J. Schubert, Drago Guggiana, Christian Bartelt

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Patrick Knab, Sascha Marton, Philipp J. Schubert, Drago Guggiana, Christian Bartelt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to recognize what's happening in a video, like someone shooting a bow and arrow. Most modern AI models are like super-smart but silent magicians. They get the answer right (they say "Yes, that's archery!"), but if you ask them why, they just stare back with a black box. You can't see their thought process.

This paper introduces a new system called MoTIF (Moving Temporal Interpretable Framework). Think of MoTIF not as a magician, but as a very organized detective who solves crimes by checking off a specific list of clues, one by one, while keeping a diary of when those clues appeared.

Here is how it works, broken down into simple parts:

1. The "Clue List" (Concepts)

Instead of looking at the video as a blurry mess of pixels, MoTIF breaks the video down into a list of human-understandable "concepts."

The Old Way: A standard AI looks at the whole video at once and guesses.
The MoTIF Way: It asks, "Do I see a bow? Do I see an arrow? Do I see someone mounting the horse? Do I see them shooting?"
The Magic Trick: The paper uses a special "smart assistant" (a Vision-Language Model) to automatically generate this list of clues from the training videos. It doesn't need a human to write the list; the AI figures out that "bow" and "shoot" are the important words to look for.

2. The "Diary" (Time Matters)

This is the most important part. In a photo, a bow is just a bow. In a video, time is everything.

If you see a bow, then a person, then a shooting motion, that's archery.
If you see a shooting motion, then a bow, then a person, that's weird and probably not archery.

Most AI models mix all these clues together into a big smoothie, losing the order. MoTIF is different. It keeps a separate diary for each clue.

It has a "Bow Diary" that tracks when the bow appears.
It has a "Shoot Diary" that tracks when the shooting happens.
It has a "Mount Diary" for the horse.

It uses a special "attention" mechanism (think of it as a spotlight) that only looks at the Bow Diary to decide when the bow is important, and the Shoot Diary for the shooting. It never mixes the diaries up. This ensures that if the AI says "Bow," you know exactly when in the video it saw the bow.

3. The "Verdict" (Prediction)

Once the diaries are filled out, MoTIF combines the notes.

It looks at the "Bow" entry: "Appeared at 2 seconds."
It looks at the "Shoot" entry: "Appeared at 5 seconds."
It combines these with a mathematical formula (Log-Sum-Exp pooling) to make a final decision: "Archery."

Because it kept the diaries separate, it can tell you exactly why it made that choice. It can say, "I chose Archery because I saw a bow at 2 seconds and a shooting motion at 5 seconds."

4. The "What-If" Test (Intervention)

The paper shows that because the system is so transparent, you can actually edit its mind to fix mistakes.

Scenario: The AI sees a video of someone sitting in a barber chair and mistakenly thinks, "Shaving a beard."
The Fix: The researchers can manually turn off the "Barber Chair" clue in the system.
The Result: The AI immediately changes its mind and says, "Oh, wait, without the chair, this is actually 'Applying Lipstick'!"
This proves the system isn't just guessing; it's actually relying on the specific clues you can see and touch.

Why is this a big deal?

The authors tested MoTIF on several video datasets (like people doing breakfast tasks, sports, and random actions).

Accuracy: It performs almost as well as the "black box" models that no one can understand.
Interpretability: Unlike the black boxes, MoTIF can show you a map of when it saw the bow, the horse, or the arrow.
The Trade-off: The paper admits that keeping the diaries strictly separate (so you can trust the clues) sometimes makes the AI slightly less accurate than if it were allowed to mix the clues together. However, they found that this "mixing" makes the AI harder to understand, so they chose to keep the diaries separate to prioritize trust.

In Summary

MoTIF is like a video detective that solves crimes by checking off a list of clues (concepts) in a specific order (time). It doesn't just guess; it keeps a record of exactly when it saw the bow, the arrow, and the horse, allowing humans to look over its shoulder and understand exactly how it reached its conclusion. It bridges the gap between "smart but mysterious" AI and "smart and explainable" AI.

Technical Summary: MoTIF – Temporal Concept Bottleneck Model for Interpretable Video Classification

Problem Statement

While Concept Bottleneck Models (CBMs) have successfully enabled interpretable image classification by structuring predictions around human-understandable concepts, extending this paradigm to video remains challenging. Unlike static images, video data contains a temporal component where concepts evolve over time, and many actions cannot be inferred from a single frame. Existing approaches often struggle with two main issues:

Temporal Modeling vs. Interpretability: Standard transformers excel at capturing long-range dependencies but employ dense temporal feature mixing that obscures concept-level attributions, making them unsuitable for interpretable CBM modeling.
Concept Extraction: Manually annotating concepts for video is labor-intensive. While Vision-Language Models (VLMs) offer a path for automatic discovery, extending this to capture temporally grounded, class-conditioned concepts without manual intervention is non-trivial.

The core challenge is to design an architecture that handles variable-length video inputs and models temporal dynamics while preserving strict concept-level interpretability and avoiding the "black box" nature of standard deep learning models.

Methodology: MoTIF

The authors introduce MoTIF (Moving Temporal Interpretable Framework), a transformer-based CBM tailored for video classification. The framework operates on sequences of window-level concept activations and consists of three primary components:

1. VLM-Based Concept Discovery Pipeline

Instead of relying on predefined concept vocabularies, MoTIF employs a task-agnostic discovery stage:

Process: A small subset of training videos is sampled per class. A Vision-Language Model (VLM), conditioned on the class label, generates natural-language candidates for objects and actions within each temporal window.
Filtering: These candidates are embedded using CLIP-based models. Near-duplicates are removed via cosine similarity filtering (threshold > 0.9), resulting in a diverse, global concept bank $C$ .
Activation: For each video window, visual embeddings are compared against the concept bank to generate a concept activation matrix $X \in \mathbb{R}^{T \times K}$ , where $T$ is the number of windows and $K$ is the number of concepts.

2. Per-Channel Temporal Self-Attention (Diagonal Attention)

The core architectural innovation is a temporal module that isolates reasoning for each concept:

Diagonal Structure: Unlike standard transformers that mix channels via full linear projections ( $W_Q \in \mathbb{R}^{C \times C}$ ), MoTIF uses depthwise 1 $\times$ 1 convolutions to generate Query, Key, and Value projections for each concept channel independently.
Mechanism: Each concept channel $c$ computes its own attention scores $W_{c,t,u}$ over the temporal sequence. This ensures that evidence from one concept cannot flow into another, preserving strict concept attribution.
Output: The output is a weighted sum of values for each concept, maintaining the independence of concept channels while modeling when and how concepts recur over time.

3. Classification and Explanation

Affine Transformation: Activations undergo a non-negative affine transformation ( $\gamma_c X + \delta_c$ ) followed by a Softplus nonlinearity to ensure non-negative, differentiable activations.
Pooling: A Log-Sum-Exp (LSE) pooling mechanism aggregates per-time-step logits into a video-level prediction. This serves as a soft approximation of max-pooling and provides a time-importance profile.
Three-Mode Explanation: MoTIF decomposes predictions into:
1. Global Concepts: Aggregated relevance over the full video.
2. Local Concepts: Relevance within specific high-weight temporal windows.
3. Temporal Dependencies: Attention maps showing how a concept channel distributes focus across time (e.g., identifying temporal anchors or persistent presence).

Key Contributions

MoTIF Architecture: A Video CBM utilizing per-channel temporal self-attention that models temporal dynamics at the individual concept level while preserving concept independence.
Multi-Granular Explanation: The framework provides three complementary explanation modes (global, local, and temporal dependency maps) to trace decision-making across time.
VLM-Based Discovery: A pipeline that extracts object- and action-centric textual concepts directly from training videos using weak supervision from class labels, eliminating the need for manual annotation and improving concept coverage.

Experimental Results

The authors evaluated MoTIF on four benchmarks: Breakfast Actions, HMDB51, UCF101, and Something-Something V2 (SSv2).

Performance vs. Global CBM: MoTIF consistently outperforms a Global CBM (which mean-pools window representations before classification). For instance, on Breakfast Actions, MoTIF improved accuracy by 11–18 percentage points over the Global CBM, demonstrating the value of temporally localized concept modeling.
Performance vs. Black-Box Models: While a gap remains with state-of-the-art black-box models (e.g., VideoMAE V2), MoTIF significantly narrows this gap while retaining interpretability. On SSv2, the space-time variant (MoTIF-ST) achieved 41.9% accuracy compared to 33.6% for the Global CBM.
Temporal Sensitivity: Ablation studies using synthetic benchmarks and window shuffling confirmed that MoTIF relies on temporal order rather than static appearance cues, particularly on datasets like SSv2 where relational labels are critical.
Intervention Analysis: Destructive interventions (zeroing top concepts) significantly reduced prediction overlap, while corrective interventions (editing concepts) successfully repaired misclassifications, validating the causal link between concepts and predictions.
Trade-offs: The authors note a controllable trade-off: enforcing diagonal attention preserves interpretability but may slightly reduce accuracy on highly complex datasets compared to full attention variants. However, full attention entangles channels, reducing interpretability.

Significance and Claims

The paper positions MoTIF as a principled framework for learning and explaining temporal concept representations. Its significance lies in:

Bridging the Gap: It demonstrates that temporal reasoning can be integrated into concept bottleneck models without discarding concept-level structure, offering a viable alternative to opaque black-box video classifiers.
Interpretability without Sacrifice: By isolating temporal reasoning per concept, MoTIF allows users to trace errors back to missing or ambiguous temporal concepts rather than opaque feature interactions.
Modularity: The framework is compatible with various embedding backbones (CLIP, SigLIP, Perception Encoders) and concept discovery methods (textual or visual), suggesting that improvements in concept quality and architectural scaling are complementary paths to better performance.

The authors conclude that while performance depends heavily on concept quality, MoTIF provides a robust, interpretable foundation for video understanding that achieves state-of-the-art performance among CBMs while remaining competitive with non-interpretable baselines.