Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders

1. Problem Statement

While Sparse Autoencoders (SAEs) have successfully decomposed language models and static image encoders into interpretable, monosemantic features, their application to video representations remains underexplored. The core challenge identified by the authors is temporal incoherence:

• Structural Mismatch: Standard SAEs treat video patches as independent tokens (flattening $T \times P$ into a 1D sequence). This ignores the spatio-temporal structure where the same semantic concept (e.g., a hand) should activate consistent features across consecutive frames.
• The Failure Mode: When applied to video, standard SAEs using hard Top-$K$ selection produce features that "flicker" across frames. The same spatial location may activate entirely different features in adjacent frames, destroying temporal coherence.
• Quantitative Impact: The authors found that standard SAEs reduce the lag-1 autocorrelation of features by 36% compared to raw features, making it impossible to reliably track concepts over time.

2. Methodology

The paper proposes a systematic framework for training SAEs on video that explicitly accounts for spatio-temporal structure.

A. Baseline Setup

• Backbones: DINOv2 (image-pretrained) and VideoMAE (video-pretrained).
• Datasets: Something-Something v2 (SSv2) and Kinetics-400 (K400).
• Architecture: Top-$K$ SAEs with an expansion factor of 8 ($H=6,144$) and $K=64$ active features.

B. Proposed Spatio-Temporal Variants

To recover temporal coherence without sacrificing interpretability, the authors introduce three contrastive variants and a hierarchical grouping mechanism:

1. Temporal Contrastive (T-SAE): Extends the Temporal SAE concept from text to video. It applies an InfoNCE loss encouraging consistent activations for the same spatial position across consecutive frames ($t$ and $t+1$).
2. Separate Contrastive (ST-SAE): Adds a spatial contrastive loss between horizontally adjacent patches within a frame, alongside the temporal loss. This allows independent control over spatial and temporal coherence.
3. Raster-Scan Contrastive (R-SAE): Serializes the 2D grid of patches (row-by-row, frame-by-frame) into a 1D sequence. A single contrastive loss is applied to adjacent elements in this sequence, naturally unifying spatial and temporal neighbors.
4. Matryoshka Hierarchical Grouping (+M): Partitions the dictionary into a high-level group (20% of features) and a low-level group (80%). The high-level group is trained to be self-sufficient via an auxiliary reconstruction loss, aiming to concentrate action-relevant semantic features into a compact subset.

C. The Control Mechanism

The authors identify a tunable trade-off controlled by the contrastive strength (loss coefficient $\lambda$ or temperature $\tau$):

• Low Weight: Prioritizes reconstruction fidelity ($R^2$).
• High Weight: Prioritizes temporal coherence and action discrimination, often at the cost of reconstruction quality.

3. Key Contributions

1. Systematic Empirical Study: The first comprehensive analysis of SAEs on video, demonstrating that they can decompose video into interpretable concepts (scene elements, objects, actions).
2. Diagnosis of Temporal Incoherence: Identified that hard Top-$K$ selection is the primary cause of temporal instability. They showed that simple post-hoc smoothing (e.g., EMA) recovers coherence but destroys discriminative power, whereas learned contrastive objectives improve both simultaneously.
3. Correction of Evaluation Artifacts: Revealed that standard "Monosemanticity Score" (MS) metrics suffer from backbone-alignment artifacts. When evaluated with a neutral backbone (CLIP), DINOv2 and VideoMAE produce equally monosemantic features ($MS \approx 0.68$), contradicting previous findings that suggested VideoMAE was inherently more interpretable.
4. Controllable Pareto Frontier: Established that contrastive strength is a single axis that allows practitioners to navigate a trade-off between reconstruction, coherence, and interpretability.

4. Key Results

• Temporal Coherence:
  • Standard SAEs drop lag-1 autocorrelation to 0.277 (from raw 0.435).
  • The Temporal+M variant with high contrastive weight ($\tau=0.2$) not only recovers coherence but exceeds raw features (Lag-1 = 0.462 vs. 0.435).
• Action Discrimination:
  • SAE features improve action classification accuracy by +3.9% over raw features.
  • Causal Ablation: Contrastive training concentrates predictive signals. Removing just 10 features from a Temporal SAE caused a 9.3 percentage point drop in accuracy, compared to a much smaller drop for standard SAEs, proving the features are functionally critical and identifiable.
• Text-Video Retrieval:
  • SAE features significantly boost retrieval performance. On DINOv2, R@1 improved by 2.8× (from 4.6% to 13.0%).
  • This improvement persists even when controlling for dimensionality (using reconstructed 768-d features), confirming that sparse decomposition itself aligns visual and textual semantics better.
• Matryoshka Efficiency:
  • The +M variants concentrate action-relevant information into the top 20% of the dictionary, allowing for efficient feature usage without significant performance loss.

5. Significance and Implications

• Mechanistic Interpretability for Video: This work provides the first tools to "read" what video models are learning internally, moving beyond static image analysis to dynamic, temporal understanding.
• Design Principle: It establishes that contrastive strength is the primary hyperparameter for tuning SAEs in video contexts. Practitioners can now select an operating point based on their specific need (e.g., high reconstruction for compression, high coherence for tracking).
• Methodological Rigor: The paper highlights the importance of using neutral backbones for evaluation metrics to avoid alignment artifacts, a finding that likely impacts how interpretability is measured across different vision models.
• Future Directions: The results suggest that hybrid approaches (combining soft sparsity with hard floors) and multi-resolution decompositions are promising next steps for video feature analysis.

In summary, the paper demonstrates that while standard SAEs fail to preserve the temporal structure of video, Spatio-Temporal SAEs with contrastive objectives can recover and even enhance temporal coherence, leading to more interpretable, stable, and discriminative video representations.