Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies

Technical Summary: Cross-Modal Action Recognition in Egocentric Video Using Mamba

Problem Statement
Egocentric action recognition presents unique challenges distinct from fixed-camera scenarios, including erratic camera motion, frequent hand occlusion, and the difficulty of maintaining consistent visual representations over time. While Transformer-based architectures are well-suited for constructing strong spatio-temporal representations and handling long sequences, their self-attention mechanism scales quadratically with sequence length, limiting their feasibility for online video applications. Furthermore, while skeletal motion data can mitigate occlusion issues and provide clean motion cues, existing Mamba-based models have not yet integrated skeleton information with RGB streams within a unified framework.

Methodology
The authors propose a cross-modal architecture built upon State Space Models (SSMs), specifically leveraging the Mamba model for its linear time complexity. The architecture comprises three primary components:

1. Video Stream: A VideoMamba module extracts visual embeddings from RGB video frames.
2. Skeleton Stream: A dedicated encoder built on a stack of Mamba blocks processes temporal hand skeleton data.
3. Fusion Module: This component integrates the two modalities into a unified representation for classification.

A central methodological innovation is the design of four distinct strategies for CLS Token Mixing. In Mamba and Transformer models, the Class (CLS) token acts as an information sink, concentrating contextual information for final classification. The authors argue that discarding the pretrained CLS tokens from the unimodal encoders wastes learned representations. Instead, they propose initializing a new "Mixed CLS" token using the pretrained unimodal CLS tokens via the following strategies:

• Naive: Discards both pretrained CLS tokens and initializes a new, trainable token from scratch.
• Average: Initializes the Mixed CLS token as the element-wise average of the video and skeleton CLS tokens.
• Weighted: Computes a linear combination of the two tokens using a learnable scalar parameter $\alpha$ (derived via a sigmoid function): $CLS_{mix} = \alpha CLS_{video} + (1-\alpha) CLS_{skeleton}$.
• Context-based: Dynamically computes the weighting parameter $\alpha$ using a Mamba block that processes the concatenated non-CLS token sequences from both modalities, allowing the fusion weight to adapt based on input content.

Experimental Setup
The proposed architecture and fusion strategies were evaluated on the H2O dataset, an egocentric video dataset providing synchronized hand skeleton annotations and action labels. The experiments utilized two model configurations (Tiny and Small) with varying embedding dimensions and depths (24 Mamba blocks). The VideoMamba encoders were initialized with checkpoints pretrained on the Something-Something V2 dataset, while the skeleton encoder was trained directly on H2O.

Key Results
Experimental results on the H2O dataset demonstrate that integrating skeletal data significantly improves performance over video-only baselines, particularly when both modalities provide informative signals.

• Performance Gains: The Average strategy achieved the best performance. In the Tiny configuration, it yielded a 10.17% gain in Top-1 accuracy over the VideoMamba baseline. In the Small configuration, the gain was 25.49%.
• Strategy Comparison: The Weighted strategy performed comparably to the Average strategy, with learned parameters converging toward a balanced combination ($\alpha \approx 0.5$), effectively approximating the averaging approach.
• Baseline Performance: The Naive strategy, which discards pretrained CLS tokens, consistently underperformed, confirming the importance of preserving learned unimodal representations.
• Context-based Limitation: Contrary to expectations, the more flexible Context-based strategy did not outperform the simpler fusion methods. The authors note this may be due to architectural limitations or the scale of the dataset used.

Significance and Claims
The paper claims that combining visual and skeletal modalities within a Mamba framework consistently outperforms video-only baselines when both modalities carry meaningful information. The primary contribution is the systematic evaluation of CLS token fusion strategies, establishing that simple averaging of pretrained tokens is currently the most effective approach for this specific task and dataset.

The authors modestly frame the work as a step toward efficient, cross-modal egocentric understanding. They identify future directions, including evaluating these strategies on larger datasets to determine if the saturation of the Context-based strategy is a data or architectural constraint, and exploring the use of skeleton data as "privileged information" during training to improve visual-only inference at test time.