Technical Summary: VIMCAN

Problem Statement

While deep learning has significantly advanced multimodal 3D human pose estimation (HPE), state-of-the-art pipelines face two critical limitations:

1. Depth Ambiguity: Vision-based methods struggle to lift 2D keypoints to 3D poses accurately due to the inherent ill-posed nature of the problem.
2. Computational Inefficiency: Current high-performing models rely heavily on Transformer architectures. The quadratic complexity ($O(L^2)$) of self-attention mechanisms regarding sequence length ($L$) leads to prohibitive memory usage and computational costs, making real-time processing of long sequences impractical, especially on resource-constrained devices.

While Mamba architectures offer linear complexity ($O(L)$) and efficient sequence modeling, they currently lag behind attention-based methods in capturing complex spatial dependencies required for precise multimodal fusion.

Methodology

The authors propose VIMCAN (Visual-Inertial Mamba-Cross-Attention Network), a hybrid architecture designed to balance efficiency, accuracy, and robustness by integrating RGB keypoints and wearable IMU data.

1. Hybrid Architecture

VIMCAN combines the strengths of two distinct mechanisms:

• Mamba (STMamba): Utilized for efficient temporal modeling and sequence processing. The network employs Spatio-Temporal Mamba (STMamba) modules to process visual and inertial features independently.
  • Skeleton-aware STMamba: Processes visual features ($F^V$) by reordering keypoints according to parent-child joint relationships to incorporate kinematic priors.
  • Part-aware STMamba: Processes inertial features ($F^I_g$) from specific body parts (e.g., torso, limbs), utilizing bidirectional spatio-temporal selective scans (SS2D) to aggregate information across spatial and temporal dimensions.
• Cross-Attention: Employed as the backbone for multimodal fusion. While Mamba handles temporal dynamics efficiently, Cross-Attention is used to capture intricate cross-modal spatial dependencies between visual and inertial tokens, addressing the spatial reasoning limitations of pure Mamba.

2. Feature Extraction and Fusion

• Input: The system takes $J=17$ 2D keypoints and $I=6$ IMU unit quaternions.
• Grouping: IMUs are partitioned into $G=5$ anatomical groups (torso, left/right arms, left/right legs) based on kinematic priors.
• Fusion Process:
  1. Features are lifted to a common embedding dimension via linear layers.
  2. Independent STMamba modules extract spatio-temporal features.
  3. Cross-Attention Module: Visual features act as queries ($Q$), while inertial features serve as keys ($K$) and values ($V$). This allows the model to dynamically attend to relevant inertial data for specific visual joints.
  4. The fused features undergo global modeling via a stack of skeleton-aware STMamba layers before pose regression.

3. Training Strategy

• Variable-Length Inference: Unlike conventional approaches requiring fixed-length inputs, VIMCAN natively supports variable-length sequences without padding or masking.
• Loss Function: A composite loss is used, comprising:
  • Mean Per-Joint Position Error (MPJPE).
  • Normalized MPJPE (N-MPJPE) to correct for scale drift.
  • Mean Per-Joint Velocity Error (MPJVE) for temporal smoothness.
  • Temporal Consistency (TC) loss to suppress abrupt fluctuations.

Key Contributions

1. Novel Hybrid Architecture: VIMCAN is the first work to apply Mamba to vision-inertial multimodal 3D HPE, successfully combining Mamba's linear-time sequence processing with Cross-Attention's robust spatial fusion capabilities.
2. Efficiency and Scalability: The architecture achieves real-time inference (>60 FPS) on consumer-grade hardware while supporting variable-length sequences, overcoming the quadratic complexity bottleneck of Transformers.
3. State-of-the-Art Performance: The method addresses depth ambiguity and computational complexity simultaneously, outperforming existing Transformer-based and other SOTA approaches.

Experimental Results

The authors evaluated VIMCAN on the TotalCapture and 3DPW datasets.

• TotalCapture:
  • With 6 IMUs and MediaPipe 2D detection, VIMCAN achieved an MPJPE of 33.2 mm and P-MPJPE of 25.7 mm, outperforming prior GCN-Transformer hybrids.
  • Using Ground-Truth 2D keypoints, VIMCAN reached an MPJPE of 17.2 mm and P-MPJPE of 13.8 mm, surpassing the previous best (Wang et al.) by 11.4 mm in MPJPE.
  • Efficiency: VIMCAN required only 29% of the memory footprint of the GCN-Transformer baseline while delivering 1.3× higher throughput.
• 3DPW:
  • VIMCAN achieved an MPJPE of 45.3 mm, outperforming previous methods (e.g., 53.9 mm for Wang et al., 60.3 mm for Liu et al.).
• Ablation Studies:
  • Removing the Cross-Attention module in favor of Cross-Mamba resulted in a significant performance drop (17.2 mm vs. 24.3 mm MPJPE), validating the necessity of attention for spatial fusion.
  • Anatomical grouping and skeleton-aware scanning were proven essential for accuracy.

Significance and Claims

The paper claims that VIMCAN establishes a new benchmark for efficient multimodal 3D HPE. By bridging the gap between the efficiency of Mamba and the spatial reasoning of Cross-Attention, the method enables:

• Real-time processing of long sequences on consumer hardware (exceeding 60 FPS).
• Superior accuracy in both seen and unseen subject scenarios.
• Robustness against depth ambiguity through effective visual-inertial fusion.

The authors conclude that VIMCAN offers a superior balance of accuracy and efficiency compared to RNN, CNN, and Transformer-based predecessors, making it highly suitable for real-time applications in motion capture, rehabilitation, and human-computer interaction. A noted limitation is the reliance on strict sensor calibration, which future work aims to address through adaptive alignment techniques.