Multimodal Skeleton-Based Action Representation Learning via Decomposition and Composition

1. Problem Statement

The paper addresses the challenge of multimodal skeleton-based action recognition, specifically the trade-off between model efficiency and performance effectiveness.

• The Dilemma: Existing methods typically rely on late fusion (combining predictions from separate models for different modalities like joints, bones, and motion). While effective, this approach requires multiple backbone networks, leading to high computational overhead and parameter counts. Conversely, early fusion (combining raw data or embeddings before a single backbone) is computationally efficient but often fails to capture the distinct characteristics of each modality, resulting in suboptimal performance.
• The Gap: There is a lack of self-supervised frameworks that can efficiently learn rich multimodal representations using a single shared backbone without sacrificing the discriminative power of individual modalities.

2. Methodology: Decomposition and Composition (D&C)

The authors propose a self-supervised learning framework named Decomposition and Composition (D&C). The core architecture utilizes Embedding Fusion, where different modalities (Joint, Bone, Motion) are mapped to a unified embedding space and processed by a single shared encoder backbone (specifically, a two-stream Transformer network for spatial and temporal features).

The training strategy consists of three main components:

A. Decoupled Spatial-Temporal Encoding

Instead of treating the skeleton sequence as a monolithic block, the model employs a two-stream architecture:

• Temporal Stream: Processes the sequence by retaining the time dimension and flattening spatial dimensions.
• Spatial Stream: Processes the sequence by retaining the joint dimension and flattening the time dimension.
• This decoupling allows the model to learn specific spatial and temporal discriminative features separately before fusing them for the final representation.

B. Unimodal Feature Decomposition (UFD)

To ensure the unified multimodal features retain the specific information of each modality, the model performs decomposition:

• The unified fused embedding is passed through the encoder to produce a fused representation.
• This fused representation is then projected and reconstructed to match the features of individual unimodal branches (Joint, Bone, Motion).
• Loss Function: A Mean Squared Error (MSE) loss aligns the decomposed fused features with the ground-truth unimodal features. This forces the shared backbone to encode sufficient information to recover specific modality details, preventing information loss during fusion.

C. Multimodal Feature Composition (MFC)

While Decomposition ensures unimodal fidelity, it lacks direct optimization for the multimodal synergy. The Composition strategy addresses this:

• The model simulates a late fusion scenario by averaging the features of the individual unimodal branches (which are processed separately in this specific step) to create a "pseudo-multimodal" target.
• The unified fused representation (from the single backbone) is then aligned with this "pseudo-multimodal" target.
• Loss Function: An MSE loss minimizes the distance between the single-backbone fused features and the late-fused features. This acts as a self-supervised guide, teaching the efficient single-backbone model to mimic the performance of a more expensive late-fusion model.

D. Viewpoint-Invariant Training

To further enhance robustness, the authors introduce a viewpoint-invariant training strategy.

• Instead of relying solely on data augmentation, the model treats samples of the same action captured from different camera viewpoints as positive pairs.
• This encourages the model to learn features that are invariant to both data augmentation and camera perspective, leveraging the multi-view nature of datasets like NTU RGB+D.

3. Key Contributions

1. Efficient Multimodal Framework: The paper is one of the few works to study efficient multimodal skeleton-based action representation learning, achieving high performance with a single shared backbone rather than multiple independent networks.
2. Decomposition and Composition Strategy: A novel self-supervised training method that:
   • Decomposition: Guarantees rich unimodal information is preserved within the fused features.
   • Composition: Uses late-fusion features as self-supervised guidance to enhance multimodal representation learning.
3. Spatial-Temporal Decoupling: The integration of a two-stream learning framework tailored for skeleton data, refining both spatial and temporal feature learning.
4. State-of-the-Art Performance: The method achieves superior results across multiple benchmarks while maintaining low computational costs (FLOPs and inference time).

4. Experimental Results

The method was evaluated on three major datasets: NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD II.

• Action Recognition:

  • The proposed method outperforms existing state-of-the-art (SOTA) methods (e.g., UmURL, CMD, ActCLR) across all evaluation protocols (Cross-Subject, Cross-View, Cross-Setup).
  • Notably, on the complex NTU-120 dataset, the method achieves 91.8% accuracy (Cross-View) using Joint+Bone+Motion modalities, surpassing the previous best by a significant margin.
  • Efficiency: Despite using a Transformer backbone, the inference speed is faster than GCN-based methods due to parallelism, and it is significantly more efficient than late-fusion methods that require multiple backbones.

• Action Retrieval:

  • The method achieved the best performance in KNN-based action retrieval tasks, demonstrating high-quality feature clustering.
  • The single-modality (Joint) performance of the proposed method reached the level of multimodal performance in baseline methods, highlighting the effectiveness of the viewpoint-invariant learning.

• Semi-Supervised Learning:

  • In scenarios with limited labeled data (1% and 5% of training data), the method significantly outperformed fine-tuned baselines, proving the robustness of the learned representations.

• Transfer Learning:

  • Models pre-trained on NTU datasets and fine-tuned on PKU-MMD II showed superior transferability compared to other methods.

• Ablation Studies:

  • Confirmed that both Decomposition and Composition are necessary; using only one yields suboptimal results.
  • Demonstrated that adding explicit inter-modal consistency loss is unnecessary when Decomposition is used, as it implicitly handles alignment.
  • Showed that Viewpoint-Invariant training improves accuracy by approximately 1%.

5. Significance

This work provides a critical solution to the efficiency-accuracy trade-off in multimodal skeleton-based action recognition. By proving that a single backbone can learn representations as effective as (or better than) complex late-fusion ensembles through the Decomposition and Composition paradigm, the paper offers a scalable path for deploying action recognition systems in resource-constrained environments. The introduction of viewpoint-invariant learning further bridges the gap between laboratory datasets and real-world applicability, making the approach highly relevant for practical computer vision applications.