Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition

1. Problem Statement

The paper addresses the challenge of Driver Action Recognition (DAR) within smart car cabins. This task is distinct from general action recognition due to:

• Constrained Environments: Limited body movement, subtle local motions (e.g., hand gestures), and fixed backgrounds.
• Input-Dependent Reliability: Different sensor modalities (RGB, Infrared, Depth) vary in reliability depending on lighting conditions, occlusion, and specific action types.
• Limitations of Existing Methods: Current multimodal approaches rely on fixed fusion modules or predefined cross-modal interactions. These static mechanisms fail to adapt when the reliability of a specific modality changes between samples. Furthermore, standard Transformer backbones often rely on coarse contextual patterns, missing the fine-grained spatio-temporal cues necessary to distinguish subtle driver actions.

2. Methodology

The authors propose a novel framework combining Mixture-of-Modality-Experts (MoME) with a Holistic Token Learning (HTL) strategy.

A. Mixture-of-Modality-Experts (MoME)

Instead of static fusion, MoME treats each modality as a specialized "expert" network.

• Dynamic Gating: Unlike traditional Mixture-of-Experts (MoE) that route based solely on input features, MoME employs an output-aware gating mechanism. The gate analyzes the class tokens ($T_{cls}$) generated by each expert before fusion.
• Adaptive Collaboration: This allows the system to dynamically weigh experts based on their predictive confidence and action-specific response patterns for a given input, rather than relying on fixed weights learned during training.
• Fusion: The final embedding is a weighted sum of all expert outputs, distributed by the gating mechanism to ensure robust decision-making.

B. Holistic Token Learning (HTL)

To address the lack of fine-grained detail in standard Transformers, HTL introduces a unified token-level learning strategy involving two levels of knowledge transfer:

1. Intra-Expert Self-Guidance (Deep-to-Shallow):
   • Uses KL Divergence to align deeper, more mature semantic representations (class tokens) with shallower spatio-temporal tokens within the same expert.
   • Goal: Encourages early layers to preserve subtle motion cues and reduces over-reliance on coarse background patterns.
2. Inter-Expert Mutual Guidance:
   • Uses Mean Squared Error (MSE) to align the class tokens of different experts.
   • Goal: Facilitates knowledge transfer across modalities (e.g., from RGB to Depth) while preserving the specialization of each expert. This creates a consensus on high-level semantics without collapsing modality diversity.

Loss Function:
The total loss combines standard classification loss ($L_{cls}$), intra-expert refinement ($L_{intra}$), and inter-expert consistency ($L_{inter}$):
$$L = L_{cls} + \alpha L_{intra} + \beta L_{inter}$$

3. Key Contributions

1. MoME Framework: The first application of a Mixture-of-Experts architecture specifically for multimodal driver action recognition, utilizing an output-aware gating mechanism to dynamically coordinate modality experts based on their internal representations.
2. Holistic Token Learning (HTL): A novel strategy that extends beyond simple class-token distillation. It simultaneously refines class tokens (high-level semantics) and spatio-temporal tokens (low-level motion details) through both self-guidance and mutual guidance.
3. Fine-Grained Focus: The method specifically targets the subtle, fine-grained motion patterns typical of in-cabin environments, which are often overlooked by standard pre-trained backbones.

4. Experimental Results

The framework was validated on the Drive&Act dataset, a large-scale multimodal benchmark containing RGB, Infrared (IR), and Depth data.

• Performance:
  • The proposed MoME + HTL achieved 73.82% Mean-1 Accuracy and 83.94% Top-1 Accuracy.
  • It outperformed state-of-the-art baselines, including CM2-Net (83.51% Top-1) and Multifuser (82.39% Top-1).
  • Notably, the improvement in Mean-1 Accuracy (+1.72% over CM2-Net) indicates the model is particularly effective at recognizing difficult or less frequent classes, not just dominant categories.
• Ablation Studies:
  • MoME vs. MoE: Output-aware gating (MoME) outperformed standard input-based gating (MoE), validating the importance of using expert outputs for routing decisions.
  • HTL vs. MoDE: HTL significantly outperformed the existing MoDE (Mixture-of-Distilled-Experts) framework. This confirms that relying solely on class-token distillation is insufficient; refining spatio-temporal tokens is crucial for fine-grained tasks.
  • Single-Modality Adaptation: Even when applied to single-modality models, HTL outperformed traditional deep supervision and self-distillation methods, proving its generalizability.
• Visualization: Attention maps showed that HTL-enhanced models focus more selectively on subtle action regions (e.g., hands) rather than diffuse background noise.

5. Significance

This paper makes a significant contribution to Multimodal Visual Analytics and Intelligent Transportation Systems (ITS) by:

• Solving the Reliability Problem: It provides a robust solution for scenarios where sensor data quality fluctuates (e.g., low light), dynamically shifting reliance to the most reliable modality.
• Enhancing Interpretability: By refining spatio-temporal tokens, the model offers better interpretability, clearly identifying where and when subtle actions occur.
• Advancing Human-Vehicle Interaction: Improved fine-grained recognition enables safer and more intelligent interventions in smart cabins, moving beyond simple fatigue detection to comprehensive activity understanding.

In summary, the authors successfully demonstrate that combining adaptive expert collaboration with comprehensive token-level knowledge transfer is the key to unlocking robust, fine-grained understanding in complex, constrained multimodal environments.