Technical Summary: MMDG-Bench

Problem Statement

Multimodal Domain Generalization (MMDG) aims to train models that generalize to unseen target domains using only labeled source data, leveraging complementary signals such as video, audio, text, and vibration. Despite a growing body of literature proposing specialized MMDG methods, the field lacks a consensus on whether reported performance gains reflect genuine algorithmic progress or are artifacts of inconsistent evaluation protocols. Current research is fragmented, varying significantly across datasets, modality configurations, and experimental settings. Furthermore, existing benchmarks predominantly focus on action recognition and standard accuracy, often neglecting critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness (misclassification and out-of-distribution detection). This lack of standardization obscures a reliable assessment of the field's advancement.

Methodology

To address these limitations, the authors introduce MMDG-Bench, the first unified and comprehensive benchmark for MMDG. The benchmark is designed to standardize evaluation across diverse tasks, modalities, and robustness axes.

Scope and Scale

• Datasets: Six datasets spanning three distinct task families:
  • Action Recognition: EPIC-Kitchens (eight classes across three kitchens) and HAC (Human, Animal, Cartoon). Modalities include Video, Audio, and Optical Flow.
  • Mechanical Fault Diagnosis: HUST Motor (four operating conditions). Modalities include Vibration and Acoustic signals.
  • Sentiment Analysis: CMU-MOSI, CMU-MOSEI, and CH-SIMS. Modalities include Video, Audio, and Text.
• Methods: Nine representative MMDG methods are evaluated alongside an Empirical Risk Minimization (ERM) baseline and an Oracle model (trained on target data):
  • Baselines: ERM.
  • Specialized Methods: RNA-Net (Relative Norm Alignment), SimMMDG (modality-shared/specific decomposition), MOOSA (masked translation and jigsaw puzzles), CMRF (cross-modal representation flattening), NEL (nonpolarized learning), JAT (joint adversarial training), MBCD (collaborative distillation), and GMP (gradient modulation).
• Experimental Protocol:
  • Settings: Both Multi-source (leave-one-domain-out) and Single-source (all source-target pairs) settings.
  • Hyperparameters: A rigorous search protocol involving default parameters plus 10 random-search trials per algorithm-dataset pair, with model selection based on training-domain validation.
  • Scale: A total of 7,402 neural networks were trained across 95 unique cross-domain tasks.

Evaluation Metrics

Beyond standard accuracy, MMDG-Bench systematically assesses:

1. Corruption Robustness: Performance under realistic perturbations (e.g., wind noise on audio, defocus blur on video).
2. Missing-Modality Generalization: Performance when modalities present during training are absent during inference.
3. Trustworthiness:
   • Misclassification Detection (MisD): Ability to identify incorrect predictions (AURC, AUROC, FPR95).
   • Out-of-Distribution (OOD) Detection: Ability to detect semantically novel inputs (AUROC, FPR95).

Key Results

The comprehensive evaluation yields five primary findings:

1. Marginal Gains over Baselines: Under fair, standardized comparisons, recent specialized MMDG methods offer only marginal improvements over strong baselines. In many cases, a carefully tuned ERM baseline matches or outperforms specialized approaches.
2. No Consistent Dominance: No single method consistently outperforms others across all datasets, modality combinations, or task families. Performance rankings fluctuate significantly; for instance, a method performing well on action recognition may rank lowest in fault diagnosis.
3. Large Gap to Oracle: A substantial performance gap persists between current MMDG methods and the Oracle model (trained on target data). For example, on the HAC dataset, the Oracle achieves ~92.8% accuracy, while the best MMDG method reaches only ~70.9%, indicating MMDG remains far from solved.
4. Ineffectiveness of Trimodal Fusion: Adding a third modality (e.g., Video+Audio+Flow) does not consistently outperform the strongest bimodal configurations. In several cases, adding a modality degrades performance, suggesting that modality competition is a significant bottleneck that current fusion strategies do not fully resolve.
5. Vulnerability to Real-World Challenges: All evaluated methods exhibit significant degradation under corruption and missing-modality scenarios. Crucially, clean benchmark performance does not reliably predict robustness; methods ranking highly on clean data often drop significantly under corruption. Furthermore, some methods that improve raw accuracy compromise model trustworthiness (e.g., confidence calibration).

Significance and Claims

The paper claims that MMDG-Bench provides a rigorous foundation for future research by exposing the limitations of current evaluation practices. The authors argue that progress in MMDG has been partially overestimated due to inconsistent protocols. By standardizing backbones, data splits, and hyperparameter searches, the benchmark isolates algorithmic contributions from evaluation artifacts.

The study concludes that evaluating clean cross-domain performance alone is insufficient for deploying robust multimodal systems. Future research must prioritize:

• Developing algorithms that offer genuine generalization beyond marginal gains over ERM.
• Addressing modality competition and adaptive fusion mechanisms.
• Building resilience to real-world corruptions and sensor failures.
• Jointly optimizing predictive accuracy and model trustworthiness (uncertainty calibration).

MMDG-Bench is released to enable reproducible, directly comparable evaluations, aiming to shift the field toward more reliable and deployment-ready multimodal systems.