MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization

Technical Summary: MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization

Problem Definition: Fusion Overfitting
The paper addresses Multimodal Domain Generalization (MMDG), the challenge of deploying multimodal models (e.g., audio-visual) in target environments where recording conditions differ from the training sources. The authors identify a specific failure mode in standard MMDG architectures, termed Fusion Overfitting.

In standard end-to-end training, separate encoders for each modality are optimized jointly with a fusion module to minimize a loss function (e.g., cross-entropy) on the fused output. The authors argue that this joint optimization causes encoders to exploit cross-modal co-occurrences—statistical relationships between modalities that arise from source-specific recording conditions (e.g., the precise synchronization of a knife sound and motion in a quiet kitchen) rather than learning domain-invariant features. Consequently, encoders produce features that are only useful when combined with the other modality's specific output. When deployed to a target domain with different conditions (e.g., a noisy kitchen), these learned co-occurrences break down, leading to performance degradation. Empirical analysis confirms that fusion training causes encoders to collapse into low-rank subspaces, discarding feature dimensions that could generalize independently.

Methodology: Modality-Entropy Regularization (MER-DG)
To counteract Fusion Overfitting, the authors propose MER-DG, an architecture-agnostic regularizer that maximizes the entropy of each encoder's feature distribution independently. The core insight is that maximizing entropy preserves feature diversity, ensuring that domain-invariant features are not discarded during the specialization process required for fusion.

The method decomposes the differential entropy maximization objective into two additive loss terms applied to the feature embeddings $Z$ of each modality:

1. Marginal-Entropy Loss ($L_{marg}$): This term prevents per-dimension collapse by enforcing a variance floor. It uses a hinge loss to ensure the standard deviation ($\sigma_d$) of each feature dimension remains above a target scale ($\gamma=1$), preventing dimensions from becoming inactive.
2. Spectral-Entropy Loss ($L_{spec}$): This term minimizes redundancy across dimensions by maximizing the log-determinant of the correlation matrix. It encourages features to be uncorrelated, spreading information across independent axes rather than collapsing into a low-dimensional subspace.

The total training objective combines the base task loss ($L_{base}$) with the sum of these regularization terms across all modalities:
$$L_{total} = L_{base} + \lambda \sum_{m=1}^{M} (\alpha_{marg} L_{marg}(Z^{(m)}) + \alpha_{spec} L_{spec}(Z^{(m)}))$$

Key Contributions

1. Identification of Fusion Overfitting: The paper formally identifies and analyzes the phenomenon where joint optimization in MMDG leads encoders to overfit to source-specific cross-modal co-occurrences at the expense of domain-invariant features.
2. MER-DG Proposal: The authors introduce a simple, additive regularizer that maximizes feature entropy. It is designed to be compatible with any multimodal framework containing separate encoders and a fusion module.
3. Empirical Validation of Mechanism: Through extensive analysis (cross-domain alignment, spectral rank, domain classification, and standalone encoder performance), the paper demonstrates that standard fusion training degrades encoder diversity and that MER-DG successfully restores it.
4. Performance Gains: The method is validated on EPIC-Kitchens and HAC benchmarks, showing consistent improvements over standard fusion and state-of-the-art (SOTA) MMDG methods.

Experimental Results
The authors evaluated MER-DG on four baselines (Standard Fusion, Cross-Modal Contrastive, SimMMDG, and CMRF) across multi-source and single-source settings.

• Multi-Source DG: MER-DG achieved average improvements of approximately 5% over standard fusion architectures and 2% over SOTA methods (SimMMDG and CMRF) on EPIC-Kitchens and HAC.
• Single-Source DG: Similar improvements were observed in the more challenging single-source setting, with gains up to 3.00% on SimMMDG for the HAC dataset.
• Standalone Encoder Recovery: A critical finding is that MER-DG restores the standalone performance of individual encoders to near-independent training levels. For example, on EPIC-Kitchens, the Video encoder's out-of-domain accuracy improved from 56.53% (standard fusion) to 58.61% (MER-DG), approaching the 59.46% achieved by an independently trained unimodal model. This confirms that the regularizer specifically counteracts the feature loss induced by fusion training.
• Ablation: Both the marginal and spectral loss terms were found to be necessary for optimal performance; using either alone yielded suboptimal results compared to the combined approach.

Significance and Claims
The paper claims that MER-DG addresses a fundamental limitation in multimodal learning: the tendency of joint optimization to discard domain-invariant information in favor of source-specific correlations. By preserving representational diversity through entropy maximization, MER-DG enables models to generalize robustly when cross-modal co-occurrences shift. The authors emphasize that the method requires minimal computational overhead, integrates seamlessly as an additive loss term, and does not rely on specific architectural changes, making it a practical solution for improving the robustness of multimodal systems in real-world deployment scenarios.