Technical Summary: ADMC – Attention-based Diffusion Model for Missing Modalities Completion

Problem Statement

Multimodal emotion and intent recognition is critical for automated human-computer interaction, relying on the fusion of speech, text, and visual data. However, real-world applications frequently encounter missing modalities due to sensor malfunctions, incomplete data collection, speech recognition errors, user silence, or visual occlusion.

Existing approaches to this problem face two primary limitations:

1. Over-coupling: Traditional methods often jointly train feature extraction and classification networks. This creates strong dependencies between modalities, causing a sharp decline in performance when specific modalities are missing.
2. Imprecise Generation: Current reconstruction techniques (e.g., simple zero-padding, low-rank imputation, or standard autoencoders) often fail to generate missing features that align with the authentic multimodal feature distribution, leading to suboptimal recognition outcomes.

Methodology

The authors propose ADMC (Attention-based Diffusion model for Missing modalities feature Completion), a framework designed to decouple feature extraction from generation and classification to preserve modality-specific characteristics while accurately reconstructing missing data. The method operates in three distinct stages:

1. Independent Modality-Specific Feature Extraction

To prevent over-coupling, ADMC independently trains feature extraction networks for each modality (Acoustic, Visual, Textual) using single-modality classification tasks.

• Mechanism: Networks ($Enc_a$, $Enc_v$, $Enc_t$) are trained separately and fixed after training.
• Goal: This preserves the unique characteristics of each modality, ensuring that the representation of an available modality remains robust even when others are absent.

2. Attention-based Diffusion Network (ADN)

The core generative component is the ADN, which operates in the latent space of full-modality features to generate missing modality features.

• Forward Process: Multimodal features are gradually corrupted by adding Gaussian noise over $N$ steps. A multi-head self-attention network predicts the added noise at each step to capture inter-modal dependencies.
• Reverse Process: Starting from noise, the network denoises to reconstruct the full feature set. Crucially, during inference, the known (available) modalities are masked and preserved ($F_o$), while the missing parts are generated ($F_{tva}$).
• Equation: The reverse step updates the latent representation by combining the denoised output with the ground-truth available modalities using a mask $m$:
  $$F^{n-1}_{tva} = m \odot F^{n-1}_{o} + (1 - m) \odot F^{n-1}_{tva}$$

3. Modality Completion and Enhancement

The framework addresses two scenarios:

• Missing Modality Scenario: Missing features are replaced with Gaussian noise and reconstructed via the ADN.
• Full-Modality Scenario (MMER): The authors introduce a MultiModal Enhancement Recognition (MMER) method. Even when all modalities are present, the ADN is used to generate cross-modal features (e.g., generating text features from audio and visual inputs). These generated features are concatenated with original features to provide supplemental cues, enhancing the final recognition.

4. Multimodal Feature Fusion (MF)

A separate Multimodal Fusion network, trained specifically for each missing-modality scenario, aggregates the features. It employs a self-attention mechanism to calculate modality weights and performs weighted aggregation followed by mean pooling to produce the final classification prediction.

Key Contributions

1. Novel Framework: Introduction of ADMC, which integrates independently trained feature extraction networks with an attention-based diffusion model to handle missing modalities.
2. ADN Architecture: Proposal of an Attention-based Diffusion Network that captures inter-modal dependencies via self-attention, generating missing features that closely align with authentic multimodal distributions.
3. Dual-Scenario Performance: Demonstration that the method excels not only in missing-modality scenarios but also enhances performance in full-modality contexts through cross-modal generation (MMER).
4. State-of-the-Art Results: Validation on standard benchmarks showing significant improvements over existing methods.

Experimental Results

The method was evaluated on the IEMOCAP (emotion recognition) and MIntRec (intent recognition) datasets.

• Performance Gains:
  • IEMOCAP: Achieved an average improvement of 1.9% in Weighted Accuracy (WA) over previous state-of-the-art methods.
  • MIntRec: Achieved an average improvement of 9.4% in WA.
• Comparison: ADMC outperformed methods based on Autoencoders (CRA), Cycle Consistency (MMIN, IF-MMIN), and other diffusion-based approaches (IMDer).
• Ablation Studies:
  • Removing independent training (w/o ITFN) caused performance drops, confirming that joint training leads to over-coupling.
  • Removing the diffusion network (w/o ADN) and using zero-padding resulted in significant degradation, proving the necessity of generative reconstruction.
  • Replacing the self-attention mechanism with a standard U-Net (w/ U-Net) yielded lower performance, highlighting the importance of explicit cross-modal dependency modeling.
• Robustness: Under varying missing rates ($p \in 0.1$ to $0.5$), ADMC maintained the highest performance and suffered the least degradation compared to baselines.
• Efficiency: While diffusion models add inference time, the authors note that with 1,000 steps, the latency is ~50ms per sample on an RTX 4090. They suggest that acceleration techniques (e.g., DDIM) could reduce steps to 50–100 with negligible performance loss.

Significance and Claims

The paper claims that ADMC addresses the fundamental issue of over-coupling in multimodal learning by decoupling feature extraction from the generation and fusion processes. By preserving the unique characteristics of individual modalities and using a diffusion model guided by self-attention to reconstruct missing data, the framework achieves robust performance across both incomplete and complete data scenarios.

The authors emphasize that their approach provides a state-of-the-art solution for missing modality completion, offering a practical path forward for automated human-computer interaction systems operating in real-world, imperfect data environments. They also include a responsible innovation statement, acknowledging potential biases in emotion recognition and positioning the tool as a decision-support system requiring human oversight.