Lightweight Prompt-Based Enhancement for Missing-Modality Multimodal Sentiment Analysis

Problem Statement

Multimodal Sentiment Analysis (MSA) integrates textual, acoustic, and visual cues but often suffers performance degradation in real-world scenarios due to sensor failures, privacy constraints, or bandwidth limitations that cause modality missing. While recent prompt-learning-based methods (e.g., MPLMM) offer parameter-efficient solutions for reconstructing missing modalities, the authors identify three fundamental limitations in existing approaches:

1. Over-smoothing of Features: Cross-modal attention mechanisms act as low-pass filters, attenuating high-frequency cues (e.g., acoustic prosody, facial micro-expressions) vital for emotion discrimination. This affects both generated and authentic features traversing frozen attention layers.
2. Reliability-Blind Fusion: Current strategies often apply equal-weight fusion to authentic and generated modalities. This ignores the intrinsic reliability gap, where fusing low-quality generated features with high-fidelity authentic ones dilutes the signal-to-noise ratio (SNR).
3. Distribution Shift in Frozen Backbones: Methods that keep the Transformer backbone frozen to preserve pre-trained alignment cannot accommodate distribution shifts between generative-prompt outputs and authentic features. Simple fine-tuning of the backbone has been shown to disrupt pre-trained alignment, leading to performance drops.

Methodology: AFEP-RAF-GDA

The authors propose a lightweight enhancement framework built atop the MPLMM baseline, comprising three synergistic modules that operate on a frozen backbone. The total addition is approximately 52.6K trainable parameters (~1.05% of the backbone).

1. Authentic Feature Enhancement Pipeline (AFEP)

• Goal: Recover fine-grained, high-frequency details lost in authentic features due to over-smoothing.
• Mechanism:
  • Detail Injection Enhancement (DIE): Uses a bottleneck Multi-Layer Perceptron (MLP) to extract residual detail information from over-smoothed features.
  • Detail Gating Mechanism (DGM): Employs a sigmoid gate to adaptively control the injection amount of enhanced details, initialized conservatively to ensure training stability.
  • Generated Modality Suppression (GMS): Ensures enhancement is applied only to authentic modalities (reliability indicator $r_m=1$), leaving pseudo-features of missing modalities unchanged.

2. Reliability-Aware Fusion (RAF)

• Goal: Dynamically adjust fusion weights based on modality reliability to amplify authentic signals and suppress generated noise.
• Mechanism:
  • Uses binary reliability signals ($r_m \in \{0,1\}$) of other modalities as input.
  • Applies Feature-wise Linear Modulation (FiLM) via an MLP to generate scaling ($\gamma$) and shifting ($\beta$) parameters.
  • Modulates features via channel-wise affine transformations ($h''_m = (1+\gamma_m) \odot h'_m + \beta_m$) before concatenation, avoiding the need for explicit weight normalization.

3. Content-Guided Distribution Adapter (GDA)

• Goal: Correct distribution shifts in generated features before they enter the frozen cross-modal attention layers, without disrupting pre-trained knowledge.
• Mechanism:
  • Guidance Signal Construction: Constructs a signal by mean-pooling projected features of available (authentic) modalities and concatenating them with their reliability indicators.
  • FiLM Conditional Correction: Uses a FiLM network to generate scaling ($\lambda$) and shifting ($\delta$) parameters based on the continuous semantic content of authentic modalities (unlike RAF's binary signals). This corrects variance compression and mean shifts.
  • Missing-Aware Gating: Ensures correction is applied only to missing modalities.
  • Initialization: Uses zero initialization for the final layer weights to ensure identity mapping at the start of training, preserving pre-trained alignment.

Key Contributions

1. Systematic Problem Analysis: The paper identifies and addresses over-smoothing, reliability blindness, and distribution shifts from the perspectives of signal processing, information theory, and distribution adaptation.
2. Novel Modules:
   • AFEP: Recovers high-frequency details from authentic features via residual learning.
   • RAF: Achieves reliability-aware fusion using FiLM conditioning on modality source reliability.
   • GDA: Corrects distribution shifts in generated features using authentic content guidance, preserving the frozen backbone's pre-trained knowledge.
3. Synergistic Interaction: The authors demonstrate that the three modules are complementary rather than independently additive. For instance, the combination of AFEP and RAF (B0) underperforms AFEP alone under random missing conditions due to interference, which is resolved only when GDA is added to correct the distribution mismatch.
4. Complementary Evaluation Protocols: The study evaluates the model under two protocols: 70% random missing and fixed missing (TMDC-style), revealing that module effectiveness depends on both the evaluation metric and the missing-modality pattern.

Experimental Results

Experiments were conducted on the CMU-MOSI dataset using three random seeds (666, 777, 888).

• Random Missing Protocol (70% drop rate):

  • The complete model (AFEP-RAF-GDA, denoted as B) achieved the best performance across all four primary metrics (MAE, Corr, F1, Acc-2) among E0-initialized configurations.
  • Improvements: Acc-2 improved to 0.706 (from 0.684 baseline) and F1 to 0.705 (from 0.683).
  • Ablation: The pairwise combination AFEP+RAF (B0) underperformed the AFEP-only configuration, confirming the necessity of GDA to resolve non-additive interference.

• Fixed Missing Protocol:

  • The complete model B achieved the highest average Acc-2 (0.668) and F1 among E0-initialized configurations.
  • GDA Alone (E3): Surprisingly, GDA alone achieved the best mean MAE (1.218) and Corr (0.411), suggesting its specific strength in regression calibration under fixed patterns.
  • Comparison: The complete model outperformed the reproduced TMDC baseline in 5 out of 6 missing scenarios, despite using a single-stage training protocol versus TMDC's two-stage fine-tuning.

• Efficiency: The method adds only ~52.6K trainable parameters on top of a baseline with approximately ~5.0M frozen parameters, representing a ~1.05% increase in total model parameters.

Significance and Claims

The paper claims that the proposed framework offers a lightweight, modular alternative to full backbone fine-tuning for handling missing modalities. The significance lies in the synergistic design:

• The modules address distinct failure modes (feature quality, fusion reliability, and distribution alignment) that, when combined, resolve interference issues present in partial combinations.
• The effectiveness of the modules is metric- and pattern-dependent: While the complete model is optimal for random missing scenarios, specific configurations (like GDA alone) excel in specific fixed-missing regression tasks.
• The approach preserves pre-trained alignment knowledge while adapting to distribution shifts, avoiding the performance degradation associated with simple fine-tuning.

Limitations Acknowledged by the Authors:

• The reliability signal is currently binary; continuous quality-aware weighting is a future direction.
• Validation is limited to the CMU-MOSI dataset; cross-dataset and cross-lingual generalization (e.g., to CMU-MOSEI) is not empirically supported.
• The 70% missing rate is fixed; adaptability to dynamic missing ratios is not addressed.
• Statistical confidence on the magnitude of gains would require a larger number of seeds.
• The model remains less robust in specific video-related and text-missing patterns (e.g., when both text and audio are missing).