Technical Summary: Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition

1. Problem Definition

Multimodal Emotion Recognition in Conversations (MERC) aims to detect emotional states by integrating textual, auditory, and visual data. Despite the potential of multimodal approaches, a persistent challenge is modality imbalance. In many scenarios, certain modalities (typically text) dominate the learning process, while others (audio and visual) are underutilized or suppressed. This phenomenon, described as "modality inhibition," leads to suboptimal fusion strategies and reduced overall performance.

The paper identifies two specific dimensions of this imbalance:

• Property Discrepancy: Different modalities exhibit distinct learning behaviors (e.g., audio features may overfit faster than visual features).
• Quality Discrepancy: Modalities provide varying levels of discriminative information, causing greedy learning dynamics to prioritize the most informative modality while neglecting weaker ones.

Existing solutions, such as gradient modulation or auxiliary loss functions, often rely on complex gradient manipulations, specific architectural assumptions, or static weighting schemes that fail to adapt dynamically to the evolving learning state of the model.

2. Methodology

The authors propose Self-Paced Curriculum Learning (SPCL), a plug-and-play module designed to mitigate modality imbalance by dynamically structuring the training process. The framework consists of two core components: a Difficulty Measurer and a Learning Scheduler.

2.1 Difficulty Measurer (Dual-Level Design)

Unlike conventional approaches that assess difficulty solely at the utterance level, the proposed SPCL introduces a dual-level difficulty assessment:

1. Utterance-Level Score ($l_{ij}$): Derived from the cross-modal loss of a specific utterance. A higher loss indicates greater misalignment among unimodal representations for that specific instance.
2. Conversation-Level Score ($s_i$): A novel component that captures broader dialogue dynamics. It is calculated as the standard deviation of the aggregated unimodal scores within a conversation. A high standard deviation indicates significant modality misalignment (imbalance) across the conversation, while a low value suggests coherent multimodal fusion.

The final difficulty score ($\rho_{ij}$) for an utterance is computed using the harmonic mean of the utterance-level loss and the conversation-level score. This formulation ensures that neither recognition difficulty nor modality misalignment dominates the calculation, penalizing extreme values and prioritizing samples where both factors are balanced.

2.2 Learning Scheduler

The scheduler employs a hard regularizer to control the training pace. It acts as a binary gate:

• Samples with a difficulty score $\rho_{ij}$ below a dynamic threshold $\lambda$ are included in the loss calculation (weight = 1).
• Samples above the threshold are excluded (weight = 0).

The threshold $\lambda$ is updated exponentially over epochs ($\lambda(t) = \alpha \lambda(t-1)$), starting from a small initial value $\epsilon$. This ensures a strictly progressive curriculum where the model first learns from easy, well-aligned samples and gradually incorporates more challenging instances as its competence increases.

2.3 Integration

The SPCL module integrates into existing MERC architectures (e.g., DialogueGCN, MMGCN) without altering the underlying network structure. The standard loss function is replaced by a masked loss $L_{SPCL}$, which only accumulates gradients from samples selected by the scheduler.

3. Key Contributions

1. Novel Framework: Introduction of an SPCL-based framework specifically tailored for MERC to address modality imbalance through adaptive sample selection.
2. Dual-Level Difficulty Measurer: A mechanism that combines utterance-level recognition performance with conversation-level modality discrepancy (via standard deviation of unimodal scores) to provide a holistic view of sample difficulty.
3. Dynamic Learning Scheduler: A hard regularizer-based scheduler that progressively introduces samples based on model competence, ensuring a stable and balanced learning trajectory.
4. Plug-and-Play Compatibility: The method is designed to be architecture-agnostic, supporting arbitrary numbers of modalities, optimizers, and loss functions, making it applicable to diverse baseline models.

4. Experimental Results

The authors evaluated SPCL on two benchmark datasets: IEMOCAP and MELD, integrating the module into four baseline models (DialogueGCN, BiDDIN, MMGCN, MM-DFN).

• Performance Gains:

  • IEMOCAP: SPCL achieved improvements in weighted F1-score (w-F1) ranging from +1.2% to +6.6% over baseline models across different modality settings. Notably, in the AV (Audio-Visual) setting for DialogueGCN, gains reached +10.4%.
  • MELD: Improvements were even more pronounced, with gains reaching up to +10.4% over baselines in certain configurations.
  • SPCL consistently outperformed other imbalance-mitigation strategies, including RNA loss, OGM-GE, FAGM, and OPM.

• Ablation Studies:

  • Removing either the utterance-level or conversation-level score resulted in consistent performance degradation, confirming the necessity of the dual-level design.
  • Hard Regularization (binary masking) was found to be superior to soft regularization strategies (Linear and Logistic), providing more stable performance across modalities.
  • Pacing Strategy: Exponential pacing yielded better results than Cosine, Moving Average, or Competence-based pacing, highlighting the importance of a smooth curriculum expansion.

• Modality Balance: Analysis of modality ratios showed that SPCL effectively reduced the dominance of the text modality and increased the contribution ratios of audio and visual modalities, leading to a more balanced multimodal representation.

5. Significance and Claims

The paper claims that SPCL offers a practical and effective solution to the modality imbalance problem in MERC. Its significance lies in:

• Holistic Evaluation: By incorporating conversation-level dynamics, the method addresses the unique challenges of dialogue structures that utterance-level metrics miss.
• Robustness: The approach improves model stability and generalization, particularly in scenarios where non-textual modalities are weak or missing (e.g., AV settings).
• Practicality: As a plug-and-play module, it requires no architectural changes to existing models and maintains compatibility with standard training pipelines.
• Real-World Applicability: The substantial performance gains on benchmark datasets underscore the potential of SPCL for real-world emotion recognition applications where modality data may be noisy or imbalanced.

The authors acknowledge limitations, noting that the method's effectiveness is sensitive to hyperparameter tuning and that the curriculum progression may not align perfectly with datasets containing very brief or fragmented dialogues. Future work is proposed to extend SPCL to large multimodal models (LMMs) and cross-lingual datasets.