Technical Summary: HiRoC

Problem Statement

Multimodal Emotion Recognition in Conversation (MERC) aims to infer speakers' affective states from heterogeneous cues (text, audio, visual) within evolving dialogue contexts. While existing graph-based methods have demonstrated potential in modeling conversational structures, they suffer from three primary limitations:

1. Irrelevant Context Propagation: They often introduce history through dense or predefined connections, propagating irrelevant contextual information that introduces noise.
2. Indistinct Dependency Modeling: They frequently fail to distinguish between the different roles of same-speaker (intra-speaker) and cross-speaker (inter-speaker) dependencies, treating them with identical propagation mechanisms.
3. Loss of Corrective Cues: They typically obtain unified multimodal representations before prediction, which may average out useful corrective cues carried by inconsistent modalities (cross-modal disagreement), particularly in complex scenarios like sarcasm or hesitation.

Methodology

The authors propose HiRoC (Selective History Routing and Disagreement-Aware Calibration), a framework designed to selectively route emotion-relevant history and calibrate predictions using cross-modal disagreement. The architecture consists of four main components:

1. Historical Relevance Filter (HRF)

Before graph propagation, the HRF explicitly quantifies the nonlinear relevance between the current target utterance and all historical utterances.

• Mechanism: It constructs an interaction representation combining concatenation, difference, and element-wise multiplication of the target and historical vectors. A Multi-Layer Perceptron (MLP) followed by a sigmoid function generates a continuous dependency score ($\alpha_{t,i}$) for each historical node.
• Output:
  • History-Enhanced Representation ($X^m$): A weighted aggregation of historical representations where weakly related context is suppressed.
  • Dependency Score Matrix ($A^m$): A lower-triangular matrix serving as a structural prior for the subsequent routing graph.

2. Speaker-Typed Routing Graph (STRG)

Guided by the dependency priors from HRF, the STRG propagates information along sparse, temporally directed paths while separating dependency types.

• Edge Selection: To prevent noise and selection bias (where high-frequency speakers dominate), the model employs a Top-K edge selection strategy:
  • Intra-speaker: Global Top-K selection among same-speaker history.
  • Inter-speaker: Per-speaker Top-K selection (selecting Top-K from each other speaker individually before unioning) to ensure balanced participation of low-frequency speakers.
• Propagation: Messages are passed separately through intra-speaker and inter-speaker branches using learnable parameter matrices, then concatenated to form the routing representation ($Z_{strg}$).

3. Multimodal Hypergraph Backbone

Parallel to the STRG, a hypergraph backbone models high-order interactions.

• Structure: It utilizes two types of hyperedges:
  • Utterance Hyperedges: Bind the three modal nodes (text, audio, visual) within a single utterance.
  • Modal Hyperedges: Connect all nodes of the same modality within a sliding window.
• Integration: The hypergraph output ($Z_{hyper}$) captures high-order utterance- and modality-level interactions. It is integrated with the STRG output via a representation-level fusion strategy, where the sparse speaker-typed routing information acts as a complementary structural term.

4. Cross-Modal Residual Correction (CMRC)

To handle multimodal inconsistency, the model performs calibration at the decision level rather than absorbing disagreement into a unified feature representation.

• Process: The model generates a main prediction based on the fused representation. Simultaneously, it extracts unimodal predictions from the audio and visual branches.
• Disagreement Measurement: The Jensen–Shannon (JS) divergence is computed between the text-dominant distribution and the non-textual modalities.
• Correction: These divergence scores drive a gating mechanism to generate dynamic correction coefficients ($\beta$). The final prediction is a residual sum of the main logit and the weighted residual corrections from audio and visual modalities, capped by a hyperparameter $\beta_{max}$ to prevent over-correction.

Key Contributions

1. Selective History Routing: The introduction of HRF and STRG enables the selective retention and propagation of emotion-relevant history, addressing the coarse introduction of context in existing graph methods.
2. Complementary Structural Modeling: A novel scheme combining STRG (for sparse, speaker-typed temporal dependencies) with a hypergraph backbone (for high-order multimodal interactions).
3. Decision-Level Calibration: The CMRC module exploits cross-modal disagreement explicitly at the decision stage, preserving modality-specific corrective cues that might otherwise be averaged away.
4. State-of-the-Art Performance: The framework achieves superior performance on standard benchmarks.

Experimental Results

The framework was evaluated on two widely used datasets: IEMOCAP (dyadic conversations) and MELD (multi-party conversations).

• Performance: HiRoC outperformed representative state-of-the-art methods (including DialogueGCN, MMGCN, M3NET, and AdaIGN) on both datasets in terms of Accuracy and Weighted F1-score.
  • On IEMOCAP, HiRoC achieved 73.88% Accuracy and 73.91% w-F1.
  • On MELD, HiRoC achieved 68.10% Accuracy and 67.06% w-F1.
• Ablation Studies: Removing any core component (HRF, STRG, or CMRC) resulted in performance degradation, confirming the necessity of history filtering, speaker-typed propagation, and disagreement-aware calibration.
• Fine-Grained Analysis:
  • Routing Strategy: Per-speaker Top-K routing significantly outperformed global Top-K, particularly in multi-party settings (MELD), by preventing high-frequency speakers from dominating cross-speaker history.
  • Transition Types: HiRoC showed consistent improvements across Same-Speaker Non-Shift, Same-Speaker Shift, and Cross-Speaker Trigger scenarios, with notable gains in capturing speaker-internal continuity.
  • Error Correction: The model effectively corrected baseline errors, particularly for "Angry," "Happy," and "Excited" categories, though distinguishing fine-grained negative emotions (e.g., Frustrated vs. Angry) remained challenging.

Significance and Claims

The paper claims that HiRoC advances MERC by shifting the paradigm from "dense, unified representation" to "selective, structured, and calibrated reasoning." Its significance lies in:

• Noise Reduction: By filtering irrelevant history before propagation, the model avoids aggregating weakly related context.
• Structural Fidelity: By separating intra- and inter-speaker dependencies, the model better captures emotional continuity versus interaction-triggered changes.
• Robustness to Inconsistency: By treating cross-modal disagreement as a calibration signal rather than a feature fusion problem, the model leverages conflicting cues to refine predictions rather than diluting them.

The authors note that while the results are promising, the current evaluation assumes complete multimodal inputs. Future work is identified as exploring robustness under incomplete or degraded multimodal conditions (e.g., missing audio or visual signals).