Technical Summary: GRCF for Multimodal Sentiment Analysis

1. Problem Statement

Current Multimodal Sentiment Analysis (MSA) research predominantly relies on point-wise regression, where models predict a continuous sentiment score for each sample independently. While straightforward, this approach suffers from two critical limitations:

1. Sensitivity to Label Noise and Ordinal Neglect: It treats each instance in isolation, failing to capture the inherent ordinal hierarchy of human emotion (e.g., that "desperate" is more negative than "sad"). This leads to unstable predictions, particularly near neutral boundaries, and poor correlation alignment.
2. Limitations of Existing Pairwise Methods: Recent ordinal learning frameworks (e.g., TMSON, MOAC) attempt to address this by learning relative order through pairwise comparisons. However, they introduce new trade-offs:
   • Uniform Importance: They assign equal weight to all pairwise comparisons, failing to adaptively focus on "hard-to-rank" samples (those with ambiguous or close scores).
   • Static Margins: They employ fixed ranking margins that do not reflect the varying semantic distances between sentiment groups (e.g., treating the gap between "slightly positive" and "strongly positive" the same as "neutral" and "slightly positive").

2. Methodology: GRCF

The authors propose the Two-Stage Group-wise Ranking and Calibration Framework (GRCF), which adapts the philosophy of Group Relative Policy Optimization (GRPO) from reinforcement learning to supervised regression. The framework consists of two distinct stages:

Stage 1: Structural Foundation (Ordinal Learning)

The primary goal is to establish a robust, ordinally-consistent latent space. This stage introduces a Group-Aware Ranking Loss ($L_{group}$) composed of three key mechanisms:

• Overlapping Ordinal Groups: Instead of static bins, continuous scores are mapped to $K=5$ overlapping groups (e.g., $G_0: [-3.0, -1.5]$, $G_1: [-2.0, 0.0]$, etc.). This design captures semantic ambiguity at decision boundaries and ensures numerically close samples share group memberships.
• Dynamic Margin ($m_{ij}$): The margin for a pairwise loss is not static. It is computed based on the group difference ($\Delta G_{ij}$) between two samples. If samples belong to different groups with a large semantic gap, the margin increases proportionally. If they share a group (indicating semantic similarity), the margin is minimized. This enforces a "semantically proportional" structure.
• Advantage-Weighted Optimization: Inspired by GRPO, the framework calculates a standardized advantage ($A_{ij}$) based on the reward (prediction difference). Unlike standard pairwise methods that treat all pairs equally, GRCF uses a one-sided penalty weight $w_{ij} = \text{ReLU}(-A_{ij})$. This adaptively up-weights "difficult" pairs (misordered or ambiguous samples) while ignoring "easy" pairs where the ranking is already correct.

To prevent absolute score drift, Stage 1 also includes:

• Distribution Regularization ($L_{reg}$): Aligns the mean and standard deviation of the predicted batch distribution with the ground truth.
• Boundary Loss ($L_{bound}$): Enforces predictions to remain within the valid annotation range (e.g., $[-3, 3]$).

Stage 2: Fine-Tuning (Calibration)

While Stage 1 excels at relative ordering, it may suffer from a "score offset" (drifting absolute values). Stage 2 addresses this by:

• MAE-Driven Objective: A Mean Absolute Error (MAE) loss is introduced to align prediction magnitudes with absolute ground-truth scores.
• Differential Learning Rates: The fusion and output layers are fine-tuned with a higher learning rate, while the foundational encoders use a lower rate to prevent catastrophic forgetting of the learned ordinal structure.
• Regularization: The ranking loss ($L_{group}$) and boundary loss from Stage 1 are retained as regularizers to preserve the ordinal manifold.

Extension to Classification

For binary classification tasks (e.g., Sarcasm, Humor), the framework adapts by decomposing the group-wise philosophy into:

• Separation Loss: Pushes inter-class clusters apart using a weighted margin.
• Compactness Loss: Pulls intra-class samples together.
• Dynamic Adaptation: The dynamic margin mechanism is replaced with a fixed margin suitable for discrete labels, while the advantage-weighting mechanism focuses on modality conflicts (for sarcasm) or fine-grained proximity (for sentiment).

3. Key Contributions

1. Group-wise Computing in Ordinal Learning: A novel approach that fully exploits inter-sample relationships by mapping continuous scores to overlapping ordinal groups.
2. GRPO-Inspired Advantage Weighting: A mechanism that controls the model to focus optimization resources on confusable, hard-to-rank samples rather than treating all pairs equally.
3. Dynamic Margin via Overlapping Intervals: A method that captures semantic ambiguity and homogeneity near decision boundaries, ensuring the ranking margin scales proportionally with the true semantic gap.
4. Adaptation of GRPO to Supervised Regression: Successfully transferring the Group Relative Policy Optimization philosophy from sparse-reward RL to dense-reward regression tasks, solving the sparsity problem inherent in discrete RL but naturally present in continuous sentiment labels.

4. Experimental Results

The framework was evaluated on three core regression benchmarks and two classification tasks:

• Regression Benchmarks (CMU-MOSI, CMU-MOSEI, CH-SIMS v2):
  • GRCF achieved State-of-the-Art (SOTA) performance on CMU-MOSEI across all metrics (Acc7, Acc2, F1, MAE, Corr).
  • On CMU-MOSI, it achieved SOTA results on Acc7, Acc2, MAE, and Corr.
  • On CH-SIMS v2, it achieved SOTA on Acc3, Acc2, and F1, and tied for the best on Acc5.
• Classification Tasks (MUStARD, UR-FUNNY v2):
  • Demonstrated strong generalizability, achieving 75.00% Acc2 on MUStARD (Sarcasm) and 73.64% on UR-FUNNY v2 (Humor).
• Ablation Studies:
  • Removing Stage 1 (relying on direct regression) caused a massive performance drop (e.g., -11.55% Acc7 on MOSI), proving the necessity of the ordinal structural foundation.
  • Removing Stage 2 led to catastrophic failure in absolute calibration (e.g., -43.10% Acc3 on SIMS v2).
  • Both GRPO weighting and Dynamic Margin were critical for datasets with fine-grained labels (MOSI/MOSEI).
• Robustness:
  • The model showed remarkable stability against additive Gaussian noise in non-textual modalities.
  • Cross-dataset transfer learning (training on MOSEI, testing on MOSI) showed that the ordinal manifold learned in Stage 1 generalizes well, even when absolute calibration initially failed.

5. Significance and Claims

The paper claims that GRCF represents a significant shift in MSA methodology by moving away from isolated point-wise regression toward a relational, group-aware ordinal framework.

• Philosophical Shift: The authors argue that human sentiment is inherently comparative and ordinal. GRCF simulates human cognition by focusing on "difficult" distinctions and acknowledging that semantic distances between sentiment groups vary.
• Algorithmic Innovation: By adapting GRPO, the framework solves the "static constraint" problem of previous ordinal methods and the "uniform importance" problem of pairwise learning.
• Generalizability: The framework is not limited to regression; its ability to decouple structural ordinal learning from absolute calibration allows it to effectively transfer to binary classification tasks, demonstrating a unified approach to multimodal affective computing.

The authors conclude that the performance gains are driven by the Two-Stage Ranking and Calibration design rather than model scaling, as GRCF maintains a parameter count competitive with existing baselines while delivering superior results.