Technical Summary: Sufficiency-Informed Evidential ValvE (SIEVE)

1. Problem Statement and Motivation

Multimodal Sentiment Analysis (MSA) aims to infer human affect by integrating language, acoustic, and visual signals. While existing robust MSA methods address the challenge of missing modalities (e.g., corrupted audio or occluded video), they predominantly adhere to a "repair-first" paradigm. These methods, whether employing explicit reconstruction or implicit compensation, intervene to repair missing modalities for every incomplete sample before making a prediction.

This paper challenges the fundamental assumption that every missing modality must be repaired. Through a per-sample oracle analysis on the CMU-MOSI dataset, the authors demonstrate that:

• Full-modality input is optimal for only a small fraction of samples (15.5% for MISA and 33.2% for GCNet).
• Every possible subset of modalities is the preferred input for some samples.
• For certain samples, adding or repairing a missing modality introduces redundancy or noise, potentially degrading prediction performance compared to using only the observed modalities.

Consequently, the central question shifts from "how to repair" to "whether to repair" for a specific sample. The paper posits that a model should first assess if the observed modalities carry sufficient evidence for direct prediction; if so, repair is unnecessary and potentially harmful.

2. Methodology: SIEVE Framework

The authors propose Sufficiency-Informed Evidential ValvE (SIEVE), a plug-and-play framework that makes a learnable, sample-level decision on whether to invoke a repair module. SIEVE is "repair-agnostic," meaning it can sit atop any existing explicit or implicit repair backbone without modifying its internal design.

2.1 Dual-Branch Prediction

For each input sample with a missing modality mask, SIEVE constructs two parallel pathways:

1. Direct Branch: Predicts sentiment directly from the observed modalities (using learned placeholders for missing ones).
2. Repair Branch: First applies a pre-existing repair module ($R_\omega$) to reconstruct or compensate for missing information, then predicts sentiment using both the observed and repaired representations.

2.2 Empirical Sufficiency Signal

Since there is no ground-truth label indicating whether a specific sample's observed modalities are "sufficient," SIEVE derives an empirical sufficiency signal from the training process itself. It computes the per-sample loss gap ($A_i$) between the two branches:

• If the Direct Branch loss is lower than the Repair Branch loss, the observed modalities are deemed empirically sufficient.
• This gap serves as a soft, internal supervision signal ($s^*_i$) to train the routing mechanism.

2.3 Evidential Sufficiency Valve

To route samples between the two branches, SIEVE employs an Evidential Valve based on Subjective Logic and Evidential Deep Learning.

• Parameterization: The valve predicts the parameters ($\alpha_i, \beta_i$) of a Beta distribution representing the probability of sufficiency. $\alpha_i$ accumulates evidence for sufficiency, while $\beta_i$ accumulates evidence for insufficiency.
• Uncertainty Modeling: The distribution also captures epistemic uncertainty ($u_i$). High uncertainty indicates the model is unsure whether repair is needed.
• Uncertainty-Aware Routing: The routing decision is not based on a fixed threshold. Instead, it uses a dynamic threshold ($\theta(u_i)$) that combines the predicted sufficiency score with the epistemic uncertainty and a "pattern base rate" (the historical probability of sufficiency for a specific missing pattern).
  • If sufficiency is high and uncertainty low, the sample is routed to the Direct Branch.
  • If sufficiency is low or uncertainty is high (favoring the repair branch based on prior patterns), the sample is routed to the Repair Branch.

2.4 Optimization

Training follows a two-stage protocol:

1. Warmup: Both branches are trained equally while the valve is frozen to stabilize the loss gap signal.
2. Joint Training: The system optimizes three losses:
   • Prediction Loss: Weighted by the soft routing probability.
   • Valve Calibration Loss: Aligns the predicted Beta distribution with the empirical sufficiency target.
   • Evidence Regularizer: Penalizes evidence assigned in the "wrong direction" (e.g., predicting high sufficiency when the repair branch actually performed better), stabilizing the uncertainty estimation.

3. Key Contributions

1. Challenging the Repair-First Assumption: The paper provides empirical evidence that forcing repair on all incomplete inputs is suboptimal, as modality utility is sample-dependent.
2. SIEVE Framework: A novel, plug-and-play selective repair framework that learns to decide "whether to repair" without external sufficiency labels, utilizing a dual-branch loss gap and evidential uncertainty modeling.
3. Performance Gains: SIEVE consistently improves diverse repair backbones (explicit and implicit) across different missing rates and datasets, approaching the theoretical "per-sample achievable optimum" (the best possible performance if the correct branch were chosen for every sample).

4. Experimental Results

Experiments were conducted on CMU-MOSI and IEMOCAP under varying missing rates (up to 70% missing modalities).

• Comparison with Baselines: SIEVE integrated with three representative backbones (GCNet, CorrKD, MoMKE) consistently outperformed the baselines alone and other state-of-the-art robust MSA methods. For instance, GCNet + SIEVE achieved the best overall performance on both datasets.
• Robustness Across Missing Rates: Improvements were observed across all missing rates, including the full-modality setting ($MR=0.0$), suggesting that selective routing helps avoid unnecessary noise even when all data is present.
• Ablation Studies: Removing the valve calibration loss, evidence regularizer, or the dynamic threshold significantly degraded performance, validating the necessity of uncertainty-aware routing and proper supervision.
• Routing Behavior Analysis:
  • The valve learned meaningful sufficiency signals, correctly identifying that text-heavy patterns are often sufficient for direct prediction.
  • t-SNE visualizations confirmed that the Direct and Repair branches occupy distinct regions in the representation space, proving they are not redundant.
  • SIEVE significantly narrowed the gap to the "Achievable Optimum" (the oracle upper bound), recovering a large portion of the potential performance lost by using a single static strategy.

5. Significance and Claims

The paper claims that SIEVE establishes "whether to repair" as a first-class question alongside "how to repair" in missing-modality MSA. By treating modality sufficiency as a sample-dependent property and modeling the uncertainty of this decision, SIEVE enables robust multimodal systems to selectively intervene. The authors argue that this selective intervention principle is not limited to sentiment analysis but is a valuable design pattern for broader robust multimodal systems.

The paper remains modest regarding its scope, acknowledging that its empirical study is currently restricted to sentiment analysis on specific datasets and that the training overhead (evaluating both branches) is non-zero, though manageable. Future work is suggested to explore broader applications (e.g., medical diagnosis, action recognition) and amortized training schemes to reduce computational costs.