Technical Summary: GAUGE

Problem Statement

Multimodal classification models typically assume the availability of all required modalities (e.g., images, text, sensor data). However, real-world scenarios frequently involve incomplete inputs due to sensor failures, privacy constraints, or transmission errors. Existing approaches to handle this incompleteness generally fall into two categories:

1. Recovery-free methods: Predict directly from available modalities, avoiding imputation errors but discarding potentially recoverable cross-modal cues.
2. Recovery-based methods: Explicitly impute missing modalities to restore complementary cues. However, reconstructed content often contains low-fidelity, noisy, or semantically inconsistent evidence.

Current dynamic fusion and robustness mechanisms operate at a coarse modality level, applying a single control decision (e.g., a weight or gate) to the entire output of a modality encoder. This approach fails to address within-modality heterogeneity, where informative and misleading signals may coexist within the same recovered modality. Consequently, existing methods cannot selectively suppress unreliable components while retaining reliable ones within a single recovered modality, compromising prediction reliability.

Methodology: GAUGE

The authors propose GAUGE (Granularity-Adaptive coUnterfactual Gating of Evidence), a lightweight framework designed to perform fine-grained evidence control for incomplete multimodal classification. GAUGE operates in three phases:

1. Complete & Encode

• Imputation: A pre-trained, frozen imputer ($\Gamma$) reconstructs missing modalities.
• Granularity-Adaptive Encoding: Both observed and recovered inputs are encoded into fine-grained evidence units. These units correspond to the native granularity of the encoders (e.g., image patches, text tokens, or tabular feature embeddings). If an encoder emits a single global embedding, the unit corresponds to the modality level; otherwise, it corresponds to sub-modality components.
• Sequence Formation: All evidence units are concatenated with a learnable [CLS] token to form a multimodal sequence $\tilde{T}_i$.

2. Score & Gate (Counterfactual Attribution)

Instead of explicitly intervening on each unit (which would require $N$ additional forward passes), GAUGE estimates the counterfactual effect of replacing each evidence unit with a reference representation (set to zero) using a first-order Taylor approximation.

• Prediction-Aware Saliency: An ungated forward pass determines the predicted class $\hat{y}_i$. The saliency objective is defined as the negative logit of this class ($L_{sal} = -z_{i, \hat{y}_i}$).
• Taylor Evidence Scoring: A single backward pass computes the gradient of the saliency objective with respect to each evidence unit. The Taylor Evidence Score ($TE_{i,j}$) is calculated as the $\ell_1$ norm of the channel-wise products between the gradient and the replacement direction ($d_{i,j} = 0 - e_{i,j}$). This aggregates absolute contributions to prevent sign cancellation.
• Efficiency: This process yields scores for all $N$ units in a single forward-backward pass, avoiding the prohibitive overhead of exact counterfactual evaluation.
• Normalization & Gating: Scores are normalized using an N-adaptive rule (z-score for large $N$, raw scores for small $N$) and mapped to continuous gates ($g_{i,j} \in (0,1)$) via a sigmoid function with learnable threshold ($\tau$) and temperature ($\rho$).

3. Predict & Update

• Attention Bias Injection: The continuous gates are converted into additive attention-logit biases ($B_i$) injected into the Key positions of every Transformer block. Specifically, $B_i$ applies a log-bias of $\log(\max\{g_{i,r}, \epsilon\})$ to the attention logits of key $r$.
• Effect: Lower gates result in stronger negative biases, down-weighting low-scoring evidence units without altering the backbone architecture or sequence length.
• Training: The model is trained by minimizing the cross-entropy loss on the gated prediction. The Taylor scores are detached, preventing gradients from flowing through the scoring mechanism, ensuring the imputer remains frozen and the process remains efficient.

Key Contributions

1. Granularity-Adaptive Paradigm: The paper formulates a control paradigm that treats both observed and reconstructed modalities as collections of encoder-emitted evidence units. This enables fine-grained modulation at the finest available representation level while maintaining compatibility with modality-level settings.
2. GAUGE Framework: A lightweight, counterfactual-attribution gating mechanism that derives prediction-aware evidence scores via a first-order Taylor formulation. It obtains all unit-level scores in a single forward-backward pass, bypassing the need for explicit per-unit interventions.
3. Theoretical Analysis: The authors provide a Taylor remainder analysis characterizing the error of the first-order approximation relative to the exact counterfactual effect. They establish an upper bound relating the Taylor evidence score to the magnitude of the exact counterfactual effect, validating the method's principled nature.
4. Empirical Performance: Extensive experiments across six benchmarks (including PolyMNIST, MST, CelebA, DVM, CAD, and Infarction) demonstrate that GAUGE outperforms strong baselines (recovery-free, recovery-based, and dynamic recovery methods) across diverse incomplete-input settings, particularly under severe missingness.

Results

• Performance: GAUGE achieved the best or tied-best performance in 9 out of 12 incomplete-input settings reported.
• Comparison with DyMoc: When compared to DyMoc (a dynamic recovery baseline using the same backbone and imputation pipeline but operating at the modality level), GAUGE outperformed it in 10 of 12 settings.
  • On the CelebA dataset with the text modality entirely missing, GAUGE improved accuracy by 5.90 percentage points over DyMoc.
  • On clinical benchmarks (CAD and Infarction), GAUGE showed consistent gains, with improvements becoming more pronounced as the missing rate increased.
• Efficiency: In terms of scoring time, GAUGE reduced the per-sample scoring time from 96.43 ms (exact counterfactual scoring) to 3.11 ms at $N=91$, achieving a 31.01$\times$ speedup.
• Qualitative Analysis: Visualizations show that GAUGE assigns non-uniform gates within a single recovered modality (e.g., imputed tabular features), down-weighting specific low-scoring components while retaining high-scoring ones. This contrasts with modality-level gating, which would uniformly suppress the entire modality.

Significance and Claims

The paper claims that GAUGE establishes a principled and scalable framework for fine-grained evidence control under modality incompleteness. By transitioning multimodal evidence modulation from coarse modality-level aggregation to the finest granularity exposed by Transformer encoders, GAUGE effectively mitigates the trade-off between discarding missing modalities and incorporating noisy imputations.

The authors emphasize that their approach is pass-efficient, requiring only a single forward-backward scoring pass regardless of the number of evidence units, making it suitable for scalable applications. The theoretical analysis provides a bound on the approximation error, grounding the method in counterfactual reasoning theory.

Limitations and Future Work:
The paper acknowledges that GAUGE currently relies on pre-trained backbones and frozen imputers inherited from existing baselines. Future work is proposed to explore extending this pass-efficient counterfactual gating paradigm to broader multimodal reasoning and generation architectures. The authors do not claim GAUGE solves all multimodal challenges but position it as a significant step toward reliable prediction in high-stakes applications like medical diagnosis and autonomous driving where inputs are often incomplete.