Unified Multimodal Uncertain Inference

1. Problem Statement

Current Natural Language Inference (NLI) and multimodal reasoning tasks largely rely on categorical (binary) judgments (e.g., entailment vs. non-entailment). However, human reasoning is fundamentally probabilistic, operating on degrees of belief and confidence rather than binary classifications.

• The Gap: While "Uncertain Natural Language Inference" (UNLI) exists for text, extending this to audio, video, and cross-modal settings is limited. Existing multimodal inference tasks are typically binary, lacking a framework for fine-grained probabilistic reasoning across modalities.
• The Consequence: Multimodal AI systems deployed in high-stakes domains cannot express calibrated uncertainty, limiting their trustworthiness. Current models often fail to integrate clues across vision, audio, and text to form a unified, calibrated belief estimate.

2. Methodology

The authors propose a new task and a corresponding training framework to address these limitations.

A. Task Definition: Unified Multimodal Uncertain Inference (UMUI)

• Input: A premise (text, audio clip, video, or a combination) and a hypothesis (a declarative statement).
• Output: A scalar probability estimate $p \in [0, 1]$ representing the likelihood that the hypothesis is true given the premise.
• Scope: The task covers unimodal (text-only, audio-only, video-only) and multimodal (audio-visual, text-visual, etc.) scenarios.

B. Data Collection

• Source: Built upon the WIKIVIDEO dataset (claims vs. video content) and CLOTHO (audio captions).
• Annotation: The authors curated a human-annotated evaluation set with scalar probability judgments (0–100%) for audio-visual premise-hypothesis pairs.
• Protocol: Expert annotators judged 10 specific events (e.g., Notre-Dame fire, earthquakes). They indicated which modalities informed their judgment and provided a probability score.
• Scale: 2,526 human-labeled video-claim pairs with high inter-annotator agreement (Krippendorff's $\alpha \approx 0.71$).

C. The CLUE Framework (Calibrated Latent Uncertainty Estimation)

To train models that produce well-calibrated probabilities, the authors introduce CLUE, which consists of three core components:

1. Self-Consistent Teacher Calibration:

   • Since fine-grained human labels are expensive to generate at training scale, the authors use a Teacher Model (Qwen3-VL/Omni) to generate synthetic labels.
   • Technique: The teacher generates $N=5$ independent reasoning traces per sample. These are aggregated via mean (average confidence) or max (majority vote) to create a calibrated distillation score.
   • Result: This self-consistency approach significantly reduces Mean Squared Error (MSE) and improves accuracy compared to single-shot teacher outputs.

2. Latent Distribution Confidence Modeling:

   • Instead of generating a discrete token (e.g., "0.8"), the model predicts a latent distribution over confidence tokens.
   • Mechanism: The interval $[0, 1]$ is discretized into $N=100$ bins, each associated with a special token <CONF_i>.
   • Training Objective: The model minimizes the Kullback–Leibler (KL) divergence between its predicted distribution $P$ and a target Gaussian distribution $Q \sim \mathcal{N}(y, \sigma^2)$, where $y$ is the ground-truth probability.
   • Inference: The final scalar probability is reconstructed via weighted aggregation of the predicted distribution, allowing for smooth, fine-grained estimates rather than discrete steps.

3. Modality-Specific Batching:

   • To address gradient instability caused by mixing modalities with vastly different sequence lengths (e.g., short text vs. long video), the authors implement modality-specific batching.
   • Samples are grouped by modality (all-text, all-audio, all-video) within a batch rather than mixing them. This ensures balanced gradient magnitudes and reduces padding overhead.

3. Key Contributions

1. UMUI Task: Formalized the first task for calibrated probabilistic inference spanning text, audio, video, and their combinations.
2. Dataset: Released a human-annotated dataset with scalar probability judgments for audio-visual reasoning, extending existing binary annotations to fine-grained uncertainty.
3. CLUE Model: Introduced a training paradigm combining self-consistent teacher distillation, latent distribution modeling, and modality-aware batching.
4. Efficiency: Demonstrated that a 3B-parameter model (CLUE) outperforms baselines up to 32B parameters in calibration and accuracy.

4. Experimental Results

The authors evaluated CLUE against baselines including Audio Flamingo, Qwen2-Audio, Qwen2.5-VL, and Qwen3-VL across CLOTHO (audio), UNLI (text), and WIKIVIDEO (video/audio-visual).

• Binary Performance (Accuracy):
  • On CLOTHO, CLUE achieved near-perfect accuracy (97.5%), vastly outperforming audio-only baselines (~49%).
  • On WIKIVIDEO, CLUE outperformed all baselines (including 32B models) across vision-only, audio-only, and audio-visual settings.
• Scalar Performance (MSE):
  • CLUE achieved the lowest Mean Squared Error (MSE) on UNLI (5.7 vs. 7.5 for the next best 32B model) and WIKIVIDEO.
  • Distribution vs. Token: The Distribution-based (CLUE-D) variant consistently outperformed the Token-based (CLUE-T) variant, proving that modeling the internal confidence distribution yields smoother, more human-aligned probability estimates.
• Cross-Modal Transfer:
  • Training on a mixture of modalities improved performance even in text-only settings. CLUE-D (3B) outperformed TMTO (14B, text-only trained) on UNLI, suggesting that auxiliary modalities provide reciprocal benefits for single-modality reasoning.
• Calibration:
  • Temperature scaling on baselines yielded minimal improvement, whereas CLUE's end-to-end latent distribution approach provided robust calibration without post-hoc tuning.

5. Significance and Conclusion

• Trustworthy AI: This work bridges the gap between binary classification and human-like probabilistic reasoning, enabling AI systems to express how sure they are, which is critical for high-stakes applications (e.g., disaster response, medical diagnosis).
• Efficiency: The results challenge the notion that larger models are always better for calibration; a smaller, well-calibrated 3B model can outperform massive 32B models when trained with the correct uncertainty-aware objectives.
• Methodological Shift: The paper advocates for moving away from discrete token generation for probability estimation toward latent distribution modeling, which captures the nuance of uncertainty more effectively.
• Limitations: The training data relies on synthetic labels from teacher models rather than massive human annotation, though the authors verified high alignment between teacher self-consistency and human judgment. Future work may explore active learning to incorporate more direct human scalar feedback.