🤖 AI

Variational Visual Question Answering for Uncertainty-Aware Selective Prediction

This paper introduces "Variational VQA," a variational Bayesian approach that significantly enhances the reliability and selective prediction capabilities of large Vision Language Models by improving calibration and reducing hallucinations, particularly in low-error-tolerance scenarios.

Original authors: Tobias Jan Wieczorek, Nathalie Daun, Mohammad Emtiyaz Khan, Marcus Rohrbach

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tobias Jan Wieczorek, Nathalie Daun, Mohammad Emtiyaz Khan, Marcus Rohrbach

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant but overconfident tour guide for a trip through a strange, new city. This guide knows the city's main landmarks perfectly, but when asked about obscure alleyways or tricky questions, they don't say, "I don't know." Instead, they confidently point in the wrong direction, insisting, "Oh, that's definitely the museum!"

This is exactly the problem with today's most advanced Vision-Language Models (VLMs). These AI systems can look at a picture and answer questions about it with incredible accuracy. However, they suffer from a dangerous flaw: they are terrible at knowing when they are wrong. They are prone to "hallucinations"—making things up with 100% confidence.

This paper introduces a new method called Variational VQA (VarVQA) to fix this. Think of it as giving the tour guide a "uncertainty radar" so they know when to stop and say, "I'm not sure, let's ask someone else," rather than leading you off a cliff.

Here is the breakdown of how they did it, using some everyday analogies:

1. The Problem: The "Confident Idiot"

Current AI models are trained using a standard method (called AdamW) that is like a student cramming for a test by memorizing answers. They get great scores, but if they encounter a question they haven't seen before, they guess wildly and confidently.

The Risk: In safety-critical fields like medical diagnosis or helping the blind, a confident wrong answer is worse than no answer at all.

2. The Solution: The "Variational" Approach

The authors propose a new training method called Variational Learning.

The Old Way (AdamW): Imagine a single student taking a test. They give one answer. If they are wrong, they are just wrong.
The New Way (VarVQA): Imagine a committee of 64 slightly different students (a "posterior distribution") all taking the same test.
- If the committee all agree on the answer, the AI is confident.
- If the committee is split (some say "Yes," some say "No," some say "Maybe"), the AI knows there is uncertainty.

This method uses a special mathematical tool called IVON (which is like a super-smart, efficient way to train this committee) that doesn't slow the computer down too much.

3. The New "Risk-Averse" Selector

Even with a committee, how do you decide when to speak up? The paper introduces a new rule for the AI, called a Risk-Averse Selector.

The Analogy: Imagine you are betting money on the committee's answer.
- Standard Method: You just look at the average vote. If 51% say "Yes," you bet on "Yes."
- VarVQA's Method: You look at the average vote, BUT you also check how much the committee is arguing. If the committee is split 51% vs 49%, the "argument" (variance) is high. The new rule says: "If the committee is arguing too much, don't bet. Just say 'I don't know'."

This is crucial for High-Stakes situations. It's better to skip a question (abstain) than to lose a huge bet (make a dangerous error).

4. The Results: "Knowing What You Don't Know"

The researchers tested this on two types of tasks:

Visual Question Answering (VQA): Looking at a photo and answering questions.
Visual Reasoning: Looking at two photos and deciding if a statement is true or false.

The Findings:

Accuracy: The new method is just as good at getting the right answer as the old method.
Safety: When the AI does get it wrong, the old method is usually very confident about the mistake. The new method is much more likely to say, "I'm not sure," and skip the question.
The "One Sample" Trick: Surprisingly, even if you only ask one member of the committee (instead of 64), the new method is often more reliable than the old method's best guess.
Low Error Tolerance: In situations where you can only afford 1 mistake out of 100 (very strict safety), the new method performs significantly better.

5. Why This Matters

Think of this as upgrading a self-driving car.

Old AI: The car sees a weird shape on the road, guesses it's a rock, and drives over it at high speed.
New AI (VarVQA): The car sees the weird shape, realizes its "committee" is confused, and gently slows down to ask the human driver for help.

Summary

This paper proves that we can make powerful AI models safer and more trustworthy without making them slower or requiring massive new hardware. By teaching the AI to understand its own uncertainty (using a "committee" approach) and being brave enough to say "I don't know," we can deploy these models in real-world situations where mistakes are costly.

It's not about making the AI smarter; it's about making the AI humble.

1. Problem Statement

Vision Language Models (VLMs) have achieved near-human performance on Visual Question Answering (VQA) benchmarks but suffer from critical reliability issues:

Overconfidence and Hallucinations: Models often confidently predict incorrect answers, especially on out-of-distribution (OOD) or adversarial inputs.
Lack of Uncertainty Quantification: Standard VLMs trained with optimizers like AdamW lack a robust mechanism to express "I don't know," making them unsuitable for safety-critical domains (e.g., medical diagnosis).
Limitations of Current Solutions:
- External Selectors: Adding a separate head or tool to estimate uncertainty introduces overhead and new failure points without improving the base model's reliability.
- Bayesian Methods: Traditional Variational Bayes (VB) approaches have historically been ineffective or too computationally expensive for large transformer architectures.
- Monte Carlo (MC) Dropout: While useful, it often requires significant inference compute and may not offer the best sample efficiency.

The core challenge is to enable Selective Prediction (where a model abstains from answering when uncertain) without sacrificing accuracy or incurring prohibitive computational costs.

2. Methodology: Variational VQA (VarVQA)

The authors propose Variational VQA, a framework that integrates Variational Inference directly into the training of large multimodal transformers using the Improved Variational Online Newton (IVON) optimizer.

A. Training with IVON

Instead of learning point estimates of weights ( $\theta$ ) via AdamW, the model learns a posterior distribution $q(\theta)$ over the weights.

Objective: Minimize the variational lower bound:
$L(q(\theta)) = \lambda \mathbb{E}_{q(\theta)}[\bar{\ell}(\theta)] + D_{KL}(q(\theta) \parallel p(\theta))$
IVON Optimizer: This method approximates the Hessian matrix online to perform natural gradient updates. It updates both the mean ( $m$ $m$ ) and variance ( $v$ $v$ ) of the weight distribution.
- Unlike Laplace approximation, it does not require a second pass over data.
- Unlike MC Dropout, it learns a fixed posterior form, making it easier to use for downstream tasks like ensembling.
Efficiency: IVON achieves accuracy comparable to AdamW with negligible training overhead (approx. 0.8% to 4.2% increase depending on model size).

B. Inference Strategies

The paper evaluates two inference modes:

VarVQA Mean: Uses only the learned mean parameters ( $m$ ) for a single forward pass. This matches the inference speed of AdamW but retains some reliability benefits.
VarVQA Sampling: Samples $S$ weights from the posterior $q(\theta)$ to generate $S$ predictions. These are aggregated to compute a mean likelihood ( $\bar{\mu}$ ) and a variance ( $\bar{\sigma}^2$ ).

C. Risk-Averse Selector ( $g_{BPA}$ )

A key innovation is the Beyond Predictive Averaging (BPA) selector. Standard methods use the mean probability ( $\bar{\mu}$ ) as the confidence score. The authors propose penalizing high-variance predictions to create a conservative confidence bound:
$g_{BPA}(x) = \bar{\mu}_{\kappa}(x) - \bar{\sigma}_{\kappa}(x)$
Where $\kappa$ is the index of the most likely class.

Rationale: This approximates the 16th percentile of the predicted class distribution (assuming Gaussianity), providing a lower confidence bound.
Benefit: It effectively reduces false positives in high-stakes scenarios by forcing the model to abstain when prediction variance is high, even if the mean probability is high.

3. Key Contributions

First Application of Variational Learning to Multimodal Selective Prediction: Demonstrates that variational methods (via IVON) are effective for large VLMs (ViLT, BEiT-3) for selective prediction, a domain previously dominated by unimodal studies.
Superior Calibration and Reliability: VarVQA achieves significantly lower Expected Calibration Error (ECE) and better selective prediction performance compared to AdamW and MC Dropout, particularly in high-stakes scenarios (low error tolerance, e.g., $\le 1\%$ ).
Sample Efficiency: VarVQA outperforms MC Dropout with the same inference compute budget. Notably, a single posterior sample often yields more reliable answers than AdamW.
Novel Risk-Averse Selector: The introduction of $g_{BPA}$ , which leverages output variance, consistently outperforms standard predictive averaging in high-stakes settings.
Robustness to OOD: The method shows improved robustness when tested on mixed In-Distribution (VQAv2) and Out-of-Distribution (AdVQA) data.

4. Experimental Results

Experiments were conducted on VQAv2 (VQA) and NLVR2 (Visual Reasoning) using ViLT and BEiT-3 (base and large).

Accuracy: VarVQA matches or slightly surpasses AdamW accuracy (e.g., BEiT-3 Large on VQAv2: 78.89% vs. 78.59%).
Calibration (ECE): VarVQA significantly reduces ECE. For BEiT-3 Large on VQAv2, ECE dropped from 0.039 (AdamW) to 0.018 (VarVQA).
Selective Prediction (High-Stakes):
- Metric: Coverage at Risk (C@R) and Effective Reliability ( $\Phi_c$ ).
- Performance: At a strict risk tolerance of 0.5% (1 error per 200 samples), VarVQA improved coverage by 7–9% on VQAv2 and 9–14% on NLVR2 compared to AdamW.
- Comparison: VarVQA with 64 samples outperformed Deep Ensembles (which require training 3x models) in many metrics, while VarVQA Mean (1 pass) often outperformed AdamW.
OOD Robustness: In mixed VQAv2/AdVQA settings, VarVQA maintained better calibration and relative selective prediction performance as the OOD fraction increased, whereas AdamW's performance degraded more sharply in terms of reliability.
Qualitative Analysis: Visual examples show VarVQA correctly abstaining on "hallucinated" answers where AdamW confidently predicted wrong answers, and correctly answering questions where AdamW abstained.

5. Significance and Conclusion

This paper establishes Variational Learning as a viable, efficient, and superior alternative to standard optimization for making large VLMs safer and more trustworthy.

Practical Impact: By "baking in" reliability during training rather than adding it as an afterthought, VarVQA enables the deployment of VLMs in safety-critical applications where false positives are costly.
Efficiency: It challenges the notion that Bayesian methods are too expensive for large models, showing that IVON provides a favorable trade-off between compute and reliability.
Future Directions: The authors suggest that future work could focus on variance propagation in a single forward pass to eliminate sampling costs entirely and exploring full-covariance posteriors for even richer uncertainty modeling.

In summary, Variational VQA provides a principled, computationally feasible framework to mitigate hallucinations and overconfidence in multimodal models, significantly enhancing their utility in real-world, high-stakes environments.