Technical Summary: Visual Semantic Entropy (VSE)

1. Problem Definition

Vision-Language Models (VLMs) excel at Visual Question Answering (VQA) but often produce confident predictions on visually ambiguous inputs, leading to biased and unreliable outcomes. Reliable uncertainty estimation (UE) is critical for filtering these predictions. However, existing UE methods exhibit three specific failure modes when applied to VLMs:

1. Suppressed Output Variation: Standard entropy-based methods like Semantic Entropy (SE) rely on stochastic decoding to generate diverse outputs. The authors demonstrate that when a VLM forms a highly confident visual embedding (even for ambiguous images), stochastic decoding yields nearly identical outputs. Consequently, SE underestimates uncertainty because it fails to capture ambiguity inherent in the visual input.
2. Inflated Distances from Wording: Methods that aggregate pairwise semantic distances (e.g., SNNE) treat answers as categorical labels or rely on embedding distances. Because language is discrete, semantically equivalent answers with different phrasings can appear distant in embedding space. This linguistic variation inflates uncertainty estimates even when there is no genuine semantic disagreement.
3. Textual Dominance in Joint Perturbations: Recent approaches probe uncertainty via input perturbations, including textual paraphrasing or joint text-image perturbations. The authors show that textual paraphrasing induces significantly larger semantic shifts in the multimodal embedding space than visual perturbations. As a result, output variability in these methods is driven primarily by prompt sensitivity (linguistic changes) rather than visual ambiguity, leading to uncertainty estimates that reflect text sensitivity rather than visual evidence.

2. Methodology: Visual Semantic Entropy (VSE)

To address these limitations, the authors propose Visual Semantic Entropy (VSE), a framework designed to probe visual ambiguity while mitigating linguistic noise. The method consists of three core stages:

A. Visual Perturbation (Fixed Text)

Unlike methods that perturb the question or jointly perturb text and image, VSE keeps the text query $q$ fixed and perturbs only the input image $v$.

• Mechanism: A visual perturbation operator $\mathcal{T}$ generates $M$ perturbed views $v_m = \mathcal{T}(v; \xi_m)$. The authors use Gaussian noise with a fixed, small standard deviation ($\sigma=20$) to induce local visual variations without altering high-level semantic content or moving the sample outside the local neighborhood of the original input.
• Rationale: This isolates visual uncertainty. By avoiding textual changes, the method prevents the "prompt sensitivity" issue where paraphrasing dominates output diversity.

B. Answer Sampling

The VLM generates $M$ answer samples $\{a_m\}$ corresponding to the $M$ perturbed image views. This process elicits alternative interpretations of the visual evidence that stochastic decoding alone might miss when visual embeddings are overconfident.

C. Prototype Semantic Aggregation (ProtoSem)

To prevent linguistic variations from inflating uncertainty, VSE aggregates answers at the semantic level rather than treating them as raw tokens.

1. Clustering: The answer variants are clustered into $K$ groups $\{c_k\}$ based on semantic distance (using a pre-trained semantic similarity model like DeBERTa).
2. Prototype Selection: For each cluster $c_k$, a prototype answer $p_k$ is selected as the member minimizing the average distance to other members in the same cluster.
3. Uncertainty Calculation: Uncertainty is computed as the mass-weighted pairwise dispersion among these prototypes:
   $$\tilde{u} = \sum_{k \neq k'} w_k w_{k'} \, \mathrm{d}(p_k, p_{k'})$$
   where $w_k = |c_k|/M$ is the empirical probability (mass) of cluster $k$. This metric captures disagreement between distinct semantic meanings while ignoring intra-cluster wording variations.

3. Key Contributions

1. Analysis of Failure Modes: The paper provides a systematic analysis identifying three specific limitations of current VLM uncertainty estimators: (1) overconfident visual embeddings suppressing output diversity, (2) wording variations inflating distance-based metrics, and (3) textual perturbations dominating output variability in joint perturbation schemes.
2. Visual Semantic Entropy (VSE): The authors introduce VSE, a novel UE method that perturbs only the image to probe visual ambiguity and uses prototype-based semantic aggregation to quantify uncertainty. This design explicitly avoids the confounding factors of prompt sensitivity and linguistic noise.
3. Comprehensive Benchmarking: The study evaluates VSE across five modern VLMs (Qwen2.5-VL, Gemma3, Intern3.5-VL, LLaVA-NeXT, Qwen3-VL) and five diverse benchmarks (OKVQA, AOKVQA, MMVet, VILP, VLM-are-biased), establishing a new state-of-the-art for VQA uncertainty estimation.

4. Experimental Results

The evaluation uses the Area Under the ROC Curve (AUC) to measure how well uncertainty scores distinguish correct from incorrect predictions.

• General Performance: VSE consistently outperforms existing methods (Verbalized, Logit-based, Consistency-based, and Entropy-based) across all tested models and datasets. For instance, on the AOKVQA dataset, VSE achieved AUC scores ranging from 0.724 to 0.798 across different models, outperforming the next best entropy-based method (VL-Uncertainty) by up to 9.2% in some cases.
• Visual Ambiguity Focus: On adversarial datasets designed to expose visual bias (VILP and VLM-are-biased), VSE showed the most significant gains. For example, on VILP with Qwen2.5-VL, VSE achieved an AUC of 0.650, substantially outperforming the baseline SE (0.535) and VL-Uncertainty (0.489). This confirms VSE's ability to capture uncertainty arising specifically from visual ambiguity.
• Ablation Studies:
  • Adding visual perturbation to existing entropy methods (SE and SNNE) consistently improved their performance, validating the need to probe the visual input space.
  • Replacing raw answer sampling with Prototype Semantic Aggregation (ProtoSem) further improved results, confirming that clustering semantically equivalent answers reduces noise from linguistic variations.

5. Significance and Claims

The paper claims that VSE establishes a new state-of-the-art for uncertainty estimation in VQA by directly addressing the source of ambiguity: the visual input. The authors argue that reliable uncertainty estimation requires:

• Probing the visual input space rather than relying solely on decoding stochasticity.
• Avoiding textual perturbations that introduce prompt sensitivity.
• Aggregating predictions at the semantic prototype level to mitigate linguistic noise.

The authors position VSE as a step toward more trustworthy VLM deployment, particularly in scenarios where visual evidence is ambiguous. They acknowledge limitations, including the additional computational cost associated with perturbation-based sampling and sensitivity to the semantic distance function used for clustering, but emphasize that their approach provides a more faithful reflection of visual uncertainty than current alternatives.