💻 computer science

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

This paper introduces VISTAQA, a comprehensive benchmark and the GROVE evaluation metric designed to jointly assess the correctness of visual question answering and the precision of pixel-level evidence grounding, revealing a significant performance gap in current multimodal models that fail to align answers with visual evidence.

Original authors: Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen, Milan Ganai, Sean Sedwards, Marco Pavone, Krzysztof Czarnecki

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mozhgan Nasr Azadani, Yimu Wang, Yongpeng Zhu, Lihong Chen, Milan Ganai, Sean Sedwards, Marco Pavone, Krzysztof Czarnecki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a detective to solve a mystery based on a single photograph.

In the past, if the detective gave you the right name for the culprit, you would hire them. You didn't care how they knew. They might have guessed based on a hunch, a stereotype, or a lucky guess. If they said, "It was the butler," and they were right, they got a gold star. Even if they pointed at the wrong person in the photo and said, "See? That's the butler!" you still gave them the gold star because the name was correct.

This paper argues that this old way of testing AI is broken. It's like hiring a detective who gets the right answer but can't show you the evidence.

The Problem: The "Lucky Guess" AI

The authors say that current AI models (called Multimodal Large Language Models) are great at guessing the right words. But they are terrible at proving why they are right.

The "Text-Only" Trap: If an AI says, "There are three legs on that dog," but the dog in the picture clearly has four, the AI might still say "three" because it learned from text that dogs usually have four, or it just guessed. If it gets the number right by accident, we reward it.
The "Grounding-Only" Trap: Conversely, some AI models are good at pointing at things in a picture (like drawing a circle around a dog) but are bad at answering questions. They might point perfectly at the dog but say, "This is a cat."

The paper calls this a "modality gap." The AI's brain (text) and its eyes (vision) aren't talking to each other properly.

The Solution: VISTAQA (The "Show Me Your Work" Test)

To fix this, the authors created a new test called VISTAQA.

Think of VISTAQA as a strict teacher who doesn't just grade your final answer; they grade your work.

The Rules: To get a passing grade, the AI must do two things at the exact same time:
1. Answer the question correctly (e.g., "The car is red").
2. Draw a mask (like a highlighter) on the exact pixels of the red car in the image to prove it saw it.

If the AI gets the answer right but highlights the wrong car, it fails. If it highlights the right car but says it's blue, it fails. It must do both perfectly to get a high score.

The Dataset: A Mixed Bag of Challenges

The test isn't just one type of question. The authors built a library of 1,157 expert-curated puzzles covering:

Six Types of Thinking: From simple "What color is this?" (Perception) to complex "Which object is closer to the robot arm?" (Reasoning).
Six Different Worlds: Indoor rooms, outdoor streets, math diagrams, science charts, robot labs, and self-driving car scenes.
The "Trick" Questions: About 27% of the questions are traps. They ask about things that aren't in the picture (e.g., "How many red elephants are in this kitchen?"). A smart AI should say, "There are none," and leave the picture blank. A "hallucinating" AI will try to draw a red elephant that doesn't exist.

The Scorecard: GROVE

How do you grade a test where you have to be right in two different ways? You can't just add the scores. If you get 100% on the text but 0% on the picture, you shouldn't get a high average.

The authors invented a new scoring system called GROVE.

The Analogy: Imagine a two-legged stool. If one leg is broken, the stool falls over. You can't say, "Well, the other leg is perfect, so the stool is fine."
How it works: GROVE multiplies the "Text Score" and the "Picture Score." If either one is zero (or very low), the final score crashes. This forces the AI to be good at both or get a bad grade.

The Results: The AI is Still Learning

The authors tested the smartest AI models available today (including models from Google, OpenAI, and others) on this new, stricter test.

The verdict? Even the best models struggled.

They were great at guessing the right words.
They were okay at pointing at things.
But when asked to do both at the same time, their scores dropped significantly.

The paper concludes that while AI is getting smarter at talking, it still hasn't fully learned to "see" and "prove" its answers simultaneously. There is a huge gap between what the AI says and what it actually sees.

Summary

Old Way: Did you get the right answer? Yes? Good job. (Ignores if you cheated or guessed).
New Way (VISTAQA): Did you get the right answer AND can you point to the proof in the picture?
The Score (GROVE): If you miss either part, you fail.
The Finding: Current AI is still like a student who can memorize the answer key but doesn't understand the lesson. They need to learn to show their work.

Technical Summary: VISTAQA

Problem Statement

Current evaluations of Multimodal Large Language Models (MLLMs) suffer from a fundamental disconnect between textual answer correctness and visual evidence grounding. Standard benchmarks assess performance primarily through textual accuracy, rewarding models that produce correct answers even when those answers are not supported by the appropriate visual evidence (hallucinations) or when the model relies on language priors rather than visual perception. Conversely, existing grounding benchmarks often treat segmentation masks as the primary prediction target rather than as explicit evidence supporting a textual answer, or they evaluate text and localization as independent metrics on separate datasets. This lack of joint evaluation leaves the coupling of reasoning and grounding as an open challenge, failing to determine whether a model's reply is truly grounded in the image—a critical requirement for interpretability and reliability in safety-critical domains.

Methodology

The VISTAQA Benchmark

To address these gaps, the authors introduce VISTAQA, a comprehensive benchmark designed for the joint evaluation of free-form answer correctness and pixel-level evidence grounding in Visual Question Answering (VQA).

Dataset Composition: VISTAQA comprises 1,157 expert-curated samples. Unlike automated generation pipelines which the authors found unreliable for creating complex image-question-mask triplets, VISTAQA was constructed using a rigorous multi-stage pipeline involving domain-appropriate image sourcing, hybrid mask generation, LLM-assisted question generation, and multi-round human verification.
Scope: The benchmark spans six task types (Identification, Attribute, OCR, Spatial, Counting, Comparison) and six visual domains (Indoor, Outdoor, Autonomous Driving, Robotics, Science, Mathematics).
Hallucination Awareness: Approximately 27% of the samples (314) are specifically designed as hallucination-aware cases where the queried entity is absent or the question is unanswerable, requiring models to correctly identify the absence of valid visual evidence.
Annotation: Each sample includes a question, a reference answer, and a segmentation mask specifying the visual evidence required to support that answer.

The GROVE Evaluation Metric

The paper introduces GROVE (GROunded eVidence Evaluation), a unified metric designed to enforce joint correctness.

Joint Sensitivity: A prediction is considered fully correct only when both the textual answer and the corresponding evidence mask are valid.
Mechanism: GROVE computes a per-sample geometric mean of smoothed text and mask scores.
- Text Score ( $S_a$ ): A binary correctness signal (0 or 1) determined via an LLM-as-a-judge protocol (using Qwen 2.5-14B) to evaluate semantic correctness rather than string matching.
- Mask Score ( $S_m$ ): Measures the fidelity of predicted segmentation masks using Intersection over Union (IoU) with ground truth, normalized by the maximum number of instances to penalize both missed and spurious predictions. It also handles hallucination cases by rewarding empty predictions when no entity exists.
- Smoothing: To prevent score collapse (where a single failure zeroes out the entire score), a floor parameter $\epsilon$ (empirically set to 0.1) is applied to both scores before computing the geometric mean: $S = \sqrt{S'_a \cdot S'_m}$ .
Design Goal: This formulation ensures that neither dimension (text or mask) can compensate for deficiencies in the other, preventing models from achieving high scores by excelling in only one modality.

Key Contributions

VISTAQA Benchmark: The first systematic benchmark requiring models to produce correct free-form answers explicitly supported by pixel-level visual evidence across six diverse tasks and domains, including explicit hallucination scenarios.
GROVE Metric: A unified evaluation metric that jointly measures answer correctness and grounding fidelity, enforcing simultaneous satisfaction of both modalities through a geometric mean approach.
Comprehensive Evaluation: Extensive experiments across state-of-the-art baselines, including dedicated reasoning segmentation models (e.g., LISA, SESAME, GLaMM) and hybrid pipelines pairing frontier MLLMs with segmentation models (e.g., Qwen3-VL + SAM3, GPT-5.4 + SAM3).

Experimental Results

The authors evaluated 14 models on VISTAQA without fine-tuning. The results highlight a substantial gap between answer accuracy and visual evidence alignment:

Low Overall Performance: Even the strongest systems achieve limited performance under GROVE. The top-performing model, GPT-5.4-T + SAM3, achieved a GROVE score of only 45.53%.
Modality Gap: While some hybrid pipelines achieved relatively high individual text accuracy (e.g., >60%) or mask IoU scores, the joint GROVE score remained low. This indicates that correct answers and accurate evidence often occur on different samples; models do not consistently align the two.
Task and Domain Challenges:
- Spatial reasoning and OCR emerged as the most challenging tasks.
- Mathematics was consistently the most difficult domain for all model families, likely due to the abstraction required to ground symbolic reasoning in pixels.
- Hallucination Robustness: Segmentation models tended to predict masks even when no valid entity was present, while some hybrid pipelines over-predicted "no mask," leading to trade-offs between hallucination detection and grounding quality on valid samples.
Grounding Complexity: Models generally degraded in performance when moving from single-instance to multi-instance grounding scenarios, with dedicated grounding models showing the most severe drops.

Significance and Claims

The paper claims that VISTAQA and GROVE expose a fundamental limitation in current multimodal reasoning: textual correctness alone cannot determine whether a reply is truly grounded in the image.

Revealing the Gap: The benchmark demonstrates that even the most advanced MLLMs struggle to align their answers with correct visual evidence simultaneously. High performance on traditional VQA or segmentation benchmarks does not guarantee reliable, grounded reasoning.
Evaluation Rigor: By preventing compensation between modalities, GROVE provides a more rigorous assessment of model capabilities, particularly for safety-critical applications like autonomous driving and robotics where hallucinations can have severe consequences.
Future Direction: The authors posit that VISTAQA serves as a foundation for future research in grounded multimodal evaluation, emphasizing the need for models that can not only reason but also explicitly justify their reasoning with visual evidence.

The paper acknowledges limitations, noting that the static nature of the benchmark does not cover video or multi-image grounding, and that the construction of high-quality samples remains labor-intensive due to the current unreliability of models in autonomously generating such triplets.