When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

Technical Summary: When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

Problem Statement

Traditional visual grounding benchmarks evaluate Large Vision-Language Models (LVLMs) as a one-shot mapping task: given an image and a sufficiently informative referring expression, the model must localize the target. This formulation relies on a restrictive assumption that the input description is complete and unambiguous. However, real-world reference is often collaborative, iterative, and established through interaction where information may be incomplete, ambiguous, or underspecified. Current benchmarks fail to test whether models can recognize missing information, ask for clarification, or synthesize answers across dialogue turns to ground a target. Consequently, it remains unclear whether LVLM failures in grounding stem from poor visual matching or an inability to perform effective information seeking and synthesis.

Methodology

The authors introduce a controlled evaluation framework for interactive visual grounding that systematically varies the amount of target information provided upfront and the extent to which it must be acquired through dialogue.

1. Task Formalization:
The framework simulates a "Director-Matcher" game.

• Director: A simulated agent (using GPT-5.4) that sees a set of ordered target items and holds a full description of each.
• Matcher: The LVLM being evaluated, which sees a randomized candidate set containing the targets and distractors.
• Process: For each target, the Matcher must select the correct item. The interaction allows for clarification questions, cross-turn integration, and the revision of earlier selections.

2. Interaction Protocols:
The study defines four protocols to isolate specific capabilities:

• STATIC: The Matcher receives a full, human-produced description and selects without interaction. This tests pure visual matching.
• FULL: The Matcher receives the same full description but is allowed optional follow-up questions. This tests the ability to verify, refine, or repair an initial description.
• PARTIAL: The Matcher receives an underspecified attribute (extracted from the full description) and must seek missing information to identify the target.
• NONE: No initial description is provided. The Matcher must construct a target representation entirely through proactive yes/no questions.

3. Datasets and Setup:
The evaluation spans four human-grounded visual contexts: Dogs, Baskets-1, Baskets-2, and Tangrams. These datasets cover real vs. abstract objects, spoken vs. text references, and nameable vs. non-nameable objects.

• Models: Eight LVLMs were tested, including proprietary models (GPT-5.4/5.5, Gemini-3.1-Pro/Flash) and open-weight models (Gemma-4-31B, Qwen3-VL-8B/32B).
• Metrics: Primary metrics include Accuracy (percentage of correct matches) and Question Rate. Secondary metrics include Calibration (comparing reported confidence to empirical accuracy) and Self-repairs (frequency and success of revising selections).
• Scale: The main experiments involved 17,152 target selections across eight models, with follow-up studies expanding the total to 38,848 selections.

Key Results

1. Performance Gap with Human Baselines:
Current LVLMs perform significantly below task-level human baselines across all datasets and protocols. The performance gap is most pronounced in the Tangrams dataset and under the NONE protocol. Even when provided with high-quality human descriptions (STATIC), models struggle to match human-level accuracy, suggesting limitations in comprehending complex referring expressions even without interaction.

2. The Hierarchy of Difficulty:

• STATIC vs. FULL: Interaction generally helps. Seven of eight models improved their accuracy in the FULL protocol compared to STATIC, indicating that follow-up questions can successfully refine or repair initial descriptions.
• PARTIAL: Results were mixed; some models recovered from underspecification, while others declined.
• NONE: All models performed significantly worse in the NONE protocol compared to STATIC, despite asking the most questions. This indicates that proactive, question-driven grounding remains a critical failure point. Models struggle to construct target representations from scratch through dialogue.

3. Model Scaling and Capabilities:

• Larger models generally outperformed smaller counterparts within the same family.
• Gemini-3.1-Pro achieved the best overall performance, particularly in STATIC, FULL, and PARTIAL.
• GPT-5.5 showed competitive performance in the NONE protocol, suggesting stronger capabilities in question-driven information seeking and integration.
• Merely asking more questions was insufficient; models like Gemini-3.1-Pro asked more questions than GPT-5.5 but achieved lower accuracy in NONE, highlighting the need for useful questions and effective answer synthesis.

4. Calibration and Self-Repair:

• Overconfidence: LVLMs are poorly calibrated. They frequently report confidence scores that exceed their empirical accuracy, particularly in PARTIAL and NONE conditions where information is scarce.
• Self-Repair: Models rarely revised their selections (under 2% of matches). When they did, the outcomes were inconsistent; some models corrected errors, while others changed correct selections to incorrect ones or swapped one error for another.

Follow-Up Studies

The authors conducted five follow-up studies to validate the robustness of their findings:

1. AI-Generated Descriptions: Models performed better with AI-generated descriptions than human ones, but the relative difficulty of the protocols (STATIC/FULL > PARTIAL > NONE) remained consistent.
2. Reasoning Effort: Increasing reasoning effort improved performance in FULL but showed diminishing returns in STATIC and NONE, suggesting that scaling compute alone is insufficient for proactive grounding.
3. Repeated Interaction: Models improved over multiple rounds under FULL, showing they can adapt to more efficient referring expressions, but still lagged behind human baselines.
4. Alternative Director: Replacing the GPT-5.4 Director with Gemini-3.1-Flash preserved the relative ranking of Matcher models, confirming that the results are not artifacts of a specific Director model.
5. Face Domain: The protocol-level patterns extended to fine-grained face matching, confirming the generalizability of the findings.

Significance and Claims

The paper claims that interactive visual grounding remains a significant challenge for LVLMs that extends beyond static visual matching. The authors argue that current models lack the integrated capabilities required for real-world reference:

1. Visual Matching: Mapping descriptions to visual targets.
2. Information Seeking: Assessing information sufficiency and asking effective questions.
3. Synthesis: Integrating answers across turns to construct a target representation.

The study demonstrates that while LVLMs can utilize optional clarification to improve performance, they struggle significantly when required to proactively drive the dialogue to acquire missing information. Furthermore, the models' poor calibration suggests they cannot reliably judge when they have sufficient evidence to make a grounding decision. The authors conclude that interactive visual grounding requires a synthesis of visual matching, information seeking, and synthesis, representing a critical frontier for future embodied human-AI interaction.