Technical Summary: V-REX

Problem Statement

Current Vision-Language Models (VLMs) excel at passively answering well-defined, straightforward questions with specified targets but struggle with complex, open-ended tasks requiring active, multi-step exploration. Existing benchmarks often focus on visual reasoning in constrained environments (e.g., math problems, puzzle games) where the input prompt is sufficient for the model to formulate a plan immediately without further exploration. Consequently, these benchmarks fail to evaluate a model's ability to:

1. Propose helpful sub-questions (exploring the question space) to guide reasoning.
2. Adapt plans dynamically based on collected clues in an open environment.
3. Understand intermediate steps, as most evaluations rely solely on final answer accuracy, treating the reasoning process as a "black box."

This gap limits the deployment of VLMs in real-world scenarios where goals are not fully specified at the outset and require progressive planning and active information gathering (e.g., detective work, geolocation, or complex scene analysis).

Methodology: V-REX and Chain-of-Questions (CoQ)

To address these limitations, the authors introduce V-REX (Visual Reasoning with multi-step EXploration), a benchmark and evaluation suite designed to assess exploratory visual reasoning. The core innovation is the Chain-of-Questions (CoQ) framework, which decomposes the reasoning process into a sequence of interconnected sub-questions and answers.

The CoQ Framework

Instead of evaluating open-ended generation, CoQ reformulates the evaluation into a tractable, finite space by presenting models with multiple-choice options at each step. This allows for a fine-grained disentanglement of two distinct capabilities:

1. Planning (Question Space Exploration):

   • Task: Given an original question and a history of previous sub-questions/answers, the model must select the most informative next sub-question from a set of candidates (including ground-truth questions and distractors).
   • Goal: Evaluate the model's ability to strategically chart an exploration course, identifying which information is necessary to solve the task.
   • Mechanism: Distractors are generated via a two-stage pipeline (step-level and chain-level) using LLMs to create contextually relevant but logically unhelpful questions.

2. Following (Answer Space Exploration):

   • Task: Given a pre-constructed, ground-truth chain of sub-questions, the model must answer each question sequentially based on visual cues and accumulated context.
   • Goal: Evaluate the model's ability to faithfully execute a reasoning path and extract specific information from the image.
   • Mechanism: Each step is converted into a multiple-choice question with distractor answers.

Benchmark Construction

• Dataset: V-REX comprises 702 samples and 2,504 questions, with an average of 3.57 reasoning steps per sample.
• Taxonomy: The dataset spans 4 reasoning categories and 15 real-world scenarios:
  • Deduction: Flowchart, Pattern, Property, Relationship.
  • Guessing: Responsibility, Intention, Location, Time, Topic.
  • Navigation: Map, GUI, Traffic, Trend.
  • Retrieval: Counting, Word Puzzle.
• Data Curation: Ground-truth QA chains were constructed by PhD-level annotators with cross-verification. Distractors were generated using GPT-5 and filtered to ensure they are plausible yet misleading.

Key Contributions

1. First Benchmark for Exploratory Reasoning: V-REX is the first benchmark to explicitly evaluate the multi-step exploratory reasoning capability of VLMs using the CoQ formulation.
2. Disentangled Evaluation: It separates Planning (selecting the direction of inquiry) from Following (answering the inquiry), providing a more interpretable diagnosis of model weaknesses than end-to-end accuracy alone.
3. Controlled Evaluation Protocol: By converting open-ended exploration into a finite multiple-choice setting, V-REX enables reliable quantitative analysis of intermediate steps, which is otherwise infeasible due to the vast exploration space.

Experimental Results

The authors evaluated 32 VLMs, ranging from small open-source models (<7B) to large proprietary models (e.g., GPT-5, o3).

Key Findings

1. Importance of Exploration: When provided with CoQ hints (either correct questions in the Planning task or correct answers in the Following task), VLMs consistently achieved higher accuracy on final questions, validating the necessity of structured exploration.
2. Scaling Laws and Variance: Scaling laws hold for V-REX; performance generally increases with model size. However, Planning performance exhibits significantly higher variance among models of the same size compared to Following performance. This suggests that while models converge on the ability to answer questions, their ability to plan the exploration path remains a distinguishing factor.
3. Correlation with Overall Performance: Both Planning and Following abilities are positively correlated with overall end-to-end performance (Pearson correlation of 0.858 and 0.948, respectively).
4. Model Size Imbalance: Smaller models (<7B) are significantly better at Following than Planning. As model size increases, the gap narrows, and larger models exhibit more balanced performance across both dimensions.
5. Recovery from Failure: Models are more robust to failed Planning steps (selecting a wrong sub-question) than to failed Following steps (answering a sub-question incorrectly). Errors in the answer space are more likely to propagate to an incorrect final conclusion than errors in the question selection space.

Specific Observations

• Retrieval Tasks: The CoQ process provided less benefit for Retrieval tasks (e.g., counting, word puzzles), which rely more on direct visual matching than hierarchical reasoning. In some cases, CoQ even hindered performance by introducing unnecessary cognitive noise.
• Category Differences: Larger models showed more generalized capabilities across categories, while smaller models exhibited specialized strengths (e.g., excelling in Deduction but failing in Navigation).

Significance and Claims

The paper claims that V-REX establishes a foundation for advancing deliberate, stepwise visual reasoning. By revealing that current VLMs often rely on brute-force search and lack adaptive planning, the authors highlight effective planning as a critical bottleneck for future VLM development.

The authors emphasize that V-REX provides a "fine-grained diagnosis" of reasoning capabilities, moving beyond the "black box" of final answer accuracy. The benchmark is designed not as a prompting method to improve performance, but as a rigorous evaluation tool to identify specific weaknesses in how models explore visual spaces. The results suggest that while scaling improves general reasoning, the ability to strategically plan multi-step exploration remains a distinct and challenging capability that requires targeted improvement.