Technical Summary: VRIQ – Benchmarking and Analyzing Visual-Reasoning IQ of VLMs

Problem Statement

Despite significant advancements in Vision-Language Models (VLMs), their ability to perform reliable nonverbal reasoning remains unproven. While reasoning capabilities are a cornerstone of Large Language Model (LLM) performance, extending these benefits to multimodal contexts has revealed persistent limitations. Existing benchmarks often suffer from two critical gaps:

1. Lack of Domain Parity: Many benchmarks focus exclusively on natural images with shallow reasoning requirements or abstract puzzles disconnected from real-world visual understanding, preventing direct comparison of model behavior across domains.
2. Monolithic Evaluation: Current evaluations typically treat perception (identifying visual elements) and reasoning (applying rules to those elements) as a single capability, failing to isolate the specific source of failure.

Consequently, it remains unclear whether VLM failures in complex visual tasks stem from an inability to extract visual facts (perception) or an inability to apply logical rules (reasoning).

Methodology: The VRIQ Benchmark

The authors introduce VRIQ (Visual Reasoning IQ), a diagnostic benchmark comprising 1,500 expert-authored items designed to evaluate VLMs on both perception and reasoning.

1. Benchmark Design

VRIQ employs a parallel domain construction strategy, featuring two families of puzzles that share identical logical structures but differ in visual domain:

• Abstract Domain: Geometric primitives, lines, and patterns (e.g., civil-service aptitude tests).
• Natural Domain: Real-world objects and scenes designed to require the same logical operations as the abstract counterparts.

The benchmark covers five fundamental reasoning categories established in human psychometrics:

• Sequence Completion (SC)
• Matrix Prediction (MP)
• Odd One Out (OO)
• Figure Rotation (FR)
• 3D Visualization (3D)

2. Hierarchical Diagnostic Framework

To dissect model failures, VRIQ utilizes a three-tier evaluation framework:

• Tier 1 (End-to-End Accuracy): Measures standard multiple-choice accuracy across all categories.
• Tier 2 (Diagnostic Probes): For each puzzle, the authors construct two types of probes:
  • Perceptual Probes (P-probes): Atomic, human-trivial questions testing specific visual attributes (e.g., "How many red circles?") using the original image.
  • Reasoning Probes (R-probes): Text-only instantiations of the puzzle's rule where visual facts are explicitly provided. This tests if the model can apply the logic given correct facts.
• Tier 3 (Error Categorization): Failures are classified into three distinct buckets:
  • P-only: The model fails a P-probe but passes the R-probe (Perception bottleneck).
  • R-only: The model passes all P-probes but fails the R-probe (Reasoning deficit).
  • P+R: The model fails both (Combined failure).

3. Experimental Setup

The study evaluates a diverse set of open-source (e.g., Qwen2.5/3, InternVL, LLaVA) and proprietary (e.g., GPT-4o, GPT-5.1, Gemini-2.5-pro) models. A special case is made for OpenAI o3, which is evaluated with "thinking with images" enabled, allowing it to use external tools (cropping, zooming, rotation, Python code) during inference. A human baseline was also established using 20 participants on a stratified subset.

Key Results

1. Poor Performance on Abstract Reasoning

VLMs perform near-random levels on abstract IQ questions.

• Abstract Domain: The average accuracy across all tested VLMs is 27.52%, only marginally above the random guess baseline (25%). Even the best-performing model, Qwen3-VL-32B-Thinking, achieves only 35.5%.
• Proprietary vs. Open Source: Contrary to trends in other benchmarks, proprietary models (e.g., GPT-4o at 30.3%) do not significantly outperform strong open-source models on abstract tasks.
• Human Baseline: Humans achieve 90.36% accuracy on the abstract subset, highlighting a massive gap.

2. Improved but Unreliable Natural Reasoning

Performance improves when tasks are grounded in natural images, but remains far from human-level.

• Natural Domain: Average accuracy rises to 42.53%, with gains of 10–20 percentage points over abstract tasks.
• Reliability: Despite improvements, models remain unreliable. For instance, GPT-4o achieves 54.2% on average, while open-source models cluster between 35% and 47%. Performance varies wildly across categories (e.g., a model might score 19% on 3D reasoning but 24% on sequence completion).

3. The Impact of Tool-Augmented Reasoning

The OpenAI o3 model, equipped with tool-use capabilities (image manipulation, code execution), demonstrates a substantial leap in performance:

• Abstract: 50.2% average accuracy (vs. ~28% for others).
• Natural: 85.0% average accuracy.
• Mechanism: Analysis of tool traces shows o3 frequently uses cropping and zooming to isolate elements and rotation to canonicalize orientation, effectively mitigating perceptual bottlenecks.

4. Diagnostic Findings: Perception is the Bottleneck

The hierarchical probe analysis reveals the root cause of failures:

• Perception Dominance: Approximately 56% of failures are attributed to perception alone (P-only), and 43% are combined perception-reasoning failures (P+R).
• Reasoning Rarity: Pure reasoning failures (R-only) account for only 1.1% of errors.
• Implication: When models correctly perceive visual elements, they reliably apply logical rules. The primary failure mode is the inability to extract accurate visual facts (counting, position, orientation) from the image.

5. Fine-Grained Perception Deficits

Specific perceptual categories show distinct failure profiles:

• Counting: Highest failure rate (73.8%).
• Position & Rotation: High failure rates (58.2% and 52.4% respectively).
• Color: Lowest failure rate (31.4%), suggesting models are better tuned for appearance-based attributes than spatial or quantitative ones.
• 3D/Depth: High failure rate (67.5%), indicating a lack of strong 3D geometric priors.

Significance and Claims

The paper claims that current VLMs, even with advanced architectures and tool-augmented reasoning, remain unreliable abstract reasoners. The primary limitation is not a lack of logical reasoning capacity but a perceptual bottleneck.

Key contributions include:

1. A Novel Benchmark: VRIQ provides the first multidimensional benchmark enabling direct comparison of abstract vs. natural reasoning using identical logical structures.
2. Diagnostic Framework: The hierarchical probe methodology successfully disentangles perception from reasoning, revealing that the vast majority of "reasoning" errors are actually perception errors.
3. Actionable Insights: The findings suggest that improving VLM performance requires prioritizing perceptual grounding (e.g., better spatial encoding, counting mechanisms, and 3D priors) rather than simply scaling model size or reasoning capacity. The paper posits that future progress in multimodal reasoning may depend more on orchestrating external procedures (tools) to handle perception than on further scaling single models.

The authors conclude that by shifting from descriptive to diagnostic evaluation, VRIQ offers a principled basis for developing more robust and reliable multimodal systems.