Technical Summary: ViGoR-Bench

1. Problem Statement

Despite the rapid advancement of Artificial Intelligence-Generated Content (AIGC) in visual fidelity, current generative models suffer from a "logical desert." While they can produce photorealistic images and videos, they frequently fail at tasks requiring deep physical laws, causal reasoning, or complex spatial planning.

Existing evaluation metrics (e.g., FID, CLIP-Score) and benchmarks primarily focus on perceptual quality (statistical similarity to real data) or semantic alignment, creating a "performance mirage." These metrics fail to distinguish between a model that truly understands the physical world and one that merely performs high-dimensional probability tiling. Furthermore, current benchmarks are fragmented, often restricted to single modalities (Image-to-Image or Video) and fail to evaluate the generative process (the "how") versus just the final outcome (the "what").

2. Methodology: The ViGoR Framework

The authors introduce ViGoR-Bench (Vision-Generative Reasoning-centric Benchmark), a unified framework designed to rigorously assess the reasoning capabilities of generative models.

A. Data Construction (The Engine)

The benchmark comprises 918 samples across 20 distinct sub-tasks, categorized into three primary reasoning domains:

1. Physical Reasoning: Tasks requiring embodied intelligence (e.g., sorting, spatial reasoning, object assembly, situational decision-making). Data is generated via a hybrid pipeline of generative synthesis and real-world acquisition, verified by human annotators.
2. Knowledge Reasoning: Tasks spanning Biology, Physics, Chemistry, Geography, History, Sports, and Common Sense. Data is curated from authoritative sources to ensure factual accuracy.
3. Symbolic Reasoning: Tasks requiring precise logical manipulation, including Sudoku, Jigsaw Puzzles, Maze Navigation, Block Building (LEGO), Klotski Puzzles, Algebraic Calculation, and Function Plotting.

B. Evaluation Protocol: Dual-Track Mechanism

Unlike traditional benchmarks that only score the final image, ViGoR employs a Dual-Track Process-Outcome Evaluation:

1. Process Metrics: Evaluates intermediate steps (e.g., video frames or CoT sequences). It measures:
   • Background Consistency (SBC): Preservation of the original scene structure.
   • Rule Obey (SRO): Adherence to constraints during the generation process.
   • Visual Quality (SVQ): Clarity and absence of artifacts.
   • Reasoning Accuracy/Beneficial Action (SRA/SBA): Whether intermediate steps logically progress toward the solution.
2. Result Metrics: Evaluates the final output against Ground Truth (GT) using binary scoring (Pass/Fail) for Background Consistency, Rule Obey, Visual Quality, and Reasoning Success (SRS).

C. Automated Judge

To ensure scalability and objectivity, ViGoR utilizes a Multi-Agent "Evidence-Grounded" Judge system (powered by Gemini-2.5-Pro).

• Ground Truth Alignment: The judge is provided with reference images and text descriptions to minimize hallucination.
• Reliability: Meta-evaluation shows the VLM judge achieves high alignment with human experts (MAE ~0.21–0.27, Accuracy ~73–78%) when Ground Truth references are provided, significantly outperforming evaluations without references.

3. Key Contributions

1. Holistic Cross-Modal Coverage: First benchmark to unify Image-to-Image (I2I), Sequential I2I, and Image-to-Video (I2V) tasks under a single reasoning framework.
2. Process-Aware Evaluation: Shifts the paradigm from outcome-only assessment to evaluating the logical trajectory of generation, identifying "Illusions of Reasoning" where models appear logical but fail on constraints.
3. Granular Diagnostic Analysis: Decomposes performance into fine-grained cognitive dimensions (e.g., distinguishing between visual fidelity failures and logical reasoning failures).
4. Evidence-Grounded Automation: Establishes a robust, automated evaluation pipeline that correlates highly with human expert judgment, reducing the subjectivity of "VLM-as-a-Judge."

4. Experimental Results

The authors evaluated over 20 leading models (including OpenAI's Sora 2 Pro, Google's Nano Banana Pro, and various open-source unified models).

Key Findings:

• Proprietary vs. Open-Source: Proprietary models (e.g., Nano Banana Pro, Sora 2 Pro) significantly outperform open-source counterparts, particularly in complex physical and symbolic reasoning.
• The "Illusion of Reasoning": Video generation models often exhibit high Visual Quality and Temporal Consistency (Process metrics) but extremely low Reasoning Success (Result metrics). For example, Kling 1.6 achieved a Process VQ of 77.0% but a Reasoning Success of only 1.6%.
• CoT Limitations: Explicit Chain-of-Thought (CoT) prompting improves interpretability and process scores but does not guarantee improved final accuracy. Models may "think" correctly but fail to "draw" accurately due to execution limitations.
• Complexity Scaling: Performance degrades sharply as problem complexity increases (e.g., larger Sudoku grids or maze sizes). However, models trained on Out-of-Distribution (OOD) and higher-complexity data (e.g., 8x8 mazes) showed superior generalization to simpler tasks compared to those trained on in-distribution data.
• RL vs. SFT: Reinforcement Learning (RL) demonstrated superior potential over Supervised Fine-Tuning (SFT) in advancing reasoning capabilities. A model fine-tuned with RL on 8x8 maze data achieved a 97.0% Reasoning Success score, surpassing state-of-the-art proprietary models.

5. Significance

ViGoR-Bench serves as a critical "stress test" for the next generation of intelligent vision models. It exposes the gap between visual fidelity and true visual intelligence, highlighting that current models are still far from being zero-shot visual reasoners.

• Paradigm Shift: It advocates for a shift in evaluation focus from "how realistic does it look?" to "does it understand the logic?"
• Training Guidance: The benchmark provides actionable signals for model improvement, demonstrating that training on complex, OOD data and utilizing RL can unlock latent reasoning capabilities that SFT alone cannot.
• Safety & Reliability: By identifying logical deficits, ViGoR-Bench contributes to the development of safer, more reliable AI systems suitable for critical applications in education, scientific simulation, and engineering, where physical and causal correctness is paramount.

The benchmark and demo are publicly available at: https://vincenthancoder.github.io/ViGoR-Bench/.