VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

1. Problem Statement

Vision-Language Models (VLMs) have achieved significant success in general-purpose tasks but continue to struggle with low-level visual perception, including spatial understanding, viewpoint recognition, depth ordering, and attribute discrimination.

• Root Cause: The authors hypothesize that natural image-text datasets provide insufficient supervision for these specific skills. Natural corpora lack systematic coverage of the combinatorial variations required for robust visual perception (e.g., specific spatial relations, depth cues, or viewpoint changes) and often rely on language priors rather than visual evidence.
• The Challenge: How can we generate targeted supervision to fix these weaknesses without relying on expensive human annotation or reference images?

2. Methodology: VisionFoundry Pipeline

The authors introduce VisionFoundry, a fully automated, task-aware synthetic data generation pipeline. It requires only a task keyword (e.g., "Depth Order" or "Viewpoint") as input and produces high-quality image-question-answer (VQA) triplets. The pipeline operates in a closed loop with three main stages:

A. Task-Aware Generation (LLM)

• Input: A task specification (e.g., "Spatial Understanding").
• Process: A Large Language Model (LLM, specifically GPT-5.2 in the experiments) constructs an adaptive concept pool (objects, attributes, scenes, styles).
• Output: The LLM generates a triplet consisting of:
  1. A Question with a deterministic answer.
  2. A Text-to-Image (T2I) Prompt that explicitly encodes the visual facts required to answer the question.
  3. Design Principle: The prompt is engineered to ensure "visual determinism," meaning the answer is fully derivable from the image content alone, minimizing reliance on language priors.

B. Image Synthesis (T2I Model)

• Process: A modern T2I model (Gemini-2.5-Flash-Image) generates images based strictly on the LLM-generated prompts.
• Refinement: If an initial image fails verification, the system can perform limited iterative refinement (localized edits) to better satisfy the visual statement without altering the question/answer pair.

C. Alignment Verification (Multimodal Judge)

• Process: A strong proprietary multimodal model (Gemini-3-Pro) acts as a judge.
• Mechanism: The question and answer are converted into a declarative visual statement (e.g., "The red cube is to the left of the blue sphere"). The judge verifies if the generated image aligns with this statement.
• Filtering: Only samples where the judge confirms alignment are retained. This step filters out subtle misalignments (missing objects, incorrect spatial relations) that would otherwise poison the training data.

3. Key Contributions

1. VisionFoundry Pipeline: A novel, end-to-end framework for generating synthetic VQA data from task keywords alone, eliminating the need for reference images or human labeling.
2. VisionFoundry-10K Dataset: A curated dataset of 10,000 image-question-answer triples spanning 10 specific low-level visual perception tasks:
   • Orientation and Direction
   • Viewpoint and Perspective
   • Positional and Relational Context
   • Spatial Relationship
   • State and Condition
   • Structural and Physical Characteristics
   • Color and Appearance
   • Depth Order
   • Relative Distance
   • Real-World Spatial Understanding
3. Systematic Evaluation: The authors demonstrate that targeted synthetic supervision can be generated at scale and effectively improves VLM capabilities without degrading general reasoning.

4. Experimental Results

The authors evaluated the approach on three open-source VLMs: Qwen2.5-VL-3B, MiMo-VL-7B, and Llama-3.2-11B-Vision.

Performance Gains

Training on VisionFoundry-10K yielded substantial improvements on visual perception benchmarks:

• MMVP (Visual Perception): +7% improvement.
• CV-Bench-3D (3D Spatial Reasoning): +10% improvement.
• CV-Bench-2D: +5.1% improvement.
• RealWorldQA: +1.9% improvement.

Generalization and Scaling

• Preservation of Capabilities: The models retained their broader capabilities (OCR, GUI grounding, general reasoning). While some benchmarks showed slight fluctuations, there was no broad degradation. Notably, MiMo-VL-7B showed a massive gain on MMBench (from 50.5 to 81.6), suggesting improved visual perception compensates for weaker logical reasoning in non-thinking modes.
• Data Scaling: Performance on visual perception benchmarks showed a clear positive correlation with data size. Increasing the synthetic dataset from 0.5k to 10k samples resulted in consistent gains.
• Synthetic vs. Natural Mix: An equal-sized mixture of synthetic and natural data outperformed pure natural subsets on visual perception tasks, indicating that synthetic data provides complementary signals difficult to obtain from limited real data.

Ablation Studies

• Verification Necessity: Removing the verification step resulted in lower performance, confirming that the automated filtering is critical for data quality.
• Image Source: Synthetic images generated via the pipeline outperformed natural images paired with synthetic QA, suggesting the visual generation process itself (caption-conditioned T2I reconstruction) adds value beyond just the text supervision.

5. Significance and Future Implications

• Addressing the Perception Bottleneck: The paper provides strong evidence that the visual perception bottleneck in VLMs is partly a data problem (lack of targeted supervision) rather than just a model architecture issue.
• New Training Paradigm: VisionFoundry proposes that verifier-filtered synthetic images can serve as a "first-class primitive" for VLM post-training, similar to how synthetic text data is used for LLMs.
• Systematic Coverage: Unlike natural data, which is biased toward common scenes, synthetic data can systematically cover rare, specific, and combinatorial visual scenarios (e.g., specific depth orders or viewpoints) that are crucial for robust perception.
• Automation: The fully automated nature of the pipeline (no human annotation) makes it scalable and cost-effective for continuously patching capability gaps as new evaluation benchmarks emerge.

In conclusion, VisionFoundry demonstrates that targeted, synthetic supervision is a highly effective strategy for enhancing the visual perception capabilities of VLMs, offering a promising path toward more systematic and robust multimodal training.