Technical Summary: SpatialForge

Problem Statement

While Large Vision-Language Models (VLMs) have achieved remarkable success in semantic understanding and visual question answering, they exhibit a critical deficiency in spatial reasoning. Contemporary models often struggle with fundamental geometric tasks, such as determining fine-grained depth ordering (near/far relationships), grounding precise spatial region descriptions, or inferring layouts from alternative viewpoints. This limitation stems from the fact that VLMs are predominantly trained on web-scale image-text pairs that emphasize object-centric semantics rather than topological and geometric relationships.

Existing attempts to address this gap generally fall into two categories, both of which face scalability bottlenecks:

1. Explicit 3D Integration: Incorporating 3D representations (e.g., point clouds, depth maps) into VLM architectures. This requires specialized encoders and multi-modal sensor inputs often unavailable in unconstrained, real-world deployments.
2. Scene-Centric Data Synthesis: Generating spatial QA pairs from limited sets of indoor environments (e.g., ScanNet). While effective, these approaches are constrained by the high cost of acquiring new scenes and annotating fine-grained relationships, resulting in datasets with limited scale and diversity, often biased toward indoor settings.

There is a lack of scalable, high-quality spatial supervision derived from open-world 2D images that can effectively bridge the gap between 2D observations and 3D spatial understanding without requiring architectural modifications.

Methodology: SpatialForge

To overcome these limitations, the authors propose SpatialForge, a scalable, automated data synthesis pipeline that transforms single-view "in-the-wild" 2D images into structured, 3D-aware spatial supervision. The pipeline operates without relying on explicit 3D scene data or multi-modal sensors.

1. Task Taxonomy

The framework decomposes spatial reasoning into two hierarchical cognitive levels, covering six specific tasks:

• Spatial Perception: Focuses on extracting directly observable object-level information.
  • Grounding: Localizing objects based on descriptions.
  • Referring: Identifying/describing objects within a given region.
  • Counting: Counting objects satisfying specific conditions.
• Spatial Relation: Focuses on reasoning about relationships requiring geometric inference beyond direct visual evidence.
  • Near–Far: Determining relative depth ordering.
  • Left–Right: Identifying horizontal relationships relative to a camera or human viewpoint.
  • Perspective: Interpreting spatial relations from a specific human-centric viewpoint (e.g., left/right relative to a person facing the camera).

2. The Synthesis Pipeline

The pipeline consists of four automated stages:

1. Image Filtering: Raw images are filtered for visual quality (removing blur, poor exposure) and semantic realism (using CLIP to distinguish real-world scenes from synthetic content, GUIs, or documents).
2. Data Preprocessing:
   • Global & Region Captioning: A high-capacity VLM generates holistic scene descriptions and extracts object queries. An open-vocabulary detector (Grounding DINO) localizes these objects, and the VLM generates fine-grained, spatially grounded region captions.
   • Depth Estimation: A monocular depth estimator (DepthAnythingV2) predicts dense depth maps. Object-level depth is derived by aggregating statistics (e.g., median depth) within bounding boxes.
   • Human Orientation Estimation: For human subjects, an orientation predictor classifies facing direction (toward or away from the camera) to enable perspective-taking reasoning.
3. Task Workflow: Structured QA pairs are generated based on the preprocessed data.
   • Perception tasks map captions to boxes and vice versa.
   • Relation tasks compute geometric relationships. For depth, pairs are retained only if multiple depth metrics (median and 90th percentile) agree. For perspective, left-right relations are mirrored based on the human subject's facing direction.
4. Quality Inspector: A stronger VLM acts as an independent judge to verify each generated QA pair. For grounding tasks, Intersection over Union (IoU) is checked; for text-based answers, exact string matching is used. Inconsistent samples are discarded to prevent error accumulation.

3. Dataset Construction

Using this pipeline, the authors constructed SpatialForge-10M, a large-scale dataset containing 10.2 million verified spatial QA pairs derived from 2.8 million curated images sourced from Objects365, OpenImages, and Pixmo. The dataset spans diverse open-world environments and covers the six defined spatial tasks.

Key Contributions

1. SpatialForge Engine: A scalable, automated data synthesis engine that extracts structured, 3D-aware spatial supervision from single-view 2D images, mitigating the scalability limits of explicit 3D annotations.
2. SpatialForge-10M Dataset: A large-scale, open-world spatial QA dataset covering spatial perception and spatial relations, designed to improve spatial reasoning through complementary subcategories.
3. Data-Centric Scaling Demonstration: Extensive experiments showing that fine-tuning standard VLMs on this 2D-derived data significantly enhances spatial reasoning capabilities across multiple benchmarks without requiring architectural changes or 3D-specific inputs.

Experimental Results

The authors evaluated the approach by fine-tuning Qwen3-VL-2B on SpatialForge-10M and testing it on five diverse spatial reasoning benchmarks: CV-Bench, SPAR-Bench, SpaCE10, OmniSpatial, and MindCube.

• Performance Gains: The model trained with SpatialForge showed substantial improvements over the baseline Qwen3-VL-2B.
  • SPAR-Bench: Improved from 42.6 to 50.6 (+8.0 absolute gain), with notable gains in depth ordering and spatial imagination.
  • OmniSpatial: Improved from 35.7 to 43.1, indicating better handling of viewpoint-dependent reasoning.
  • MindCube: Improved from 32.2 to 42.3, demonstrating enhanced ability in perspective-taking and mental rotation.
  • CV-Bench (3D): Improved from 83.4 to 85.2.
• Ablation Studies: The study confirmed that:
  • Neither perception nor relation data alone is sufficient; their combination yields the best results.
  • Perspective-aware supervision provides significant additional value, particularly for benchmarks requiring viewpoint switching.
  • The improvements generalize across different task formats rather than overfitting to a specific benchmark.

Significance and Claims

The paper claims that SpatialForge provides a practical and effective direction for enhancing 3D-aware understanding in VLMs. By shifting from hard-to-scale scene-centric annotations to a scalable 2D-driven data synthesis strategy, the framework successfully bridges the gap between semantic understanding and geometric reasoning.

The authors emphasize that their approach:

• Does not require architectural modifications or specialized 3D encoders.
• Does not rely on multi-modal sensor inputs, making it applicable to unconstrained, real-world deployments.
• Leverages the scale and diversity of open-world 2D image collections, overcoming the domain bias and limited scale of previous indoor-centric datasets.

The results suggest that large-scale, well-structured spatial supervision derived from 2D images is a viable and powerful signal for learning spatial awareness, enabling standard VLMs to better capture depth, layout, and viewpoint-dependent relationships.

Limitations

The authors acknowledge several limitations:

• Approximate Supervision: Inferring 3D relationships from single-view 2D images is inherently approximate, potentially affecting accuracy in complex scenarios with occlusion or perspective distortion.
• Pipeline Noise: Despite verification, the multi-stage synthesis process can introduce noise.
• Scope: The current design focuses on a limited set of spatial relations (depth, layout, perspective) and does not yet cover complex physical interactions or temporal dynamics.
• Metric Geometry: Without explicit 3D data, the method may be less suitable for tasks requiring precise metric geometry.