Technical Summary: Which Pretraining Paradigm Better Serves Spatial Intelligence?

Problem Statement

Spatial intelligence is a critical component for embodied AI, autonomous driving, and robotics, requiring visual representations that simultaneously capture semantic object information and geometric structure. Currently, two dominant pre-training paradigms serve as foundation backbones for these tasks:

1. Vision-Language Models (VLMs): Utilize language supervision to align visual observations with semantic concepts.
2. Video Generation Models (VGMs): Learn from temporally evolving visual worlds, potentially acquiring priors on dynamics and geometric consistency.

While both families are increasingly used as perception backbones, it remains unclear which paradigm provides a superior representation substrate for spatial intelligence. Previous comparative studies often evaluate complete policies (e.g., in Vision-Language-Action frameworks), where performance is entangled with action decoders, robot data mixtures, and post-training recipes. This paper addresses the need to isolate the frozen visual representations themselves to determine what spatial information is inherently encoded before any downstream fine-tuning.

Methodology

The authors propose a controlled frozen-feature probing study to compare VLMs and VGMs across three complementary axes of spatial intelligence. The core design involves freezing the foundation models and training lightweight probes with a unified architecture to read out specific information.

1. Experimental Setup

• Models: The study evaluates representative models from both families:
  • VGMs: WAN2.1 (T2V/I2V), CogVideoX, OpenSora-2.0, and Aether.
  • VLMs: InternVL3/3.5, Qwen2.5-VL, and Qwen3-VL.
• Feature Extraction:
  • A 76-frame video context is sampled.
  • For VGMs, features are extracted from hidden activations of selected transformer blocks during a single denoising pass (using empty text conditions or image-to-video conditioning). Temporal compression via VAEs aligns the 76 frames to 20 latent positions.
  • For VLMs, 20 query frames are fed directly, and visual-token hidden states are collected from language-model layers.
• Unified Probing Backbone: A lightweight, identical transformer backbone (N-layer alternating-attention) processes the sampled features. It performs frame-level attention followed by global attention across frames.
• Task-Specific Readout Heads: Three independent heads are trained on top of the frozen backbone:
  1. Semantic Tagging: A multi-label classifier predicting the presence of object categories (ScanNet20 vocabulary) in the video.
  2. Instance Grouping: A pixel-wise projection head trained with a multi-view contrastive pull-push loss to group pixels belonging to the same object instance across views.
  3. 3D Geometry Prediction: Dense heads (DPT-style) for point map and depth prediction, plus a camera head for pose estimation, supervised by VGGT-generated ground truth (DL3DV dataset).

2. Evaluation Metrics

• Semantic Tagging: mAP, APmid (performance on less frequent categories), and Mid Ratio.
• Instance Grouping: T-mIoU (mean IoU across views) and T-SR (success rate of grouping instances in all views).
• 3D Geometry: P-map Err. (point map error after alignment), AbsRel (relative depth error), and AUC@30 (camera pose accuracy).

Key Results

1. Clear Division of Strengths

The experiments reveal a distinct complementarity between the two model families rather than a single superior paradigm:

• VLMs excel at Semantics and Instances:
  • Semantic Tagging: VLMs significantly outperform VGMs (Average mAP: 92.08% vs. 69.89%). They are particularly strong on less frequent categories (APmid: 87.28 vs. 58.63).
  • Instance Grouping: VLMs also lead in grouping pixels across views (T-mIoU: 22.66 vs. 13.24), suggesting that object-centric semantic representations are crucial for instance permanence, not just geometric continuity.
• VGMs excel at 3D Geometry:
  • Geometry Prediction: VGMs achieve superior results in recovering dense geometry and camera motion (Lower P-map Err: 0.152 vs. 0.223; Lower AbsRel: 0.072 vs. 0.113; Higher AUC@30: 0.527 vs. 0.330).
  • This indicates that video-generation pre-training inherently encodes spatial consistency and geometric priors more effectively than language-aligned training.

2. Naive Feature Fusion

The authors tested a "naive" feature-level fusion where normalized features from a strong VGM (WAN2.1-T2V-14B) and a strong VLM (Qwen3-VL-8B) were concatenated and fed into the same probe.

• Result: The fused representation retained the VLM's semantic advantage (surpassing the VGM in tagging and approaching the VLM in grouping) while recovering the VGM's geometric strength (matching or slightly exceeding the VGM in depth and camera metrics).
• Implication: This demonstrates that the strengths of both families are complementary and can be combined to create a representation that excels at both semantics and geometry.

3. Stability of Findings

Ablation studies on probe depth (varying the capacity of the lightweight readout) showed that the relative ranking of models remained stable. This confirms that the observed differences are due to the information encoded in the frozen features themselves, not the capacity of the probe to learn tasks from scratch.

Significance and Claims

The paper claims that spatial intelligence is not captured by a single axis.

• Complementarity: Current foundation models exhibit a trade-off: VLMs provide strong semantic and object-centric representations, while VGMs provide accessible dense geometric and motion signals.
• Methodological Contribution: By isolating frozen representations, the study avoids confounding factors from downstream fine-tuning, offering a clearer view of what pre-training schemes inherently encode.
• Future Direction: The success of naive feature fusion suggests a promising path for building stronger spatial-intelligence backbones by effectively integrating language-aligned object semantics with generation-induced geometric representations.

The authors conclude that future spatial-understanding systems should not rely solely on recognition (VLMs) or reconstruction (VGMs) but should aim to integrate both paradigms to achieve robust spatial intelligence.