Technical Summary: OmniSpace

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated strong performance in 2D visual tasks but struggle with the geometric intelligence required for Autonomous Vehicle (AV) applications. Current approaches typically fall into two categories:

1. Pure 2D MLLMs: These treat AV tasks as standard 2D autoregressive problems. While efficient, they lack explicit geometric transparency, burying critical information like depth, scale, camera pose, and cross-view correspondence within image tokens.
2. MLLMs with External 3D Models: These integrate auxiliary 3D geometric models (e.g., depth estimators) at inference time. While effective, this introduces pipeline complexity, increases computational cost, and creates a risk of cascading failures if the external 3D model performs poorly.

The authors identify that current MLLMs are bottlenecked by two specific geometric deficiencies: weak depth estimation (inability to infer metric scene structure) and poor cross-view correspondence (inability to associate scene elements across different camera views). Diagnostic analysis reveals a strong correlation between performance on these geometric tasks and downstream AV planning and risk detection capabilities.

Methodology: OmniSpace

OmniSpace is proposed as a plug-and-play, geometry-aware adaptation framework that enhances spatial reasoning directly within the MLLM without relying on external 3D models during inference. It consists of three core components:

1. Camera Pose Injector

To inject spatial priors into visual tokens, the authors introduce a mechanism that encodes each feature map token with its corresponding Plücker ray representation.

• Mechanism: Instead of naively representing a ray as a camera center and direction (which is redundant), the model uses the Plücker coordinate $P = (o \times d, d) \in \mathbb{R}^6$. This representation is invariant to the choice of the point along the ray, ensuring geometric consistency.
• Implementation: The Plücker ray map is projected via a lightweight MLP and added to the visual features. The projector is zero-initialized, allowing the model to behave like the base MLLM initially and gradually learn to inject camera geometry.

2. Multi-view Epipolar Attention

To enforce cross-view consistency, the authors replace standard global cross-view attention with epipolar-constrained attention.

• Mechanism: For a token in one view, the model computes epipolar lines on other views based on camera calibration. Attention is restricted to tokens lying on these epipolar lines (plus a 3x3 dilation for robustness).
• Effect: This forces the model to associate corresponding scene elements across views based on geometric plausibility rather than arbitrary feature similarity, while preserving unrestricted self-attention within a single view. Like the injector, this module is added via a zero-initialized residual connection.

3. 3D Geometric Distillation with Temporal Aggregation

To equip the model with depth understanding, OmniSpace uses a 3D foundation model (VGGT) as a teacher, but only during training.

• Challenge: Directly applying 3D foundation models to single-frame AV scenes yields noisy supervision due to limited inter-view overlap and dynamic objects.
• Solution: The authors introduce temporal geometric aggregation. A symmetric temporal window of multi-view frames ($2K+1$) is fed into the 3D teacher. The resulting teacher features are averaged along the temporal dimension to create a stable, high-quality geometric target.
• Distillation: The student MLLM's visual tokens are projected to the teacher's feature dimension and trained to minimize the L2 distance between student and aggregated teacher features. The 3D teacher is discarded entirely at inference time.

Key Contributions

1. Diagnostic Investigation: The authors identified depth estimation and cross-view correspondence as the primary geometric bottlenecks for MLLMs in AV tasks and validated their correlation with downstream performance through a new benchmark (Geo-Bench).
2. OmniSpace Framework: A novel architecture integrating Camera Pose Injection, Multi-view Epipolar Attention, and 3D Geometric Distillation. This approach internalizes geometric awareness without requiring external 3D modules at inference.
3. Empirical Validation: Extensive experiments demonstrate that OmniSpace improves planning accuracy, risk detection, and language understanding across multiple benchmarks (nuScenes, Bench2Drive, nuInstruct, OmniDrive, DriveBench) while maintaining the inference efficiency of pure 2D MLLMs.

Experimental Results

The paper evaluates OmniSpace on five autonomous driving benchmarks:

• Open-loop Planning (nuScenes): OmniSpace consistently outperforms baseline MLLMs and methods relying on test-time 3D models (e.g., SpaceDrive, VGGDrive) in trajectory error (L2), collision rate, and intersection rate. Notably, it achieves these gains with significantly higher throughput (FPS) compared to methods using external 3D models.
• Closed-loop Planning (Bench2Drive): OmniSpace achieves a Driving Score of 79.65 and a Success Rate of 60.15% (using Qwen2.5-VL), surpassing previous state-of-the-art methods. The Qwen3-VL variant achieves the best performance among MLLM-based methods on this benchmark.
• Risk Detection (nuInstruct): The model achieves superior performance in risk perception (highest MAP and Accuracy, lowest MAE) without auxiliary 3D models, addressing a known weakness in standard MLLMs.
• Language Understanding (OmniDrive): OmniSpace improves captioning quality (CIDEr, ROUGE) over baselines, confirming that geometric distillation enhances spatial reasoning without degrading linguistic capabilities.
• Generalization (DriveBench): Fine-tuning on DriveLM and testing on DriveBench shows that OmniSpace generalizes better than baselines across challenging scenarios.

Ablation Studies confirm that each component (Pose Injector, Epipolar Attention, Distillation) contributes uniquely to performance. Specifically, Plücker ray encoding outperforms naive ray encoding, and epipolar attention outperforms vanilla cross-view attention. Temporal aggregation with $N=4$ frames is found to be the optimal trade-off between geometric supervision quality and computational cost.

Significance and Claims

The paper claims that OmniSpace offers a simple yet effective paradigm for achieving geometry-aware spatial reasoning in MLLMs. Its primary significance lies in:

• Decoupling Inference Complexity from Geometric Awareness: By internalizing geometric priors during training via distillation, OmniSpace avoids the computational overhead and deployment risks associated with external 3D models at inference time.
• Direct Enhancement of Spatial Intelligence: It demonstrates that MLLMs can acquire robust 3D scene understanding (depth and cross-view correspondence) directly, rather than relying on 2D visual autoregression or external pipelines.
• Efficiency and Performance Trade-off: The framework provides a competitive or superior accuracy-efficiency trade-off, making it a viable solution for real-world autonomous driving systems where both safety (geometric accuracy) and latency (inference speed) are critical.

The authors acknowledge limitations, noting that the method assumes accurate camera calibration and relies on a specific 3D teacher (VGGT), suggesting future work could explore ego-motion-aware aggregation and AV-pretrained 3D foundation models.