💻 computer science

X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras

X-Lens is a compact, real-time feed-forward model that achieves robust metric depth estimation from heterogeneous fisheye and pinhole cameras by leveraging learnable calibration tokens and a Jacobian-parameterized distortion bias, all trained on a newly released large-scale synthetic dataset called OmniScene.

Original authors: Heng Zhou, Shuhong Liu, Yonghao He, Bohao Zhang, Fa Fu, Chenhui Hou, Xianbao Hou, Lijun Han, Wei Sui

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Heng Zhou, Shuhong Liu, Yonghao He, Bohao Zhang, Fa Fu, Chenhui Hou, Xianbao Hou, Lijun Han, Wei Sui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D map of the world using a camera backpack. Most robots and self-driving cars wear a backpack with a mix of lenses: some are standard "pinhole" cameras (like your phone) that see far away clearly, and others are "fisheye" lenses that see everything around you in a giant, curved bubble.

The problem? These two lenses speak different geometric languages. The fisheye lens stretches the edges of the image like a funhouse mirror, while the pinhole lens keeps things straight. For a long time, computers struggled to mix these two views together to figure out exactly how far away objects are in real-world meters. They either got the math wrong, needed massive supercomputers to do it, or had to flatten the curved images into a weird "panorama" that lost important details.

Enter X-Lens, a new, tiny, and super-fast AI model that acts like a master translator for these mixed cameras.

The Magic Translator

Think of X-Lens as a compact, 0.04-billion-parameter "brain" (that's tiny compared to other models that weigh in at over 1 billion parameters). Instead of trying to force the fisheye and pinhole images to look the same, X-Lens accepts them exactly as they are.

It uses two clever tricks to make sense of the chaos:

Learnable Calibration Tokens: Imagine these as little sticky notes the AI sticks onto the fisheye images. These notes tell the AI, "Hey, this part of the image is stretched because of the lens, so don't trust the shape too much." This helps the AI align the curved fisheye view with the straight pinhole view without squishing the image first.
Jacobian Distortion Bias: This is like a special compass for the AI. It calculates exactly how the light rays are bending at every tiny spot in the image. By feeding this "bending map" into the AI's attention system, the model learns to ignore the distortion and focus on the actual 3D shape of the world.

The result? X-Lens can look at a mix of six cameras (four fisheye, two pinhole) and instantly spit out a dense depth map with a real-world scale. It doesn't just guess "near" or "far"; it tells you an object is exactly 11.07 meters or 31.64 meters away. And it does this at 41 frames per second, which is fast enough for a robot to dodge obstacles in real-time.

The Giant Practice Field: OmniScene

To teach X-Lens how to do this, the researchers couldn't just use existing photos because there weren't enough examples of mixed cameras with perfect 3D measurements. So, they built OmniScene, a massive synthetic world.

They created 103 different scenes (from shopping malls to sci-fi complexes) and generated 266,000 synchronized frames using a virtual six-camera rig. This is like training a pilot in a flight simulator that has seen every possible weather condition and runway before they ever touch a real plane. The data includes 1.7 million individual images with perfect, noise-free depth labels.

What the Numbers Say

The paper tested X-Lens against the biggest, heaviest models in the field. Here is what happened:

Speed: X-Lens runs at 41 FPS on a single high-end GPU, while the strongest competitors often run at 5 to 13 FPS.
Size: X-Lens uses 0.04B parameters. The next best competitor, MapAnything, uses 1.23B parameters. That means X-Lens is 88.9% smaller than the baseline.
Accuracy: On the mixed-camera test (OmniScene-Full), X-Lens reduced the error (AbsRel) by 25.4% compared to the strongest baseline. It also improved the "Scale AbsRel" (how well it guesses the real-world size) by 68.1% compared to MapAnything.

What X-Lens is NOT

It is important to know what this model doesn't do, because the paper is very clear about its limits:

It does not predict camera settings: X-Lens needs to be told exactly what the camera lenses look like (the calibration) before it starts. It cannot guess the lens type or the camera position on its own.
It is not a "fix-all" for weird lenses: The model was trained on specific types of fisheye and pinhole lenses. If you feed it a lens with an extreme, never-before-seen field of view that is totally different from the training data, the paper suggests the performance might drop slightly because it hasn't seen that specific "funhouse mirror" before.
It is not a general 3D reconstruction tool: Unlike some massive models that try to guess the camera's pose, the lens type, and the 3D map all at once, X-Lens is specialized. It focuses strictly on depth and scale, which is why it is so fast and small.

The Bottom Line

X-Lens proves that you don't need a giant, slow computer to understand 3D space with mixed cameras. By using a smart, geometry-aware design and training on a massive, high-quality synthetic dataset, it achieves state-of-the-art accuracy while being small enough to run on edge devices. It's a step toward robots that can truly "see" the world in 3D, in real-time, without needing a supercomputer in their backpack.

Technical Summary: X-Lens

Problem Statement

Modern embodied systems, including autonomous robots and Vision-Language-Action (VLA) models, increasingly rely on multi-camera rigs combining narrow-FOV pinhole cameras and wide-FOV fisheye cameras to achieve comprehensive spatial coverage. However, existing 3D depth estimation and geometric foundation models face significant limitations in this heterogeneous-camera setting:

Model Mismatch: Most methods are built on pinhole-centric assumptions or single-camera models, failing to handle the non-uniform projection geometry and view-dependent distortion of mixed rigs.
Computational Cost: Existing geometric foundation models are often computationally demanding, making real-time edge inference difficult under strict latency and memory budgets.
Data Scarcity: There is a lack of large-scale, synchronized datasets containing dense metric ground truth for heterogeneous (fisheye + pinhole) multi-view configurations.
Inefficiency: Current approaches often rely on panorama stitching (altering native geometry) or auxiliary reconstruction targets (increasing optimization complexity) to handle wide-FOV inputs.

Methodology

X-Lens is a compact, feed-forward model designed for real-time metric depth estimation from a variable number of calibrated fisheye and pinhole views. It avoids panoramic stitching and auxiliary reconstruction targets, focusing directly on predicting dense depth and a global metric scale.

1. Formulation and Design

Generic Ray Representation: Instead of relying on a specific pinhole intrinsic matrix $K$ , X-Lens abstracts cameras using a generic unprojection map $G$ . This maps pixel coordinates to back-projected unit rays in a shared ray space, accommodating both pinhole and fisheye (parametric distortion) models without architectural changes.
Factored Prediction: The model outputs a triplet:
- $\hat{D}$ : Normalized depth field (scale-invariant).
- $\hat{C}$ : Self-calibrated confidence map.
- $\hat{m}$ : A single global metric scalar.
- The final metric depth is reconstructed as $\hat{D}_m = \hat{m} \cdot \hat{D}$ . This isolates absolute scale into a one-dimensional pathway, allowing geometric losses to operate in a scale-invariant space.

2. Heterogeneous Projection Transformer

The core architecture is a Vision Transformer (ViT) backbone with alternating local (within-view) and global (cross-view) layers, enhanced by two key components:

Learnable Calibration Tokens: To prevent lens-specific distortions from corrupting shared visual tokens, the model injects layer- and camera-type-specific calibration tokens ( $\Theta$ ) into the sequence. These tokens attend locally to their specific view to correct distortion patterns but are dropped before the next layer to prevent persistence, ensuring they do not interfere with global metric fusion.
Jacobian-Parameterized Distortion Bias: To bridge the geometric gap between views, a bias term is injected into the cross-attention mechanism. This bias is derived from the Jacobian of the unprojection map (local ray direction and projection distortion). It modulates attention weights to favor patches with compatible 3D ray geometry and local projection structure, rather than relying solely on appearance similarity in distorted coordinates.
Scale Attention: To ensure robust metric scale prediction, the model discards the lowest 25% confidence pixels (often near fisheye boundaries) and performs confidence-weighted pooling on the remaining reliable regions before regressing the global scale $\hat{m}$ .

3. Multi-Stage Training Strategy

X-Lens employs a progressive three-stage training pipeline to isolate lens-specific adaptation from joint optimization:

Stage 1 (Pinhole Pre-training): The model is trained on multi-view pinhole data to learn robust geometric priors and scale awareness. Calibration tokens and distortion bias are disabled.
Stage 2 (Fisheye Token Adaptation): The backbone is frozen, and only the calibration tokens are updated using fisheye data. This adapts the model to wide-FOV distortion without overwriting the established pinhole priors.
Stage 3 (Heterogeneous Joint Training): The Jacobian distortion bias is activated, and the model is jointly fine-tuned on mixed fisheye-pinhole data (including the new OmniScene dataset) and pure pinhole data to reconcile cross-view interactions.

4. The OmniScene Dataset

To address data scarcity, the authors introduce OmniScene, a large-scale synthetic dataset:

Scale: ~266,000 synchronized six-view frames (1.7M images) across 103 indoor and outdoor scenes.
Configuration: A rigid six-camera rig with four fisheye (180° FOV) and two pinhole cameras.
Quality: Provides dense, noise-free metric ground truth (z-depth), validity masks, and calibrated intrinsics/extrinsics for every frame.

Key Results

Extensive experiments on real-world (KITTI360, ETH3D, ScanNet++) and synthetic (OmniScene) datasets demonstrate X-Lens's superiority in accuracy and efficiency:

Heterogeneous Performance: On the OmniScene-Full benchmark (6 views: 4 fisheye + 2 pinhole), X-Lens reduces AbsRel by 25.4% compared to the strongest baseline (MapAnything) while using 88.9% fewer parameters (0.04B vs. 1.23B).
Real-Time Inference: X-Lens achieves 41 FPS on single fisheye views and 22 FPS on full heterogeneous 6-view inputs on an NVIDIA H100 GPU, significantly outperforming larger models in speed.
Generalization: The model demonstrates strong zero-shot generalization on unseen environments and maintains competitive performance on homogeneous pinhole-only and fisheye-only settings.
Parameter Efficiency: It achieves state-of-the-art metric depth accuracy with only 0.04B parameters, a fraction of the size of other geometric foundation models (e.g., VGGT, DepthAnything3).

Significance and Claims

The paper claims that X-Lens represents a significant step toward real-time spatial intelligence for embodied systems. Its primary contributions are:

Native Heterogeneous Support: It is the first compact model to natively support raw, mixed fisheye and pinhole inputs without requiring panorama stitching or hardware-specific stereo volumes.
Efficiency-Accuracy Trade-off: By decoupling depth perception from full 3D reconstruction (pose, intrinsics, point maps) and focusing on metric depth and scale, X-Lens achieves a balance of high accuracy and low latency suitable for edge deployment.
Data Resource: The release of OmniScene provides a critical resource for training and evaluating heterogeneous multi-view depth perception, addressing a major gap in the field.

The authors position X-Lens not just as a depth estimator, but as a geometry-aware visual encoder capable of bridging the gap between metric depth modeling and real-time downstream applications like robotic planning and interaction. They acknowledge limitations regarding the reliance on ground-truth calibration and potential sim-to-real gaps with extreme, unseen lens configurations, suggesting future work in temporal aggregation and broader intrinsic diversity.