Technical Summary: Masked Depth Modeling for Spatial Perception (LingBot-Depth)

1. Problem Statement

Accurate 3D spatial perception is critical for physical-world applications such as autonomous driving and robotic manipulation. While RGB-D cameras are the only modality capable of providing real-time, absolute metric scale, and pixel-aligned dense geometry, they suffer from inherent hardware limitations. Specifically, stereo matching algorithms used in these sensors are highly susceptible to appearance ambiguities, leading to severe data corruption and missing values (holes) in challenging scenarios like low-texture surfaces, specular reflections, and complex lighting.

Current approaches often treat these missing measurements as noise to be discarded. However, this paper argues that these inaccuracies are not merely errors but "masked" signals that inherently reflect underlying geometric and appearance ambiguities. The core challenge is to leverage these naturally occurring missing regions to learn robust, metric-scale depth representations that can complete the geometry even where sensors fail.

2. Methodology: Masked Depth Modeling (MDM)

The authors propose LingBot-Depth, a depth completion model based on a novel Masked Depth Modeling (MDM) framework. This approach adapts the principles of self-supervised masked image modeling (e.g., MAE) but shifts the objective from appearance reconstruction to depth map prediction using RGB-D inputs.

2.1 Core Architecture

The model utilizes a Vision Transformer (ViT-Large) encoder and a ConvStack decoder.

• Separated Patch Embedding: The RGB image and the depth map are processed through separate patch embedding layers to generate aligned RGB and depth tokens.
• Positional Embeddings: The system employs both shared 2D spatial positional embeddings and modality-specific embeddings (distinguishing RGB from depth tokens) to capture spatial location and input source identity.
• Joint Embedding: The encoder learns a joint representation by attending to the full, unmasked RGB context and the remaining valid depth tokens.
• ConvStack Decoder: Unlike standard MAE which uses a shallow Transformer decoder for pixel reconstruction, LingBot-Depth uses a hierarchical ConvStack decoder. This is better suited for dense geometric prediction. The decoder discards latent depth tokens and reconstructs the full depth map from the enhanced contextual tokens (RGB features enriched with global scene context via the [cls] token).

2.2 Masking Strategy

A key innovation is the interpretation of sensor failures as "natural masks."

• Natural Masking: Instead of random masking, the model masks depth tokens corresponding to missing or invalid sensor measurements.
• Adaptive Masking: For patches containing a mix of valid and invalid values, a high probability (e.g., 0.75) of masking is applied. If the natural mask ratio is insufficient, additional valid tokens are randomly masked to reach a target ratio of 60–90%.
• Learning Signal: The model is forced to infer missing depth values by reasoning about the correlation between the complete RGB context and the sparse valid depth tokens. This unifies the objectives of monocular depth estimation (when all depth is masked) and depth completion (when only invalid tokens are masked).

2.3 Data Curation Pipeline

To support large-scale MDM training, the authors built a scalable data pipeline generating 3 million RGB-D pairs (2M real, 1M synthetic):

• Synthetic Branch (LingBot-Depth-S): Uses self-hosted 3D assets in Blender to render RGB images, perfect depth, and stereo pairs. Crucially, it simulates real-world sensor artifacts by applying Semi-Global Matching (SGM) to the stereo pairs to generate "sensor-like" depth maps with natural imperfections.
• Real-World Branch (LingBot-Depth-R): Utilizes a modular, 3D-printed capture rig compatible with various consumer RGB-D cameras (e.g., Intel RealSense, Orbbec Gemini, ZED). Pseudo-depth labels are generated via stereo matching on captured IR pairs, with consistency checks to filter errors.
• Supplementary Data: The training set is augmented with 7 open-source RGB-D datasets, with masking strategies adapted to their specific data characteristics.

3. Key Results

3.1 Depth Completion

LingBot-Depth was evaluated on standard benchmarks (iBims, NYUv2, DIODE, ETH3D) under two protocols: block-wise depth masking and sparse SfM inputs.

• Performance: The model consistently outperformed state-of-the-art methods (OMNI-DC, PromptDA, PriorDA) across all difficulty levels.
• Metrics: On indoor benchmarks, it reduced RMSE by over 40% compared to the best competitor in extreme settings. On the challenging ETH3D sparse SfM protocol, it achieved state-of-the-art results, reducing RMSE by 47% (indoor) and 38% (outdoor).
• Qualitative: The model produces sharper depth boundaries and more coherent structures, particularly in regions with severe occlusion or sparse observations.

3.2 Monocular Depth Estimation

When used as a pretraining backbone for MoGe (replacing DINOv2), the MDM-pretrained encoder demonstrated superior performance in monocular depth estimation across 10 diverse benchmarks (indoor, outdoor, synthetic, and real-world).

• Generalization: The model showed consistent improvements in affine-invariant, scale-invariant, and disparity-invariant metrics, suggesting that MDM effectively distills 3D geometric knowledge into the encoder.

3.3 FoundationStereo Integration

Using LingBot-Depth as a prior for FoundationStereo resulted in:

• Faster Convergence: The model achieved strong performance by epoch 5, significantly outperforming vanilla baselines.
• Stability: It avoided the training instability observed in MoGe-based variants.
• Final Accuracy: It achieved the best or comparable results across all tested benchmarks (ETH3D, Middlebury, Booster, etc.) by epoch 15.

3.4 Downstream Applications

The paper demonstrates the utility of LingBot-Depth in zero-shot or minimal-finetuning scenarios:

• Video Depth Completion: Despite being trained on static images, the model produces temporally consistent depth streams. It successfully fills missing regions in transparent and reflective surfaces (e.g., glass walls, aquarium tunnels) where both structured-light and stereo sensors fail.
• 3D Point Tracking: Integrated into SpatialTrackerV2, the refined depth enables smoother and more accurate camera trajectory estimation and dynamic object tracking in scenes with extensive glass surfaces.
• Robotic Grasping: In a real-world dexterous grasping task (Rokae arm + X Hand-1), the model enabled successful grasping of challenging objects (transparent glassware, reflective bowls) that defeated raw sensor depth. Success rates improved significantly (e.g., 17/20 vs. 13/20 for steel cups; 16/20 vs. 12/20 for transparent cups).

4. Significance and Claims

The paper claims that LingBot-Depth represents a paradigm shift in handling RGB-D sensor limitations. By treating sensor failures as "natural masks" rather than noise, the model learns to infer geometry from visual context, effectively bridging the gap between sparse/invalid sensor data and dense, metric-accurate 3D perception.

The authors emphasize that their approach:

1. Surpasses Hardware: The model outperforms top-tier commercial RGB-D cameras in both depth precision and pixel coverage, particularly in difficult lighting and material conditions.
2. Unifies Tasks: The MDM framework serves as a versatile generalist, capable of functioning as both a depth completion tool and a monocular depth estimator depending on the masking strategy.
3. Enables Robotic Utility: The robust depth predictions and aligned latent representations allow for reliable manipulation of objects that typically defeat conventional sensors, demonstrating immediate practical value for spatial perception in robotics.

The authors release the code, checkpoints, and the curated 3M RGB-depth dataset to the community to facilitate further research in spatial perception.