🤖 AI

Vanishing Depth: Training Generalized Depth Adapters with Sinusoidal Depth Preprocessing for Pretrained RGB Encoders

This paper proposes a self-supervised approach that extends pretrained RGB encoders with a sinusoidal depth-encoded adapter to achieve generalized, robust metric depth understanding and superior performance across various RGB-D downstream tasks without requiring fine-tuning.

Original authors: Paul Koch, Jörg Krüger

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paul Koch, Jörg Krüger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, highly trained robot eye (a "pretrained RGB encoder") that is excellent at recognizing objects, colors, and shapes in a 2D photograph. It knows what a "chair" looks like, but it has no idea how far away that chair is. It sees a flat world.

For a robot to actually do things—like pick up a cup or navigate a room—it needs to understand depth (how far away things are in 3D space). Usually, to give a robot this skill, you have to retrain its entire brain from scratch, which is slow, expensive, and often ruins the clever things it already learned about shapes and colors.

This paper introduces a clever "plug-in" solution called Vanishing Depth and Sinusoidal Depth Preprocessing. Here is how it works, using simple analogies:

1. The "Depth Adapter" (The Universal Translator)

Think of the robot's existing brain as a master chef who knows how to cook amazing 2D pictures of food. The authors built a small, detachable adapter (a "Depth Adapter") that plugs into the chef's kitchen.

What it does: It takes the chef's 2D knowledge and mixes it with new 3D depth information.
The Magic: It doesn't force the chef to forget how to cook. Instead, it teaches the chef to also understand distance. The original brain stays exactly as it was (unspoiled), but now it can see the world in 3D.
The Benefit: You don't need to retrain the whole chef. You just plug in the adapter, and the robot is ready to work on new tasks like segmentation (cutting out objects), estimating where objects are (pose estimation), or filling in missing depth data.

2. The "Vanishing Depth" Training (The Blindfold Game)

How do you teach this adapter to understand depth without just memorizing specific pictures? The authors used a game called "Vanishing Depth."

Imagine you are teaching someone to recognize a mountain range. Instead of showing them a perfect photo, you:

Hide parts of the photo: You cover up 50% of the mountain with a blindfold (random noise).
Change the scale: You zoom in and out randomly, so the mountain looks huge or tiny.
Shift the position: You move the mountain left or right.
The Goal: The student (the AI) has to look at the visible parts and the shape of the mountain to guess what the hidden parts look like.

By doing this with thousands of different depth maps, the AI learns the true structure of depth rather than just memorizing specific pictures. It learns to handle missing data, noisy sensors, and different distances, making it incredibly robust.

3. "Sinusoidal Depth Preprocessing" (The Ruler with Infinite Marks)

Standard ways of teaching AI about depth are like using a ruler that only has marks for 1 meter, 2 meters, and 3 meters. If you show it something at 1.5 meters, it gets confused.

The authors invented a new way to measure depth called Sinusoidal Depth Preprocessing (SDP).

The Analogy: Imagine a ruler that doesn't just have straight lines, but a smooth, wavy pattern (like a sine wave) that repeats over and over.
Why it's better: This wavy pattern allows the AI to understand depth as a smooth, continuous flow rather than jumping between fixed numbers. It can handle tiny differences (like 1.001 meters) and huge differences (like 500 meters) with the same ease.
The Result: This makes the AI much more precise and stable, especially when the depth data is messy or sparse (like a laser scanner that misses a few points).

What Did They Prove?

The authors tested this "adapter" on a wide variety of tasks, and it performed better than almost everything else currently available, without needing any extra training (finetuning) for those specific tasks.

Segmentation: When asked to identify and outline objects in a room, their system achieved a top-tier score (56.05 mIoU on the SUN-RGBD dataset), beating other complex, multi-modal systems.
Pose Estimation: When asked to figure out exactly how an object is rotated in 3D space, their system significantly outperformed previous methods.
Robustness: Even when the depth data was noisy, missing, or had strange distortions, their system kept working well, whereas other systems crashed or gave bad answers.

The Bottom Line

This paper presents a "universal depth plug-in." It takes a smart, 2D-aware AI and instantly upgrades it to be 3D-aware. It does this by teaching the AI to understand depth through a game of "fill in the blanks" (Vanishing Depth) and by using a super-precise, wavy measuring tool (Sinusoidal Preprocessing). The result is a robot vision system that is accurate, robust, and ready to work on new tasks immediately, without needing to be retrained from scratch.

Technical Summary: Vanishing Depth: Training Generalized Depth Adapters with Sinusoidal Depth Preprocessing for Pretrained RGB Encoders

Problem Statement

Current state-of-the-art (SOTA) vision foundation models, particularly those based on self-supervised learning (SSL) like DINOv2, excel at extracting generalized features from RGB images. However, they lack inherent metric depth understanding, a capability critical for downstream tasks requiring absolute 3D position information, such as robotics, pose estimation, and depth completion. Existing approaches to integrate depth often rely on:

Task-specific finetuning: Which can corrupt the generalized features learned during pretraining.
Random initialization of depth encoders: Leading to a lack of generalizability across different datasets.
Multi-modal pretraining: Methods like OmniVec or MultiMAE align modalities but often fail to enforce pixel-wise metric depth precision, focusing instead on general context for classification or segmentation.

Consequently, there is a gap in the literature for a generalized RGBD encoder that can incorporate metric depth into a pretrained RGB latent space without requiring task-specific finetuning, while remaining robust to variations in depth density, distribution, and noise.

Methodology

The authors propose a framework consisting of two primary components: a Depth Adapter (DA) and Sinusoidal Depth Preprocessing (SDP), trained via a Vanishing Depth pipeline.

1. Depth Adapter (DA) Architecture

Base: The method extends existing pretrained RGB encoders (e.g., ViT-B based on DINOv2) by adding a parallel depth encoder branch.
Fusion: It employs a "Skip & Embed" (S&E) mechanism to fuse RGB and depth signals at specific intermediate layers (e.g., layers 3, 6, 9, 12). This allows the depth encoder to leverage the generalized semantic features of the frozen RGB encoder.
Training Strategy: The pretrained RGB encoder is frozen to preserve its generalized features. Only the depth adapter and the decoder heads are trained. This ensures the model does not "unlearn" the robust RGB representations.
Decoder: A lightweight U-Net decoder with four Feature Pyramid Network (FPN) heads performs multi-scale decoding for dense metric depth prediction.

2. Sinusoidal Depth Preprocessing (SDP)

Motivation: Standard depth normalization (e.g., scaling to [0,1] or using disparity maps) struggles with varying depth distributions, densities, and perturbations.
Mechanism: SDP encodes continuous depth values $d_j$ into a set of sinusoidal features:
$SDP(d_j, 2i) = \sin(l/T^{2i/c}), \quad SDP(d_j, 2i+1) = \cos(l/T^{2i/c})$
where $l = 2\pi(d_j / \max(d))$ .
Normalization: It supports both Global Max Depth (GSDP) and Local Max Depth (LSDP). For LSDP, the maximum depth value is encoded into a vector and added to the CLS token, allowing the model to decode metric depth dynamically.
Benefit: This approach provides stable feature extraction within the range of [-1, 1] regardless of the input depth distribution or absolute scale, unlike classical positional encodings which are designed for token localization.

3. Vanishing Depth Training Pipeline
To ensure the model learns a depth density and distribution invariant representation, the training pipeline employs a "vanishing depth" strategy inspired by inpainting:

Data Augmentation: Depth maps are manipulated using Perlin noise and random resized noise to remove varying percentages (1–99%) of depth pixels.
Distribution Invariance: Random offsets, scalars, and rescaling are applied to depth samples to prevent the model from relying on monocular depth cues from RGB alone.
Loss Function: A Multi-scale Balanced Scale-Invariant (SI) Loss is used. It balances the reconstruction of existing depth pixels with the prediction of missing (masked) depth pixels, ensuring the model performs well on both dense and sparse inputs.

Key Contributions

Generalized Depth Adapter: A method to extend pretrained RGB encoders to handle metric depth without finetuning the base encoder, enabling "out-of-the-box" adaptation to various RGBD tasks.
Sinusoidal Depth Preprocessing (SDP): A novel preprocessing technique that outperforms standard normalization in handling diverse depth distributions, densities, and noise levels.
Vanishing Depth Training: A self-supervised pipeline that trains the adapter to be robust against missing data and distribution shifts by simulating sparse and noisy depth inputs during training.
Zero-Finetuning Adaptability: The resulting model can be applied to downstream tasks (segmentation, pose estimation, depth completion) using only the frozen encoder and a lightweight decoder head, avoiding the computational cost and overfitting risks of full finetuning.

Experimental Results

The authors evaluated their approach across segmentation, depth completion, and 6D object pose estimation on datasets including SUN-RGBD, NYU-v2, KITTI, Void, YCBV, and others.

Segmentation: The proposed method achieved 56.05 mIoU on the SUN-RGBD dataset, outperforming SOTA depth-aware and multi-modal encoders (e.g., OmniSegmentor, DFormerV2) and significantly improving upon the RGB baseline (DINOv2). It also demonstrated effectiveness on RGB-only datasets where depth was synthesized via monocular estimation.
Depth Completion: On sparse depth completion tasks (KITTI, Void), the SDP-based adapters showed superior precision compared to norm-based depth encoding and other SOTA methods, particularly in handling sparse inputs.
Pose Estimation: In 6D object pose estimation (YCBV, HomeBrew, LM-O), the depth adapters significantly improved the performance of frozen RGB encoders (e.g., DINOv2 + DA achieved 84.1 AUC on YCBV vs. 72.5 for DINOv2 alone). The method outperformed multi-modal baselines like Omnivore and MultiMAE, which often degraded when depth was introduced.
Robustness: Ablation studies confirmed that SDP is more resilient to noise and sparsity than standard normalization. While normalization-based methods struggled with random noise and missing depth, SDP maintained stability.

Significance and Claims

The paper claims that this work bridges the gap between generalized foundation models and metric depth understanding. By avoiding task-specific finetuning, the method offers a modular and efficient solution for robotics and multi-agent systems where rapid adaptation to new environments is required.

Generalizability: The approach is not limited to specific datasets; it generalizes across varying depth densities and distributions.
Efficiency: It allows for the reuse of large, pretrained RGB encoders, reducing the need for training massive multi-modal models from scratch.
Robotics Relevance: The authors highlight the potential for Vision-Language-Action (VLA) and robot policy learning, where pretrained encoders are beneficial but often require finetuning that leads to overfitting. This method provides a way to include depth awareness without compromising the generalization capabilities of the base encoder.

The authors modestly note that while they avoided VLA and policy learning benchmarks due to the lack of standardized "out-of-the-box" comparison frameworks, the architecture is designed to support these applications in the future. They also acknowledge that for extreme depth ranges (e.g., >15m), rescaling or adjusting the maximum depth parameter is necessary, though the frequency density of SDP may stretch in such cases.