Any to Full: Prompting Depth Anything for Depth Completion in One Stage

Any2Full is a one-stage, domain-general framework that reformulates depth completion as a scale-prompting adaptation of pretrained monocular depth estimation models via a Scale-Aware Prompt Encoder, achieving superior robustness and efficiency by eliminating the computational overhead and distortions of traditional two-stage alignment methods.

The Gist

Imagine you are trying to build a 3D model of a room, but your depth sensor (like a camera that measures distance) is broken. It only gives you a few scattered dots of information, leaving huge gaps where it couldn't see. This is the problem of Depth Completion: taking a sparse, messy, incomplete map and filling in the blanks to make a smooth, solid 3D picture.

For a long time, robots and computers struggled with this because their "filling-in" tools were too rigid. They learned to guess the missing parts based on specific training data. If the lighting changed, or if the sensor was a different type, the robot would get confused and the 3D model would look warped or wrong.

Enter Any2Full, a new method that solves this problem with a clever, one-step trick. Here is how it works, using some everyday analogies:

1. The Old Way: The "Two-Step" Construction Crew

Imagine you are trying to repair a broken fence.

• The Old Method (Two-Stage): First, a crew builds a rough, wobbly wooden frame to guess where the fence might be. Then, a second crew comes in to sand it down and paint it to look nice.
• The Problem: The first crew often gets the shape wrong because they are guessing based on a specific type of wood they've seen before. By the time the second crew tries to fix it, the foundation is already flawed. The result is a fence that looks okay up close but wobbles in the wind (distorted geometry).

2. The New Way: The "Smart Architect" (Any2Full)

The Any2Full team realized they didn't need two crews. They had a Master Architect (a pre-trained AI called "Depth Anything") who has seen millions of rooms and knows exactly how 3D space usually looks. This architect has a perfect "feel" for geometry, but they don't know the exact size of your specific room yet.

• The Analogy: Think of the Master Architect as someone who can draw a perfect sketch of a house from a photo, but they don't know if the house is a dollhouse or a mansion. They just know the shape.
• The Input: You give them a few scattered measurements (the broken depth sensor data).
• The Magic Trick (Scale Prompting): Instead of asking the architect to guess the whole house from scratch, you simply whisper to them: "Hey, the distance between these two dots is 2 meters."
• The Result: The architect instantly adjusts their perfect sketch to match that specific scale. They don't need to rebuild the whole thing; they just tweak the "size knob" based on your hint.

3. The Secret Sauce: The "Scale-Aware Prompt Encoder"

The paper introduces a special tool called the Scale-Aware Prompt Encoder. Think of this as a translator or a conductor.

• The Challenge: Your broken sensor data is messy. Sometimes it has holes (missing spots), sometimes it's just a few random dots, and sometimes the dots are clustered in weird places.
• The Solution: The translator looks at these messy dots and extracts the essential rhythm of the space (the "scale cues"). It ignores the noise and the holes, focusing only on the relationships between the dots.
• The Action: It turns this messy rhythm into a clean, simple instruction (a "prompt") and hands it to the Master Architect. The Architect then uses their deep knowledge of how rooms should look to fill in the rest of the picture perfectly, guided by your instruction.

Why is this a Big Deal?

1. It's a "One-Stage" Wonder: The old methods tried to build a rough draft and then fix it. Any2Full does it in one smooth motion. It's like going from sketching a rough draft and then painting it, to just painting the final masterpiece in one go. This makes it faster (1.4x speedup) and more accurate.
2. It's "Pattern-Agnostic": Whether your sensor is missing 10% of the data, 90% of the data, or has giant holes in the middle, the translator knows how to handle it. It doesn't matter what the "broken" pattern looks like; the Master Architect just needs the scale hint.
3. Real-World Impact: The paper tested this in a real robotic warehouse. The robots were trying to grab black boxes. Black boxes are tricky because they absorb light, causing the sensors to go blind (creating "holes" in the data).
   • Before: The robot would miss the box or crush it because it couldn't "see" the edges. Success rate: ~28%.
   • After (with Any2Full): The robot could perfectly reconstruct the shape of the black box from the few dots it did see. Success rate: 91.6%.

In Summary

Any2Full is like giving a master artist a few scattered paint splatters and a single instruction on how big the canvas is. Instead of trying to guess the whole picture from scratch, the artist uses their innate knowledge of art to instantly fill in the rest, creating a perfect, detailed 3D image in a single, lightning-fast stroke. It works for any room, any lighting, and any broken sensor, making robots much better at seeing and interacting with the world.

Technical Summary

1. Problem Statement

Depth Completion aims to recover dense, metric depth maps from sparse, incomplete measurements (e.g., from LiDAR, ToF, or structured light sensors) combined with RGB images. While crucial for robotics and autonomous navigation, existing methods face two primary limitations:

• Domain Specificity: Traditional RGB-D fusion methods (e.g., CompFormer) learn priors jointly conditioned on specific training RGB distributions and depth patterns. They fail to generalize when faced with visual domain shifts (lighting, texture) or heterogeneous depth sensor patterns (varying sparsity, missing regions).
• Inefficiency and Distortion in MDE Integration: Recent approaches leverage Monocular Depth Estimation (MDE) models (like Depth Anything) for their strong geometric priors. However, current integration strategies typically use a two-stage pipeline:
  1. Predict a coarse metric depth by explicitly aligning relative MDE output with sparse metric inputs.
  2. Refine the result.
     This explicit alignment introduces structured distortions because relative depth predictions often exhibit non-linear scale inconsistencies. Correcting these in a second stage is computationally expensive and often results in artifacts, especially on unseen depth patterns (e.g., range truncation).

2. Methodology: Any2Full

The authors propose Any2Full, a one-stage, domain-general, and pattern-agnostic framework that reformulates depth completion as a scale-prompting adaptation of a pretrained MDE model.

Core Concept

Instead of generating an intermediate coarse depth map, Any2Full treats the sparse depth input as a scale prompt. It distills scale cues from the sparse input to guide the pretrained MDE model toward producing scale-consistent relative depth predictions, which are then converted to metric depth via a simple global alignment.

Key Components

1. Scale-Aware Prompt Encoder (SAPE):
   This is the core innovation, designed to handle varying sparsity levels and irregular spatial distributions (e.g., random holes). It consists of two hierarchical modules:

   • Local Enrichment Module: Uses Feature-wise Linear Modulation (FiLM) to couple scale cues from the sparse depth features with the dense geometric features of the MDE backbone. This anchors scale cues within the MDE latent space, making features robust to sparsity variations.
   • Global Propagation Module: Addresses irregular spatial distributions by diffusing scale cues across the scene. Crucially, it uses MDE geometry features as the Query and Key in a Transformer attention mechanism, using the scale cues only as the Value. This ensures that scale propagation follows the geometric structure defined by the RGB image (from the MDE) rather than being biased by the irregular sampling pattern of the sparse depth.
   • Scale Prompt Fusion: The resulting unified scale prompts are injected into the MDE decoder at multiple semantic levels via FiLM, modulating the features to produce a scale-consistent relative depth map.

2. One-Stage Inference:
   The framework performs a single forward pass. The final metric depth is obtained by applying a non-parametric least-squares fit (global scale and bias) to the predicted relative depth. This avoids the overhead of intermediate refinement networks.

3. Training Strategy:

   • Data: Trained on high-quality synthetic datasets (Hypersim, VKITTI2, TartanAir) with diverse depth sampling strategies (Random Sampling and Hole Sampling) to ensure robustness against various sparsity patterns.
   • Loss Functions: A combination of Scale- and Shift-Invariant Loss (Lssi), Gradient Matching Loss (Lgm) for edge preservation, Sparse Anchor Consistency (Lanchor), and Relative Structure SSIM (Lr-ssim).

3. Key Contributions

• One-Stage Paradigm: Introduces a novel formulation that bypasses intermediate coarse depth prediction, eliminating the structural distortions and computational overhead associated with two-stage alignment methods.
• Scale-Aware Prompt Encoder: Designs a lightweight encoder that transforms sparse, irregular depth cues into unified, pattern-invariant scale prompts, effectively decoupling scale estimation from specific depth patterns.
• Domain and Pattern Generalization: Achieves state-of-the-art zero-shot generalization across diverse visual domains (indoor/outdoor) and depth patterns (holes, range limits, varying sparsity) without task-specific fine-tuning.

4. Experimental Results

The authors evaluated Any2Full on six public benchmarks (NYU-Depth V2, iBims-1, KITTI DC, DIODE, ETH3D, VOID) and a real-world robotic warehouse dataset (Logistic-Black).

• Performance:
  • Outperforms the current state-of-the-art (OMNI-DC) by 32.2% in average AbsREL across all scenarios.
  • Achieves the lowest average rank (2.3) among all compared methods, consistently placing in the top tier across all datasets and patterns.
  • Significantly outperforms other MDE-integrated methods (PriorDA, TestPromptDC, Marigold-DC), particularly in "Range" and "Hole" patterns where two-stage methods fail due to alignment artifacts.
• Efficiency:
  • Speedup: Delivers a 1.4× speedup over PriorDA (using the same Depth Anything-Large backbone) with superior accuracy (0.49s vs 0.68s inference time).
  • Lightweight: Adds less than 20% parameters to the MDE backbone (60.6M vs 335.3M).
  • Scalability: The smallest variant (DA-S) runs at 0.09s, roughly 7× faster than PriorDA and 1000× faster than test-time adaptation methods.
• Real-World Application:
  • Deployed in a robotic warehouse for grasping black packages (which absorb infrared light, causing ToF sensor failures).
  • Improved grasp success rates for black packages from 28% to 91.6% (and up to 100% in controlled trials) by accurately reconstructing the 3D geometry of missing depth regions.

5. Significance

Any2Full establishes a new paradigm for universal depth completion. By reframing the task as a prompt-based adaptation of pretrained MDE models, it successfully leverages the robust geometric priors of foundation models while overcoming their scale ambiguity through efficient, pattern-agnostic prompting.

The method's ability to operate in a single stage with high efficiency makes it uniquely suitable for real-time robotic applications where computational resources are limited and robustness to unseen sensor patterns is critical. It demonstrates that explicit, multi-stage alignment is not necessary for high-quality metric depth recovery, offering a more elegant and robust solution for the robotics and computer vision communities.