💻 computer science

Efficient Test-Time Optimization for Depth Completion via Low-Rank Decoder Adaptation

This paper proposes a lightweight test-time adaptation method for zero-shot depth completion that achieves state-of-the-art accuracy and efficiency by updating only a low-dimensional decoder subspace, thereby avoiding the computational costs of full-network or diffusion-based optimization.

Original authors: Minseok Seo, Wonjun Lee, Jaehyuk Jang, Changick Kim

Published 2026-03-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minseok Seo, Wonjun Lee, Jaehyuk Jang, Changick Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to finish a jigsaw puzzle, but someone has only given you a few scattered pieces (the sparse depth from a LiDAR sensor) and a photo of the finished picture (the RGB image). Your goal is to fill in all the missing pieces to create a perfect 3D map of the scene. This is called Depth Completion.

For a long time, computers had two main ways to solve this puzzle, and both had a major flaw:

The "Study Hard" Approach (Training-Based): You could build a custom robot that studies millions of specific puzzles before it ever sees a real one. It's fast when it works, but if you give it a puzzle from a different box (a different camera or environment), it gets confused. Plus, building the robot takes forever.
The "Guess and Check" Approach (Test-Time Optimization): You could use a super-smart robot that knows how to guess the shape of any puzzle just by looking at the photo. But to get the missing pieces right, it has to try, fail, adjust, and try again thousands of times while you are watching. This is incredibly accurate, but it takes so long (seconds per image) that it's useless for a self-driving car that needs to react instantly.

The New Solution: "The Low-Rank Decoder Adaptation"

This paper introduces a clever new way to solve the puzzle that gets the best of both worlds: high accuracy and blazing speed.

Here is how they did it, using a simple analogy:

1. The "Brain" vs. The "Hand"

The computer model they use (called a Depth Foundation Model) is like a giant factory with two main parts:

The Encoder (The Brain): This part looks at the photo and understands the scene. It knows "that's a car," "that's a tree," and "that's a road."
The Decoder (The Hand): This part takes the brain's understanding and actually draws the final 3D map.

Previous methods tried to fix the robot by tweaking the Brain (the Encoder) every time it saw a new puzzle. But the Brain is huge and complex. Changing it is slow and computationally expensive, like trying to rewrite a whole encyclopedia every time you want to solve a puzzle.

2. The Big Discovery

The researchers looked closely at how these models work and found a secret: The "Hand" (Decoder) already knows almost everything it needs to know.

They discovered that the most important information about how deep things are is concentrated in a tiny, low-dimensional "subspace" within the Decoder. Think of it like this:

The Decoder is a massive library with millions of books.
The researchers found that all the information needed to fix the depth map is actually just written on one specific page in one specific book.
You don't need to read the whole library (the whole model) to fix the puzzle; you just need to tweak that one page.

3. The "Low-Rank" Trick

Instead of retraining the whole robot, they use a technique called Low-Rank Adaptation (LoRA).

Old Way: Rewrite the whole instruction manual for the robot every time you see a new scene. (Slow, heavy, expensive).
New Way: Keep the instruction manual exactly as it is. Just stick a tiny, sticky-note-sized "cheat sheet" on the specific page where the depth info lives. This cheat sheet tells the robot how to adjust its "Hand" to match the specific sparse pieces you gave it.

Because this "cheat sheet" is so small, the computer can calculate the adjustments in a fraction of a second.

Why is this a Big Deal?

Speed: The old "Guess and Check" methods took about 6 to 10 seconds to fix one image. The new method does it in 0.3 to 2 seconds. That's like going from a slow dial-up internet connection to 5G.
Accuracy: It doesn't just get faster; it actually gets better. By focusing only on the part of the brain that matters, it avoids making mistakes that happen when you try to change too many things at once.
No Training Needed: It works immediately on any camera or environment without needing to be retrained on new data. It's "Zero-Shot," meaning it's ready to go out of the box.

The Bottom Line

Imagine you have a master chef (the AI) who can cook any dish.

Old Method: If you give them a weird ingredient, they stop, go to the library, read 50 cookbooks, and rewrite their entire recipe book before cooking. (Accurate, but takes hours).
This Paper's Method: The chef looks at the weird ingredient, realizes they only need to adjust the salt and pepper (the low-rank subspace), and makes a tiny adjustment to the recipe on the fly. The dish comes out perfect in seconds.

This breakthrough means self-driving cars, robots, and AR glasses can finally understand 3D depth in real-time, accurately and efficiently, without needing a massive computer in their backpack.

1. Problem Statement

Depth Completion aims to generate dense depth maps from sparse depth measurements (e.g., from LiDAR) guided by RGB images. While training-based methods achieve high accuracy, they require expensive, sensor-specific datasets and retraining, limiting their generalization to new environments or sensors.

Zero-shot Test-Time Optimization (TTO) has emerged as a solution, adapting pre-trained Depth Foundation Models (DFMs) at inference time using sparse depth as a supervision signal without retraining. However, existing TTO methods suffer from a critical trade-off:

Latency: Methods like Marigold-DC (diffusion-based) or Visual Prompting require repeated forward-backward passes through the entire network (encoder + decoder) to optimize latent variables or input prompts. This results in inference times of several seconds per image (e.g., 3–40 seconds), making them impractical for real-time applications.
Efficiency vs. Accuracy: Training-based methods are fast but require data; existing TTO methods are accurate but slow.

The paper addresses the challenge of achieving state-of-the-art accuracy with high inference efficiency (near real-time) in a zero-shot setting.

2. Core Insight & Methodology

The authors propose that adapting the entire network is unnecessary. Through systematic analysis, they discovered that depth-relevant information in DFM architectures is concentrated in a low-dimensional subspace of the decoder.

Key Analysis (Section 3)

Layer-wise Correlation: The correlation between intermediate features and the final depth output is low in the encoder but increases sharply in the decoder.
PCA Visualization: The first principal component (PC1) of decoder features aligns strongly with the final depth map, indicating that the decoder encodes the geometric structure in a low-dimensional space.
Implication: Updating the encoder is computationally expensive (requiring re-computation of features) and less critical for structural alignment. Updating only the decoder's low-dimensional subspace is sufficient for effective adaptation.

Proposed Method: Low-Rank Decoder Adaptation

The method, illustrated in Fig. 2(c), follows this pipeline:

Single Forward Pass (Encoder): The frozen encoder processes the RGB image once to extract feature maps ( $f$ ). These features are cached.
Decoder-Only Optimization: The optimization loop runs only on the decoder ( $d$ $d$ ).
- Sparse Depth Alignment: Since sparse depth ( $S$ ) has unknown scale and shift, the method first estimates linear scale ( $a$ ) and shift ( $b$ ) parameters via least-squares fitting between the predicted depth and sparse measurements.
- Low-Rank Adaptation (LoRA): Instead of fine-tuning all decoder weights, the method applies Low-Rank Adaptation (LoRA). It updates only a low-rank subspace ( $r=8$ ) of the decoder parameters.
- Objective: Minimize the alignment error between the scaled/shifted predicted depth and the sparse ground truth at observed pixel locations.
Output: The adapted decoder generates the final dense depth map.

Algorithm 1 Summary:

Encode $I \to f$ (Cached).
Loop $T$ $T$ times:
- Decode $f \to \hat{D}$ .
- Estimate scale/shift ( $a, b$ ) to align $\hat{D}$ with $S$ .
- Compute loss on sparse pixels.
- Update only LoRA parameters of the decoder.

3. Key Contributions

Theoretical Insight: Demonstrated that depth-aligned signals in foundation models are concentrated in a low-dimensional decoder subspace, justifying decoder-only adaptation.
Efficient Architecture: Proposed a Decoder-Only LoRA strategy that caches encoder features and optimizes only the decoder, drastically reducing computational overhead.
Pareto Frontier: Established a new Pareto frontier for depth completion, simultaneously achieving the lowest error and the fastest inference time among zero-shot TTO methods.
Robustness: The method is model-agnostic (works with UniDepth, Depth Anything, etc.) and robust across varying sparsity levels and domain shifts (indoor/outdoor).

4. Experimental Results

The method was evaluated on five datasets: IBims-1, VOID, NYUv2, KITTI-DC, and DDAD.

Accuracy (SOTA):
- On NYUv2, the method achieved an RMSE of 0.182, outperforming TestPromptDC (0.186) and Marigold-DC (0.216).
- On KITTI-DC, it achieved an RMSE of 1.476, significantly better than TestPromptDC (1.793) and Marigold-DC (1.663).
- It consistently outperformed both training-based methods (like PromptDA) and other zero-shot TTO methods across all benchmarks.
Efficiency (Speed):
- Inference Time: The method runs in approximately 0.3s to 2.5s depending on resolution (VGA to FHD).
- Comparison:
  - Marigold-DC: ~42s (FHD).
  - TestPromptDC: ~79s (FHD, often OOM).
  - Ours: 2.53s (FHD).
- It is the fastest TTO method by a large margin, approaching real-time capabilities.
Resource Efficiency:
- Parameters: Only a tiny fraction of parameters (LoRA rank $r=8$ ) are trainable.
- Memory: By caching encoder features, it avoids the memory bottleneck of repeated encoder passes, preventing Out-of-Memory (OOM) errors at high resolutions where other methods fail.
Ablation Studies:
- Rank Sensitivity: Rank $r=8$ yielded optimal performance; lower ranks reduced accuracy, and higher ranks increased cost without gain.
- Decoder vs. Encoder: Decoder-only adaptation converged faster (40 epochs) and with higher accuracy than encoder-only or full-network adaptation.

5. Significance and Impact

Practical Deployment: This work bridges the gap between high-accuracy zero-shot depth completion and real-time application requirements. It enables the use of powerful foundation models in scenarios requiring fast inference (e.g., autonomous driving, robotics) without the cost of collecting sensor-specific training data.
Efficiency Paradigm: It challenges the assumption that full-network optimization is necessary for test-time adaptation, suggesting that low-rank decoder adaptation is a more efficient and stable alternative for geometric tasks.
Future Directions: The authors suggest extending this framework to video-based depth completion, where temporal coherence could allow reusing adapted parameters across frames, potentially achieving true real-time performance.

In summary, the paper presents a highly efficient, accurate, and practical solution for zero-shot depth completion by leveraging the low-rank structure of depth foundation model decoders, effectively solving the speed-accuracy trade-off that has plagued previous test-time optimization methods.