💻 computer science

Learning Image-Adaptive Scale Fields for Metric Depth Recovery

This paper proposes a metric depth recovery method that models scale correction as an image-adaptive scale field, utilizing a low-dimensional linear combination of semantic and geometric basis maps with weights derived from sparse anchors to achieve robust and accurate metric depth estimation.

Original authors: Yuanyan Li, Matthias Althoff

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanyan Li, Matthias Althoff

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a camera that can guess how far away things are just by looking at a single photo. This is called Monocular Depth Estimation. It's like a very talented artist who can sketch a 3D scene on a 2D piece of paper. However, this artist has a quirk: they are great at getting the shape and relative distances right (the tree is twice as far as the rock), but they are terrible at getting the actual size right. They might draw the tree as 10 feet tall when it's actually 50, or 5 feet tall when it's 10. Their drawing is "up to scale," but the scale is unknown.

In the real world, we need to know the exact distance (metric depth) for things like self-driving cars or robots. To fix the artist's scale issue, we usually give them a few "anchors"—specific points where we know the exact distance (like a LiDAR sensor telling us, "That rock is exactly 5 meters away").

The Problem with Old Methods

Traditionally, people tried to fix the artist's drawing by applying a single "global" fix. They would say, "Okay, the whole picture is too small; let's just multiply everything by 2." Or, they might try to fix different parts of the picture separately.

But real life is messy. Sometimes the sky is too far, the ground is too close, and the buildings in the middle are just right. A single "multiply by 2" rule doesn't work for the whole scene. Previous methods tried to fix this by chopping the image into tiny grids or regions, but if you don't have enough "anchors" (exact distance points) in every single tiny grid, the math breaks down, and the fix becomes unstable.

The New Solution: "Image-Adaptive Scale Fields"

This paper proposes a smarter way to fix the artist's drawing. Instead of trying to guess the exact distance for every single pixel directly, the authors treat the problem like mixing paint.

Here is the analogy:

The Palette (Basis Maps): Imagine the artist has a special palette with a few "base colors" (called Basis Maps). These aren't just random colors; they are smart patterns learned from thousands of photos.
- One pattern might represent "things getting smaller as they go up" (the sky).
- Another might represent "things getting closer as they go down" (the ground).
- Another might capture "shadows near buildings."
- These patterns are derived from the artist's original sketch and the hidden details the artist used to make the sketch.
The Mixing (Weights): The goal isn't to invent a new color for every pixel. Instead, the system asks: "How much of Pattern A, how much of Pattern B, and how much of Pattern C do we need to mix together to fix the scale?"
The Anchors (The Recipe): We only have a few "anchors" (exact distance points). The system looks at these anchors and solves a simple math puzzle (a least-squares problem) to figure out the perfect mixing ratio (the weights) for the patterns.
- Even if you only have 5 anchors in the whole picture, the system can figure out the right mix because the patterns are so smart and cover the whole image.
The Result: Once the system knows the mixing ratio, it applies that mix to the entire image. It creates a new, perfectly scaled depth map where the tree is the right height and the rock is the right distance.

Why This is a Big Deal

It works with very few anchors: Because the "patterns" (basis maps) are pre-learned and cover the whole image, the system doesn't need a dense grid of anchors. It can fix the whole picture even if you only give it a handful of exact measurements.
It's stable: Old methods would get confused if anchors were missing in a specific corner. This method looks at the whole picture and uses the global patterns to fill in the gaps smoothly.
It's explainable: You can actually look at the "patterns" the system learned. You can see, "Ah, the system learned that the ground usually needs a specific type of correction." It's not a black box; it's a clear combination of understandable parts.

The Bottom Line

The authors created a system that takes a "rough guess" of depth and a few "exact measurements," then uses a smart, pre-learned set of patterns to blend them together. This allows robots and cars to get accurate, real-world distances from a single camera, even when they don't have a lot of extra sensor data to help them. They tested this on cars driving on roads and indoor scenes, and it consistently beat the old methods, especially when data was scarce.

Technical Summary: Learning Image-Adaptive Scale Fields for Metric Depth Recovery

Problem Statement

Monocular Depth Estimation (MDE) has advanced significantly with transformer-based architectures and large-scale pretraining. However, most widely used methods produce depth maps defined only up to an unknown scale and shift. While some approaches estimate metric depth directly, many practical applications (robotics, autonomous driving, 3D reconstruction) require explicit metric alignment. This alignment typically relies on sparse metric anchors (e.g., LiDAR points or structure-from-motion constraints).

Existing methods for recovering metric depth from sparse anchors generally fall into two categories:

Global Affine Alignment: Assumes uniform scale and shift across the entire image. While stable, this fails in scenes with diverse objects or complex layouts where biases are non-uniform.
Local/Regional Approaches: Methods like piecewise affine alignment, locally weighted regression (LWLR), grid-based optimization, and region-aware adaptation attempt to model spatially varying biases. However, these often rely on structural assumptions that may not hold in real scenes or fragment the sparse anchor set, leading to unstable estimates when anchors are insufficient.

The core challenge is that recovering dense per-pixel scale and shift fields from sparse anchors is an inherently ill-posed problem. Existing solutions impose predefined structural priors that may be unreliable under extreme sparsity.

Methodology

The authors propose formulating metric depth recovery as image-adaptive scale field modeling. Instead of directly correcting the depth map, the method reformulates the correction as a low-dimensional linear combination of learned basis maps.

1. Parameterization in Log-Space

The metric depth $D_{gt}(x)$ is related to the estimated depth $D_{MDE}(x)$ via an affine transformation:
$D_{gt}(x) = s(x) D_{MDE}(x) + t(x)$
To avoid depth-dependent gradient scaling issues where large-depth pixels dominate optimization, the authors operate in log-space. They define the log-scale field $\ell(x) = \log s(x)$ and absorb the shift term $t(x)$ as a depth-dependent correction within the scale field.

2. Basis Map Generator

The log-scale field is modeled as a linear combination of $K$ image-adaptive basis maps:
$\ell(x) = \sum_{m=1}^{K} w_m E_m(x) = E(x)^\top w$
where $w \in \mathbb{R}^K$ are image-specific weights, and $E_m(x)$ are the basis maps.

The Basis Map Generator is a neural network that produces these maps based on three inputs:

Intermediate Representations ( $F(x)$ ): Features extracted from the MDE backbone prior to the depth head, capturing semantic and structural context.
Estimated Log-Depth: $\log D_{MDE}(x)$ and its spatial gradient, encoding depth-dependent geometry.
Image Coordinates ( $x$ ): To model spatial trends.

The generator architecture consists of:

A shared convolutional trunk producing features $H(x)$ .
A Basis Head predicting primitive basis maps $B(x)$ .
A Gating Head predicting normalized weights $G(x)$ (via softmax).
The final basis maps are computed as $E(x) = G(x) \odot B(x)$ . This factorization allows spatial patterns to be reused across different image regions with varying magnitudes controlled by the gating maps.

3. Weight Estimation via Least Squares

The weights $w$ are determined by fitting the model to the sparse metric anchors $A = \{a_i\}$ . For each anchor $a_i$ , the target is $y_i = \log D_{gt}(a_i) - \log D_{MDE}(a_i)$ . The weights are found by solving a regularized least-squares problem:
$w^* = \arg \min_w \|Mw - y\|_2^2 + \lambda \|w\|_2^2$
where $M$ is the matrix of basis map values at anchor locations. This yields a closed-form solution, ensuring stability even under extreme anchor sparsity.

4. Training Objective

The basis map generator is trained using:

Dense Loss: SmoothL1 loss between the recovered and ground-truth metric depth.
Anchor Loss: MSE between the predicted and target log-scale at anchor locations.
Decorrelation Loss: Encourages orthogonality among basis maps to prevent redundancy.
Gating Regularization: Negative entropy loss to prevent overly sharp gating assignments.

Key Contributions

Low-Dimensional Reformulation: The paper introduces a formulation where metric depth recovery is reduced to estimating a small set of weights for image-adaptive basis maps, rather than learning a complex spatial field directly.
Robustness to Sparsity: By using a globally aggregated least-squares solver, the method achieves stable solutions even with very few anchors (e.g., 1–9 anchors per image), outperforming methods that fragment anchors or rely on local regression.
Interpretability: The approach provides an interpretable decomposition of spatial scale variations. The learned basis maps and their weights reveal how global and local biases are structured.
General Applicability: The method is architecture-agnostic, requiring only intermediate representations and depth estimations from a pretrained MDE model. It does not require external segmentation models or specific backbone architectures.

Experimental Results

The method was evaluated on KITTI, UrbanSyn, and SUN RGB-D datasets using Depth Anything 3 (DA3) and MiDaS as representative MDE models.

Performance: The proposed method ("Basis") consistently outperformed global, piecewise, LWLR, grid-based, and region-aware baselines across all datasets and anchor densities.
- On KITTI with 10–15 anchors (Low regime), the method achieved an AbsRel of 0.15 (DA3) and 0.13 (MiDaS), significantly better than the global baseline (0.31/0.17).
- On UrbanSyn/SUN RGB-D, it achieved AbsRel of 0.08 (DA3) and 0.14 (MiDaS) in the low-anchor regime.
Extreme Sparsity: In a "drop-anchor" test reducing anchors to as few as 1, the method remained stable. Spectral analysis revealed that the dominant eigenmode of the anchor embedding matrix corresponds almost entirely to the global scale channel, explaining the robustness.
Ablation: The study confirmed that intermediate representations ( $F(x)$ ) are crucial; disabling them prevented the model from benefiting from increased basis capacity (higher $K$ ).

Significance and Claims

The authors claim that their approach offers a practical framework for incorporating sparse metric information into modern MDE systems. Key takeaways include:

Improved Accuracy: The method achieves state-of-the-art metric depth alignment accuracy across diverse scenes and anchor densities.
Robustness: The low-dimensional formulation and global least-squares estimation provide strong robustness where local methods fail due to insufficient anchors.
Interpretability: The decomposition into basis maps offers a new way to analyze and understand scale-shift biases in MDE models.

The paper modestly notes that performance still depends on the informativeness of the anchors (spatial coverage and diversity) and that validation on CNN-based MDE models remains future work. However, the proposed formulation is presented as a generalizable solution that subsumes several existing alignment strategies as special cases.