Technical Summary: Focusable Monocular Depth Estimation

Problem Formulation

Current monocular depth foundation models (e.g., Depth Anything, UniDepth, Metric3D) excel at recovering dense scene geometry across diverse domains. However, they are predominantly optimized using uniform, pixel-wise objectives that treat all image regions equally. This "image-centric" approach fails to distinguish user-specified or task-relevant target regions from the surrounding context. In applications such as robotics and embodied AI, where precise depth near a target object and sharp boundary transitions are critical, uniform global optimization often results in underweighted foreground accuracy and blurry boundaries, even when global scene geometry remains coherent.

To address this gap, the authors introduce Focusable Monocular Depth Estimation (FDE). FDE reframes the depth estimation task from uniform dense prediction to target-conditioned inference. Given an input image and a user-specified target region (via box or text prompts), the model is required to:

1. Prioritize depth accuracy in the foreground target region.
2. Preserve sharp depth transitions at the target boundaries.
3. Maintain coherent global scene geometry across the entire image.

Methodology: FocusDepth

To solve the FDE task, the authors propose FocusDepth, a prompt-conditioned monocular relative depth estimation framework. The core intuition is to combine the complementary priors of promptable segmentation (Segment Anything Model 3, or SAM3) and dense geometry foundation models (Depth Anything family, or DAs).

Architecture Overview

FocusDepth consists of two primary branches:

1. Geometry Depth Branch: Utilizes pretrained DAs (DA2 or DA3) to extract multi-scale geometry tokens ($X_g$) representing the global scene structure.
2. Prompt-Conditioned Branch: Utilizes SAM3 to process the input image and user prompt (box or text) to generate prompt-conditioned token features ($X_p$) that identify the target region.

Core Module: Multi-Scale Spatial-Aligned Fusion (MSSA)

The central innovation is the MSSA module, which injects prompt-conditioned cues into the geometry representation without disrupting the pretrained geometric prior. MSSA operates at each scale of the geometry encoder and involves three key steps:

• Scale-wise Spatial Alignment: Since both branches process the image on the same ViT patch grid, tokens are in one-to-one spatial correspondence. The prompt tokens are projected into the scale-specific geometry token space and concatenated with the geometry tokens.
• Routed Conditional Fusion: A lightweight Mixture of Experts (MoE) layer processes the concatenated tokens. A router computes input-dependent combinations of expert outputs, allowing the model to apply different corrections based on spatial location without explicit supervision for specific regions (foreground vs. background).
• Gated Fusion Output: To prevent the prompt cues from overwriting the global geometry, a learnable gate ($G_s$) controls the injection. The final feature is a weighted sum of the routed features and the original geometry tokens: $Y_s = G_s \odot F_s + (1 - G_s) \odot X_g^s$. This ensures that prompt-conditioned corrections are applied selectively while retaining the global prior.

Learning Objective and Training

The model is trained with a region-aware loss function that partitions the image into three regions based on the ground-truth mask: Foreground, Boundary, and Global. The total loss ($L$) is the sum of depth losses ($L_{fg}, L_{bd}, L_{glb}$) computed over these regions, plus an auxiliary segmentation loss ($L_{seg}$).

Training follows a two-stage strategy to ensure stability:

1. Stage 1: Only the MSSA module is optimized while all pretrained backbones are frozen, allowing MSSA to learn alignment without disturbing the geometry prior.
2. Stage 2: The depth decoder, SAM3 prompt-guided transformer, and auxiliary segmentation decoder are optimized alongside MSSA, while the encoders remain frozen.

Benchmark: FDE-Bench

To evaluate FDE, the authors establish FDE-Bench, a benchmark suite converting existing RGB-D datasets into standardized image–target–depth triplets.

• Datasets: Constructed from NYU v2, TUM RGB-D, YCB-Video, RLBench, and RoboTwin, covering indoor scenes, tabletop objects, and embodied manipulation environments.
• Scale: Contains 252.9K training and 72.5K validation triplets spanning 972 target categories.
• Evaluation Metrics: Unlike standard benchmarks that report full-image error, FDE-Bench evaluates performance across three complementary regions:
  • Foreground: Target-local depth quality.
  • Boundary: Depth-transition sharpness.
  • Global: Whole-image geometric consistency.
    Metrics reported include $\delta_1$ and AbsRel, computed in a per-target manner to prevent dominance by images with many targets.

Experimental Results

Experiments were conducted on FDE-Bench comparing FocusDepth against zero-shot foundation models (UniDepth, Metric3D, DA2, DA3) and globally fine-tuned baselines (DA2-ft, DA3-ft).

• Performance Gains: FocusDepth consistently outperforms globally fine-tuned baselines, particularly in target-centric regions. For example, on RLBench with box prompts, FocusDepth(DA3) reduced boundary AbsRel from 0.073 to 0.049 and foreground AbsRel from 0.095 to 0.056 compared to DA3-ft, while simultaneously improving global AbsRel from 0.042 to 0.030.
• Prompt Robustness: A study on prompt correctness showed that while correct prompts yield the best results, even incorrect or empty prompts result in better performance than the no-prompt baseline, suggesting the prompt-conditioned branch provides useful local and global cues even without perfect guidance.
• Qualitative Analysis: Visualizations demonstrate that FocusDepth produces sharper target boundaries and lower depth errors in the specified region compared to standard fine-tuning, without introducing artifacts in the background.

Ablation Studies

Ablation studies on RLBench highlight the importance of specific design choices:

• Spatial Alignment: Disrupting the spatial correspondence between prompt and geometry tokens (shuffling) causes the most significant degradation, increasing global AbsRel by up to 13.8%. This confirms that spatial alignment is the key design factor.
• Scale-Specific and Routed Fusion: Removing scale-specific fusion or replacing the MoE router with a single MLP consistently worsens foreground and boundary accuracy, indicating the necessity of scale-aware and conditional fusion.
• Gated Injection: Removing the gating mechanism leads to consistent degradation, confirming its role in balancing prompt corrections with the pretrained geometry prior.
• Training Strategy: A one-stage training approach weakens local refinement, validating the two-stage strategy for separating cross-model alignment from downstream adaptation.

Significance and Claims

The paper claims to establish FDE as a concrete task formulation for region-prioritized depth estimation, moving beyond the limitations of uniform image-centric objectives. The authors assert that FocusDepth successfully transfers prompt-grounded target evidence into dense geometry representations via MSSA, enabling spatially aligned and gated conditional refinement.

The significance lies in demonstrating that it is possible to improve local target accuracy and boundary fidelity without compromising global scene geometric coherence. The authors position FDE as a necessary step for applications requiring precise interaction with specific objects (e.g., robotic manipulation) where standard depth models may fail to prioritize the relevant geometry. The work concludes by noting limitations, such as the need for larger-scale joint training and further exploration of downstream task applications, which are left for future work.