💻 computer science

Beyond Visual Ambiguity: Guiding Robust Monocular Depth Estimation in Challenging Scenarios via Detailed Long Captions

The paper proposes CapDepth, a novel monocular depth estimation framework that leverages detailed long captions and a text-adaptive decoding mechanism to effectively resolve visual ambiguities and significantly improve robustness in challenging scenarios like non-Lambertian surfaces and adverse weather.

Original authors: Junrui Zhang, Jiaqi Li, Yiran Wang, Liao Shen, Zhiguo Cao

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junrui Zhang, Jiaqi Li, Yiran Wang, Liao Shen, Zhiguo Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to draw a 3D map of a room using only a single photograph. This is the challenge of Monocular Depth Estimation (MDE). It's like trying to guess how far away a friend is standing just by looking at a flat picture of them. Computers are getting pretty good at this, but they hit a wall when the picture is tricky. If the object is a shiny mirror or a clear glass window, the camera gets confused because the reflection looks like the object behind it. If the weather is terrible—like a heavy rainstorm or a pitch-black night—the camera can't see the edges of things clearly. It's like trying to solve a puzzle when half the pieces are missing or covered in fog.

To fix this, scientists have tried two main things so far. Some try to "paint over" the confusing parts of the image with a computer program, like a digital artist fixing a smudge. Others try to teach the computer by showing it millions of fake, rainy, or foggy pictures. But these methods are often like trying to fix a leaky roof with a bucket; they work for one specific problem but fail when the situation changes. Recently, researchers discovered that giving the computer a "description" of the scene along with the photo helps it understand better. However, previous attempts only gave the computer very short, simple descriptions, like "a car" or "a tree." It turns out, just like humans, computers need more context to solve the puzzle when things get messy.

This is where a new study called CapDepth steps in. The researchers, Junrui Zhang and their team, asked a simple question: What if we didn't just give the computer a short label, but a detailed, long story about the scene? Imagine instead of saying "a car," you tell the computer, "The red car is parked in front of the blue house, and the streetlight is shining above it." The paper suggests that these rich, long descriptions act like a flashlight, guiding the computer's vision through the fog and around the shiny glass.

The team built a new system to test this idea. They didn't just feed the computer a sentence; they designed a specific template that forces the description to focus on spatial relationships—telling exactly where things are in relation to each other. They then created a "smart reader" (a dynamic caption encoder) that knows how to ignore boring words like "the" or "and" and focus only on the important clues, like "in front of," "above," or "to the left." Finally, they built a "translator" (a text-adaptive decoder) that takes these important clues and uses them to correct the computer's depth map, effectively saying, "Wait, the text says the glass is in front of the wall, so the wall must be further away."

The results are quite promising. When tested on tricky images involving glass, mirrors, rain, and night scenes, this new method didn't just tweak the results; it significantly improved them. On surfaces that are hard to see through (like glass or mirrors), the new system reduced the error rate by 25.0% compared to the previous best method. In bad weather conditions, like rain or fog, it cut the error by 22.0%. The researchers found that the more specific and detailed the "story" was, the better the computer performed. They even showed that if you remove the detailed sentences and go back to simple labels, the performance drops, proving that the length and detail of the description really matter.

The paper argues that previous methods failed because they treated text as a simple tag rather than a rich source of spatial information. By switching to these "detailed long captions," CapDepth suggests that we can teach computers to be much more robust when the world gets messy. It's a reminder that sometimes, to see the world clearly, you don't just need better eyes; you need a better story to tell.

Technical Summary: Beyond Visual Ambiguity: Guiding Robust Monocular Depth Estimation in Challenging Scenarios via Detailed Long Captions

Problem Statement
Monocular Depth Estimation (MDE) aims to predict dense depth maps from single images, a task critical for applications like novel view synthesis and virtual reality. However, MDE is inherently ill-posed and under-constrained, making it susceptible to visual ambiguities in challenging scenarios. The paper identifies two primary sources of these ambiguities:

Texture Ambiguity: Caused by non-Lambertian surfaces (e.g., glass, mirrors) where reflection and transparency obscure geometric cues.
Visibility Ambiguity: Caused by adverse weather conditions (e.g., rain, fog, night) which degrade image clarity.

Existing general MDE methods (e.g., Depth Anything, Metric3D) struggle with these scenarios due to a lack of accurate depth annotations in such conditions. Prior robust MDE approaches attempt to solve these issues in isolation using scenario-specific data augmentation (e.g., inpainting transparent surfaces or synthesizing weather effects), but these methods suffer from domain gaps and lack generalizability across different challenge types. Furthermore, existing language-integrated MDE methods fail to fully leverage the potential of text modality due to three limitations: (1) reliance on simple short text inputs with limited information, (2) extraction of coarse global text features (e.g., [CLS] tokens), and (3) a feature gap between noise prediction features used in diffusion models and the precise geometric reasoning required for depth decoding.

Methodology: CapDepth
The authors propose CapDepth, a unified framework that leverages detailed long captions to guide robust MDE. The framework consists of three core components designed to address the limitations of prior work:

Detailed Long Caption Input Template:
Instead of simple prompts like "a photo of a ," the authors design a template composed of multiple "atom sentences." Each sentence explicitly describes the spatial relationship between a pair of objects (e.g., "The bottle is in front of the door"). This structure is constructed by stacking these atom sentences to create a long caption containing rich, explicit spatial information about the scene.
Dynamic Caption Encoder:
To extract effective guidance from the long captions, the authors introduce a dynamic caption encoder. Rather than using coarse global features, this module employs progressive masked querying blocks. It utilizes a learnable soft mask to dynamically distinguish and prioritize depth-relevant tokens (such as object names and spatial prepositions) while suppressing less informative tokens (like functional words or punctuation). This process extracts fine-grained, depth-relevant text features.
Text-Adaptive Decoder:
To bridge the gap between the noise-prediction features of the underlying diffusion U-Net and the geometric reasoning needed for MDE, the authors propose a text-adaptive decoder. This module integrates the extracted text features into the depth decoding process using Stable Adaptive Layer Normalization (SAdaLN).
- Text features are pooled and passed through an MLP to generate scaling ( $a$ ) and shifting ( $b$ ) vectors.
- These vectors modulate the visual features via the equation $\tilde{F} = \text{LayerNorm}(F) \cdot a + b$ .
- To ensure stable training and prevent variance explosion caused by diverse text inputs, a regularization loss ( $L_{reg}$ ) is introduced to constrain the distribution of the modulated features $\tilde{F}$ to follow a standard normal distribution.

Key Contributions
The paper outlines three primary contributions:

A Detailed Long Caption Input Template: A structured input format that conveys rich spatial relationships via multiple atom sentences, providing superior language guidance compared to short text prompts.
A Dynamic Caption Encoder: A mechanism that extracts fine-grained, depth-relevant text features from long captions using progressive masked attention, avoiding the information loss associated with global feature pooling.
A Text-Adaptive Decoder: A novel decoding strategy that leverages SAdaLN to effectively integrate text features into the visual feature stream, guiding enhanced depth prediction and ensuring training stability.

Experimental Results
The authors evaluated CapDepth on four zero-shot benchmarks covering non-Lambertian surfaces (Booster, ClearGrasp) and adverse weather (nuScenes, DrivingStereo).

Quantitative Performance: CapDepth outperforms state-of-the-art general MDE methods, prior robust MDE approaches, and existing language-integrated models.
- On non-Lambertian surfaces (Booster ToM), it achieved a 25.0% reduction in AbsRel compared to Depth Anything V2 and a 35.0% reduction compared to Depth4ToM.
- Under adverse weather conditions (nuScenes night-rain), it achieved a 22.0% reduction in AbsRel compared to RobustDepth.
Qualitative Performance: Visualizations demonstrate that CapDepth produces more accurate geometric structures in challenging regions (e.g., transparent windows, vehicles in rain) where RGB information alone is insufficient.
Ablation Studies: Experiments confirm that removing the dynamic encoder or the SAdaLN decoder significantly degrades performance, validating the necessity of fine-grained feature extraction and stable feature modulation. The study also shows a positive correlation between the number of atom sentences and model performance.

Significance and Claims
The paper claims that CapDepth successfully addresses the visual ambiguities inherent in challenging MDE scenarios by unifying the treatment of non-Lambertian surfaces and adverse weather through a single language-guided framework. The authors assert that their work demonstrates the efficacy of detailed long captions in enhancing visual perception capabilities, moving beyond the limitations of short text inputs and coarse feature integration. By explicitly guiding the depth decoding process with fine-grained spatial language, CapDepth achieves significant robustness improvements without requiring scenario-specific data augmentation. The authors modestly note that their current work is limited to English texts and suggest that future research could explore generalization across diverse languages.

More like this