🤖 machine learning

WAVE: Reversing the Guidance Hierarchy for Coarse-to-Fine Guided Depth Super-Resolution

The paper presents WAVE, a guided depth super-resolution method that reverses the traditional fine-to-coarse guidance hierarchy by employing a multi-level discrete wavelet transform to enable a coarse-to-fine reconstruction process that explicitly separates structure and detail, filters misleading high-frequency RGB cues, and fuses modalities to achieve superior performance, particularly at high upsampling factors.

Original authors: Tayyab Nasir, Daochang Liu, Ajmal Mian

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tayyab Nasir, Daochang Liu, Ajmal Mian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Depth maps are the invisible scaffolding of modern vision, providing the crucial sense of three-dimensional space that allows self-driving cars to navigate streets and augmented reality glasses to place virtual objects on real tables. While cameras can capture the rich colors and textures of a scene, the sensors that measure distance often produce images that are blurry and low in resolution, lacking the sharp detail needed for precise tasks. To fix this, scientists have long relied on a technique called guided depth super-resolution, which tries to borrow the sharp edges from a high-quality color photograph to sharpen the fuzzy depth map. The logic is sound: if a wall has a clear edge in the color photo, the depth map should show a clear edge in the same spot. However, this approach has a persistent flaw. The color photo is full of distractions—shadows, patterns, and changes in lighting—that do not correspond to actual physical boundaries. When computers try to copy these visual details directly, they often end up blurring the true edges or creating fake depth lines where none exist, much like trying to trace a complex drawing while looking at a reflection in a rippling pond.

A team of researchers at the University of Western Australia has proposed a new method called WAVE that fundamentally changes how this borrowing process works. Instead of letting the computer look at the entire color image at once and try to figure out what is important, WAVE breaks the image down into layers of detail, starting with the broadest shapes and working its way down to the finest textures. The researchers realized that existing systems make a critical mistake by starting with the small, noisy details and trying to filter them out later, a process that often leaves behind artifacts. WAVE reverses this order. It first establishes the large, global structure of the scene using the smoothest parts of the image, and only then adds the finer details, ensuring that the basic shape is correct before any complex patterns are introduced. This approach is akin to a sculptor who first carves the general form of a statue before chiseling away the fine details, rather than trying to carve the details first and hoping the overall shape emerges.

To achieve this, the system uses a mathematical tool called a wavelet transform to separate the color image into different frequency bands. Think of these bands as layers of information: one layer holds the smooth, large-scale structures of the scene, while other layers hold the sharp edges and the fine textures. The system processes these layers in reverse order of their complexity. It begins by using the smoothest layer to build a rough, accurate depth map, effectively ignoring the distracting textures and shadows that usually cause errors. Only after this solid foundation is laid does the system bring in the sharper layers to add detail. Crucially, the system also uses a separate source of knowledge, derived from a large pre-trained model that understands the meaning of objects in a scene, to act as a gatekeeper. This gatekeeper decides which of the sharp details from the color image are actually useful for the depth map and which are just visual noise. If a texture in the color photo does not match the true edge of an object, the gatekeeper blocks it, preventing the depth map from becoming blurry or distorted.

The results of this approach are significant, particularly when the original depth image is very blurry and contains very little structure to begin with. In tests where the researchers had to increase the resolution of the depth map by a factor of thirty-two, the new method produced results that were measurably more accurate than previous techniques. On specific datasets used to benchmark this technology, the new system reduced the error rate to 3.77 centimeters on one test set and 7.90 centimeters on another, outperforming the next best method in each case. The improvement was most dramatic in these extreme upscaling scenarios, confirming that starting with the big picture and refining it step-by-step is far more effective than trying to fix a messy image all at once. The researchers also found that their method is efficient, requiring fewer adjustable settings than some competing systems while still delivering sharper boundaries and cleaner surfaces.

By treating the reconstruction of depth as a structured, step-by-step process rather than a single, chaotic leap, this work offers a clearer path for machines to understand the physical world. The method successfully filters out misleading visual cues at their source, ensuring that the final depth map reflects the true geometry of the scene rather than the confusing patterns of light and shadow. As autonomous systems and virtual reality continue to demand higher precision, techniques that can reliably separate the signal from the noise will become increasingly vital. This study demonstrates that by respecting the natural hierarchy of visual information—building from the coarse to the fine—computers can learn to see the world with a clarity that was previously out of reach.

Technical Summary: WAVE - Reversing the Guidance Hierarchy for Coarse-to-Fine Guided Depth Super-Resolution

Problem Statement

Guided Depth Super-Resolution (GDSR) aims to reconstruct high-resolution depth maps by leveraging complementary information from low-resolution depth and high-resolution RGB images. However, existing methods, predominantly based on Convolutional Neural Networks (CNNs) and semantic token pipelines, suffer from an inherent fine-to-coarse bias.

In standard CNN-based pipelines, low-level spatial cues (edges, textures) emerge in early layers, while global structures form in deeper layers. Consequently, high-frequency RGB guidance (textures, color gradients, shadows) is introduced early in the processing hierarchy. The network must then implicitly learn to suppress these misleading cues in later layers once global structure is established. This non-monotone process is difficult to optimize and often leads to artifacts, false depth discontinuities, and blurred object boundaries when RGB textures do not align with true depth boundaries. Similarly, semantics-based methods that distill knowledge from foundation models often maintain this fine-to-coarse ordering, introducing global semantic tokens only at later stages.

Methodology: The WAVE Architecture

The authors propose WAVE, a multi-block, multi-branch architecture that explicitly reverses the guidance hierarchy to enable a coarse-to-fine reconstruction. Instead of processing RGB guidance from fine details to coarse structures, WAVE consumes features in the reverse order of their generation, restoring global structure first and refining details last.

1. Multi-Level Discrete Wavelet Transform (ML-DWT)

Unlike conventional methods that feed raw RGB images into the network, WAVE applies a four-level Haar wavelet decomposition to the RGB guide. This creates a pyramid of sub-bands:

Approximation bands ( $LL_l$ ): Retain smooth structural approximations at progressively coarser resolutions.
Detail bands ( $LH_l, HL_l, HH_l$ ): Carry high-frequency information such as edges and textures.
This explicit decomposition allows the network to separate structural cues from texture/noise at the source, providing a mechanism for hierarchical control that CNNs lack.

2. Semantic Token Residual Adapter (SToRA)

To incorporate semantic priors without fine-tuning the entire backbone, WAVE utilizes DINOv3 (ViT-B/16) to extract patch tokens from four specific layers. A lightweight, bias-free module inspired by DiReFT/LoRA is applied to these tokens:
$\tau^*_l = \tau_l + \varsigma W_u(W_d \tau_l)$
This SToRA module adapts the frozen foundation model tokens to the GDSR task using low-rank projections, preserving the rich pre-trained semantics while introducing task-specific adjustments.

3. HUMMA Blocks (Hierarchical Upsampling with Multi-band Multi-resolution Approximation)

The core reconstruction is performed by a series of HUMMA blocks, which operate in reverse order (from coarsest wavelet level $l=4$ to finest $l=1$ ). Each block splits processing into two dedicated branches:

Structure Branch: Processes the low-frequency depth features, the $LL$ approximation band, and the semantic tokens. It uses cross-attention where the depth map queries the semantic tokens to pull relevant structural content. Crucially, it employs an IRN-style invertible coupling mechanism. This bijective fusion ensures that both the depth modality and the photometric ( $LL$ ) information remain recoverable, preventing the network from collapsing onto a single modality or discarding critical information.
Detail Branch: Processes the high-frequency sub-bands ( $LH, HL, HH$ $L H, H L, H H$ ) separately for edges and textures.
- Edge Path: Concatenates horizontal and vertical bands, processing them through an edge encoder.
- Texture Path: Processes all three detail bands to capture multi-scale texture.
- Semantic Gating: The SToRA tokens act as a learnable gating mechanism. They query the feature maps via linear cross-attention to select relevant details. A deferred reliance mechanism (initialized to zero) allows the network to first learn stable band representations before gradually integrating semantic guidance to suppress texture-copying artifacts.

4. Final Fusion and Boundary Refinement

The structural feature map and the refined detail map are fused residually. A final Boundary Refinement stage utilizes the final-layer tokens of the frozen DINOv3 model to compute a soft semantic boundary map based on local patch dissimilarity. This map guides a fusion network to sharpen object contours in the final high-resolution depth output.

Key Contributions

Reversed Guidance Hierarchy: WAVE is the first method to consume ML-DWT sub-bands and hierarchical semantic tokens in reverse order (coarse-to-fine), countering the fine-to-coarse bias inherited from CNN extractors and standard semantic pipelines.
Explicit Source-Level Filtering: It introduces the first use of multi-level wavelet decomposition as an explicit guidance-control mechanism in GDSR. By processing structural, edge, and texture sub-bands independently, it filters misleading RGB cues at their source rather than relying on implicit suppression in deep layers.
Unified Interaction Modeling: The architecture models three previously ununified interaction classes: within/across wavelet sub-bands, between wavelet features and semantics, and between frequency-specific guidance and the depth stream.
Invertible Cross-Modal Fusion: It repurposes IRN-style invertible coupling for bijective cross-modal fusion, ensuring information preservation and preventing modality collapse.
Efficient Semantic Adaptation: It adapts frozen foundation models via a low-rank residual on output tokens (SToRA) rather than weights, achieving task-specific semantics without backbone fine-tuning or explicit semantic loss.

Experimental Results

Extensive experiments were conducted across multiple benchmarks (RGBD-D, NYU_v2, Middlebury, Lu, TOFDSR, DIML) and training protocols.

Performance: WAVE matches or outperforms existing state-of-the-art methods. The performance gains are most significant at high upsampling factors (e.g., 32×), where low-resolution inputs retain the least structure.
- At 32× on the NYU_v2 training protocol, WAVE achieved an RMSE of 3.77 cm on RGBD-D and 7.90 cm on NYU_v2, outperforming the next-best method (SPFNet) which scored 3.97 cm and 8.06 cm, respectively.
- On the Hypersim training protocol at 16×, WAVE achieved an average RMSE of 4.81 cm, outperforming the previous best (DuCos at 5.37 cm).
Qualitative Analysis: Visual comparisons demonstrate that WAVE produces sharper object boundaries, cleaner surfaces, and significantly fewer texture-copying artifacts compared to prior works.
Ablation Studies: Removing the coarse-to-fine schedule, the invertible coupling, the wavelet sub-bands, or the semantic gating mechanisms all resulted in performance degradation, validating the necessity of each component.

Significance and Claims

The paper claims that WAVE addresses the root cause of artifacts in GDSR by fundamentally altering the information flow from fine-to-coarse to coarse-to-fine. By explicitly separating structure and detail and filtering RGB cues at the source via wavelet decomposition, WAVE offers an intuitive alternative to the opaque suppression learned by standard networks.

The authors note that while WAVE performs competitively in saturated, in-domain low-scale regimes, its primary advantage lies in extreme upsampling scenarios where structural priors are scarce. The work highlights the potential of combining explicit frequency-domain decomposition with foundation model semantics to guide geometric reconstruction, suggesting that relaxing the dependence on frozen foundation models and extending the coarse-to-fine schedule to blind degradations are promising future directions.