NAIMA: Semantics Aware RGB Guided Depth Super-Resolution

1. Problem Statement

Guided Depth Super-Resolution (GDSR) aims to reconstruct high-resolution (HR) depth maps using a low-resolution (LR) depth map and a corresponding HR RGB image. While effective, current methods face a fundamental challenge: RGB-Depth Misalignment.

• The Issue: RGB images contain rich color and texture information that does not always correlate with true geometric depth discontinuities.
• The Consequence: Naive fusion of RGB features with LR depth features often leads to artifacts, blurred object boundaries, and structural inconsistencies in the output depth map, as the model mistakenly follows misleading texture cues rather than geometric edges.
• Current Limitations: Existing multi-task learning approaches (e.g., using semantic segmentation as an auxiliary task) help but often require explicit auxiliary heads or fine-tuning, which may not fully leverage the rich semantic priors available in modern foundation models.

2. Methodology: NAIMA Architecture

The authors propose NAIMA (Neural Attention for Implicit Multi-token Alignment), a framework that integrates global contextual semantic priors derived from a pretrained Vision Transformer (ViT) to guide the depth reconstruction process.

Core Components:

1. Semantic Encoder (DINOv2):

   • The HR RGB image is passed through a frozen DINOv2 backbone.
   • Instead of using raw pixel features, the model extracts intermediate patch-level token embeddings ($\tau_1, \tau_2, \tau_3, \tau_4$) from different transformer blocks. These tokens encode progressively higher-level semantic information.

2. Depth Encoder:

   • The LR depth map is iteratively encoded using a depth encoder ($E_D$) based on residual channel attention blocks, producing feature maps $E_i$.

3. Guided Token Attention (GTA) Module:

   • This is the core innovation. It employs cross-attention to align depth features with semantic tokens.
   • Mechanism:
     • The depth feature map $E_i$ acts as the Query.
     • The projected semantic tokens from DINOv2 act as Key and Value.
     • The tokens are reshaped and projected via convolution to introduce local spatial interactions, then upsampled (pixel shuffle) to match the depth feature dimensions.
     • A scaled dot-product attention mechanism computes the correlation, allowing the depth encoder to "query" relevant semantic context from the RGB image.
   • Refinement: The resulting semantics-infused depth features are fused with spatial RGB features (extracted via a separate encoder) to produce refined depth features for the next level.

4. Upsampling and Reconstruction:

   • The final refined features are passed through an upsampling network ($\phi$) and added to the bicubic-upsampled input depth map to generate the final HR depth map.

5. Loss Function:

   • The model uses a gradient-aware pixel loss, combining a standard $L_1$ reconstruction loss with an $L_1$ loss on spatial gradients ($\nabla_x, \nabla_y$). This ensures the preservation of sharp edges and structural details.

3. Key Contributions

• Semantics-Driven Framework: Introduces a novel approach to GDSR that explicitly utilizes pretrained token representations from a Vision Transformer (DINOv2) as a semantic prior, rather than relying solely on raw RGB spatial features.
• Guided Token Alignment (GTA): Proposes a cross-attention module that iteratively injects global semantic context into depth features. This allows the model to selectively filter out misleading RGB texture noise while preserving true geometric boundaries.
• State-of-the-Art Performance: Demonstrates that distilling semantic knowledge from foundation models significantly outperforms existing GDSR methods across multiple datasets and scaling factors.

4. Experimental Results

The authors evaluated NAIMA on four major datasets: NYU v2, RGBDD, Middlebury, and Lu, as well as the newer TOFDSR dataset.

• Quantitative Performance:

  • NAIMA consistently outperformed 11 benchmark methods (including DKN, FDKN, SGNet, DORNet).
  • 16x Scaling Factor: Achieved significant RMSE improvements over the second-best method:
    • 3.35% on NYU v2
    • 3.14% on RGBDD
    • 6.78% on Middlebury
    • 5.92% on Lu
  • On the TOFDSR dataset, NAIMA also achieved the lowest RMSE for 4x, 8x, and 16x scaling factors compared to SGNet and DCTNet.

• Qualitative Analysis:

  • Visual comparisons show NAIMA produces sharper edges and better-defined corner structures.
  • Error maps indicate NAIMA has the lowest error distribution, particularly near depth boundaries where other methods suffer from blurring due to RGB noise.

• Ablation Studies:

  • Removing Cross-Attention: Replacing the GTA module with simple element-wise addition caused a significant performance drop, proving that explicit attention-based alignment is crucial for effective semantic transfer.
  • Removing Gradient Loss: Training with only $L_1$ loss resulted in noticeable degradation, confirming the necessity of gradient constraints for preserving structural fidelity.

5. Significance

NAIMA represents a paradigm shift in guided depth super-resolution by moving away from purely spatial feature fusion toward semantic-aware fusion.

• Robustness: By leveraging the semantic understanding of foundation models (DINOv2), the method is robust against the "noise" of RGB textures that typically confuse depth reconstruction algorithms.
• Generalization: The approach demonstrates strong generalization capabilities, performing well even on datasets not seen during training (cross-dataset validation) and on newer datasets like TOFDSR.
• Efficiency: The use of a frozen pretrained backbone and a lightweight attention module allows for high performance without the heavy computational cost of training a massive model from scratch or using complex multi-task auxiliary heads.

In conclusion, NAIMA effectively solves the cross-modal misalignment problem in GDSR by treating semantic tokens as a distinct, guiding modality, setting a new state-of-the-art for depth map reconstruction.