Adaptive Depth-converted-Scale Convolution for Self-supervised Monocular Depth Estimation

1. Problem Statement

Monocular Depth Estimation (MDE) aims to recover 3D scene structure from 2D images. While self-supervised methods (using video reconstruction errors) avoid the need for expensive ground-truth depth labels, they face a critical challenge: Scale Ambiguity.

• The Core Issue: In a monocular video captured by a moving camera, the apparent size of an object changes continuously as its depth changes (an object appears smaller when far away and larger when close).
• Limitation of Existing Methods: Standard Convolutional Neural Networks (CNNs) use fixed-size convolution kernels (e.g., $3 \times 3$) and fixed-scale pooling. These fixed receptive fields cannot adapt to the continuous size variations of objects across frames.
• The Gap: Most prior research focuses on local deformation of filters (e.g., Deformable Convolution) to handle shape changes, but largely ignores the adaptive scale of the convolution kernel itself. The relationship between an object's depth and its required receptive field scale is underutilized.

2. Methodology

The authors propose a framework that integrates depth information directly into the convolution process to create adaptive-scale receptive fields. The framework consists of three main components:

A. Depth-to-Scale Conversion Principle

Based on the pinhole camera model, the size of an object in an image ($l_v$) is inversely proportional to its depth ($D$):
$$l_v = l_r \cdot \frac{f}{D}$$
Where $l_r$ is the real-world size and $f$ is the focal length.
The authors leverage this to determine the optimal convolution kernel size ($k_d$) for a specific depth $D$:
$$k_d = k_r \cdot \frac{D_r}{D}$$
Where $k_r$ is a reference kernel size (e.g., $3 \times 3$) at a reference depth $D_r$. As depth increases, the kernel size decreases, and vice versa.

B. Core Modules

1. Depth-converted-Scale Convolution (DcSConv):

   • Instead of using a fixed grid for sampling, DcSConv dynamically adjusts the receptive field size based on the estimated depth map.
   • It uses bilinear interpolation to sample features at fractional coordinates when the calculated kernel size is non-integer.
   • This allows the network to extract features at the "correct" scale for objects at different depths, capturing details for distant objects (small kernel) and global context for near objects (large kernel).
   • Generalization: The authors also propose GLSConv (Generalized Learned-Scale Convolution), where the depth-to-scale mapping is learned end-to-end from features rather than using a fixed mathematical formula.

2. Depth-converted Multiple Scale Fusion (DMSF):

   • An alternative approach where multi-scale features (e.g., $1 \times 1$, $3 \times 3$, $5 \times 5$) are extracted in parallel.
   • These features are fused using weights calculated via a Gaussian distance function based on the difference between the depth-converted scale and the filter sizes.

3. Depth-converted-Scale aware Fusion (DcS-F):

   • A module designed to combine the adaptive DcSConv features with standard fixed-scale convolution features.
   • It employs a Channel Attention and Spatial Attention mechanism.
   • Crucially, it injects the scale difference map (normalized depth information) into the attention blocks. This allows the network to weigh the importance of the adaptive features vs. fixed features based on the local object scale, preserving both fine details and global structure.

C. Architecture Integration

• The method is designed as a plug-and-play module for existing Encoder-Decoder architectures (e.g., Monodepth2, CADepth).
• It replaces standard decoding blocks with a Depth-converted-Scale aware feature Decoding (DcS-D) module, which progressively refines depth maps using the adaptive convolution and fusion logic.

3. Key Contributions

1. Novel Convolution Paradigm: First work to investigate adaptive convolution scale based on depth, rather than just local filter deformation. It establishes that scale adaptation is critical for MDE.
2. DcSConv & DMSF: Proposed two methods to achieve scale-aware feature extraction: direct adaptive sampling (DcSConv) and multi-scale fusion (DMSF).
3. DcS-F Module: A specialized fusion mechanism that uses depth-derived scale information to intelligently combine adaptive and standard features via attention mechanisms.
4. Plug-and-Play Design: The modules are architecture-agnostic and can be integrated into any existing CNN-based depth estimation network.

4. Experimental Results

The method was evaluated on the KITTI benchmark (Eigen split) and other datasets (Make3D, NYU V2).

• Performance Gains:
  • On Monodepth2 (ResNet-18), the proposed method reduced SqRel by 11.6% (from 0.903 to 0.798) and improved AbsRel.
  • On CADepth, it achieved a 17.1% improvement in SqRel.
  • On MonoViT (Transformer backbone), it achieved a 23.4% improvement in SqRel, demonstrating effectiveness across different backbone architectures.
• Ablation Studies:
  • DcSConv vs. Deformable Conv: DcSConv significantly outperformed Deformable Convolution, proving that scale adaptation is more important than local shape deformation for depth estimation.
  • DcSConv vs. Fixed-Scale: Simply adding more fixed-scale layers did not yield similar improvements, validating the necessity of adaptive scaling.
  • Fusion: The DcS-F module outperformed simple concatenation, summation, and standard CBAM attention, highlighting the value of injecting scale information into the fusion process.
• Generalization: The method showed strong performance on indoor (NYU V2) and outdoor (Make3D) datasets, indicating robust generalization.

5. Significance

This paper addresses a fundamental limitation in self-supervised monocular depth estimation: the inability of fixed-kernel CNNs to handle continuous scale changes caused by camera motion. By mathematically linking depth to convolution scale, the authors provide a theoretically grounded and empirically superior solution.

• Theoretical Impact: It shifts the focus from "deforming the filter shape" to "adapting the filter scale," offering a new perspective on how to handle geometric variations in vision tasks.
• Practical Impact: As a plug-and-play module, it offers a straightforward way to boost the performance of existing state-of-the-art depth estimation models without requiring massive architectural overhauls or additional ground-truth data.
• Efficiency: The method achieves significant accuracy improvements with a manageable increase in parameters, making it suitable for real-world applications like autonomous driving and robotics.