LiftFormer: Lifting and Frame Theory Based Monocular Depth Estimation Using Depth and Edge Oriented Subspace Representation

1. Problem Statement

Monocular Depth Estimation (MDE) aims to reconstruct the 3D structure of a scene from a single 2D image. This is a highly ill-posed problem because multiple 3D scenes can produce the same 2D image.

• Current Limitations:
  • Direct Regression: Predicting depth values directly from RGB features is difficult due to the vast range of depth values and the gap between color features (appearance) and geometric depth values.
  • Classification-Regression Mismatch: State-of-the-art methods often use a classification-regression approach (e.g., AdaBins), where depth is discretized into bins. However, if depth bins and depth features are learned independently without interaction, the resulting bin probabilities are often misaligned with the features, leading to suboptimal performance.
  • Edge Artifacts: Transformer-based models, while good at global context, often lack sensitivity to local inductive biases, resulting in poor performance around object edges where depth changes sharply.

2. Methodology: LiftFormer

The authors propose LiftFormer, a framework that utilizes Lifting Theory and Frame Theory to bridge the gap between image features and depth values. The architecture follows a U-Net structure with a Swin Transformer encoder, but introduces two novel subspace transformation modules.

A. Theoretical Foundation

• Lifting Theory: Instead of mapping image features directly to depth values (a difficult function), the method "lifts" the problem to a higher-dimensional covering space. The depth prediction is transformed into generating features in a subspace that corresponds to depth bins, making the learning process more robust.
• Frame Theory: The authors construct an overcomplete frame (a set of linearly dependent vectors) to represent the depth subspace. This provides redundancy and robustness compared to standard orthogonal bases.

B. Core Modules

The framework consists of two primary "lifting" modules:

1. SF-DGR (Image Spatial Feature to Depth-Oriented Geometric Representation) Subspace Transformation:

   • Goal: To transform discrete image spatial features into continuous depth-oriented features that align with depth bin centers.
   • Mechanism:
     • A DGR Subspace is constructed using a set of learnable basis vectors (frame) that correspond to depth bins. These vectors are learned globally as latent embeddings, independent of specific image color information.
     • An SF-DGR transformation projects encoder features into this DGR subspace.
     • A DGR Coefficient Prediction (DGR-CP) module calculates projection coefficients using inner products between transformed spatial features and the basis vectors.
     • The result is a set of redundant, robust depth features that represent continuous depth variations rather than discrete bin probabilities.
   • Benefit: This theoretically validates the use of "bin embeddings" and transforms the problem from predicting discrete probabilities to generating continuous geometric features.

2. DF-ER (Depth Feature to Edge-Aware Representation) Subspace Transformation:

   • Goal: To enhance local edge information and correct sharp depth transitions, which are often blurry in standard Transformer outputs.
   • Mechanism:
     • An ER Subspace is constructed with two basis vectors: one for "edge" and one for "non-edge."
     • A CNN-based ER Coefficient Prediction (ER-CP) module analyzes decoder depth features to predict the probability of a pixel being an edge.
     • The depth features are lifted into this ER subspace, effectively separating and enhancing edge-related high-frequency information.
   • Benefit: This explicitly injects edge awareness into the depth features, improving boundary sharpness without requiring explicit edge supervision.

C. Prediction Head

The final depth map is generated using an AdaBin-style approach. The high-level DGR features are used to predict bin centers (via a Bin Center Predictor), and the final depth is calculated as a weighted sum of these centers and their probabilities.

3. Key Contributions

1. Theoretical Formulation: The paper provides a theoretical justification for using bin-embedding-like representations in MDE by formulating depth prediction as a lifting problem from image space to a depth-oriented geometric representation (DGR) subspace.
2. SF-DGR Module: Introduces a subspace transformation that converts image spatial features into continuous depth features using a frame-theory-based overcomplete basis, solving the misalignment between features and depth bins.
3. DF-ER Module: Proposes an edge-aware subspace transformation that lifts depth features to enhance local high-frequency information, specifically addressing the edge-blurring issue common in Transformer-based MDE.
4. State-of-the-Art Performance: The method achieves superior results on standard benchmarks compared to existing CNN and Transformer-based methods.

4. Experimental Results

The method was evaluated on the KITTI (outdoor) and NYU Depth V2 (indoor) datasets.

• KITTI Dataset (Eigen Split):
  • RMSE: 2.038 (outperforming PixelFormer at 2.081 and BinsFormer at 2.098).
  • Abs Rel: 0.050.
  • Qualitative: The generated depth maps are smoother and clearer, with significantly fewer discontinuities at object boundaries compared to PixelFormer.
• NYU Depth V2 Dataset:
  • RMSE: 0.313 (outperforming PixelFormer at 0.322 and LocalBins at 0.351).
  • Qualitative: Better preservation of local structures (e.g., chair legs) and reduced noise.
• Ablation Studies:
  • Both SF-DGR and DF-ER modules independently improve performance.
  • Redundancy Analysis: Using 128 basis vectors (with 32 dimensions) in the DGR subspace yielded the best balance between performance and complexity; excessive redundancy (256 vectors) slightly degraded performance.
  • Generalization: The modules work effectively with both CNN-based and Transformer-based decoders.
  • Depth Range: The model shows significant improvement in both near (0-50m) and full (0-80m) depth ranges compared to P3Depth.

5. Significance

• Bridging Modalities: LiftFormer addresses the fundamental difficulty of mapping 2D color features to 3D geometric depth by introducing an intermediate, theoretically grounded subspace.
• Beyond Discretization: While many methods rely on discrete bin classification, LiftFormer uses the lifting framework to generate continuous depth features, allowing for smoother and more accurate regression.
• Edge Preservation: By explicitly modeling edges in a separate subspace, the method overcomes a known weakness of Transformer-based vision models, which often struggle with high-frequency local details.
• Theoretical Validation: The work moves MDE from purely empirical deep learning designs to a framework supported by Lifting and Frame theory, offering a new perspective on how to structure depth estimation networks.