JustDepth: Real-Time Radar-Camera Depth Estimation with Single-Scan LiDAR Supervision

Problem Statement

Accurate, low-latency 3D depth estimation is critical for autonomous systems. While cameras provide dense appearance cues, they lack metric scale. Conversely, automotive mmWave radar offers metric range and weather robustness but produces sparse, noisy data with limited angular resolution. Existing radar-camera fusion pipelines often suffer from high latency and complexity due to multi-stage architectures, reliance on pre-trained monocular depth models, or the need for auxiliary annotations (e.g., semantic masks, bounding boxes) and multi-scan LiDAR accumulation to construct ground truth. Furthermore, training with single-scan LiDAR supervision often leads to LiDAR Distribution Leakage (LDL), where the model learns to reproduce the scanner's specific sampling pattern, resulting in visible horizontal stripe artifacts in the predicted depth maps.

Methodology

The authors propose JustDepth, a single-stage radar-camera depth estimator designed for real-time inference without auxiliary annotations. The architecture consists of three primary components:

1. Fixed-Width Radar Encoding:

   • Radar returns are projected onto the image plane, retaining only the horizontal pixel coordinate ($u$) and range.
   • These points are rasterized into a fixed-width 1D grid of width $W$. For each column, the minimum range is stored; empty columns are zeroed.
   • A 1D convolutional backbone processes this grid, producing a latent feature map $r_L \in \mathbb{R}^{B \times C \times w}$. Crucially, because the grid width is fixed, the encoder's computation and latency remain constant regardless of the number of raw radar points.

2. Height Fusion Block:

   • This module aligns the 2D image features ($i_L$) and the 1D radar features ($r_L$).
   • The radar feature is repeated along the height dimension to match the image feature height, creating a column-wise token sequence of size $h \times 2C$.
   • A height-wise self-attention mechanism processes each column independently, allowing the model to learn consistent vertical positions for radar priors within each image column. This is followed by a feed-forward network and a lightweight convolutional module to restore 2D context.

3. Graph-Based Global Propagation:

   • A Graph Neural Network (GNN) with $N$ layers propagates depth cues globally.
   • Nodes represent spatial tokens in the feature map. Edges are constructed using a K-Nearest Neighbors (K-NN) search in feature space.
   • The network employs Max-Relative Graph Convolution (MR-Conv), where messages are formed by the maximum difference between a node and its neighbors. This allows depth information to propagate from radar-supported or LiDAR-supervised regions to unsupervised areas, even across large distances if the features are similar.

4. Decoders:

   • Depth Decoder: A U-Net-style upsampling path fuses the globally propagated features with skip connections from the image encoder to produce the final dense depth map.
   • Confidence Decoder (Training Only): An auxiliary head predicts a per-pixel confidence map indicating how strongly a pixel's depth is supported by radar. This branch is used solely for supervision during training to stabilize learning and is discarded at inference, incurring zero test-time cost.

5. LiDAR Distribution Leakage (LDL) Mitigation:

   • To prevent the model from overfitting to the horizontal scanlines of single-scan LiDAR, the authors employ two strategies:
     • Point Upsampling: Midpoints are inserted between suitable pairs of LiDAR points to densify supervision.
     • Synchronized Rotation with Reflection Padding: The image, radar, and LiDAR are rotated by the same random angle. Reflection padding is applied to the image to prevent black borders from acting as rotation cues. This forces the model to learn depth consistency regardless of stripe orientation.

Key Contributions

• Single-Stage, Constant-Latency Architecture: JustDepth encodes all radar returns into a fixed-width 1D representation, decoupling runtime from point count and eliminating the need for multi-stage refinement or pre-trained monocular modules.
• Training-Only Confidence Decoder: A novel auxiliary head that supervises the fusion module to localize radar-supported pixels without generating an intermediate depth map. It improves accuracy without adding latency at inference.
• LDL Mitigation Strategy: A combination of point upsampling and synchronized rotation with reflection padding effectively reduces stripe artifacts.
• VHGR Metric: The authors introduce the Vertical-Horizontal Gradient Ratio (VHGR) to quantitatively measure the imbalance between vertical and horizontal depth gradients, serving as a specific metric for quantifying stripe artifacts caused by LDL.

Experimental Results

Evaluated on the nuScenes dataset (0–50m, 0–70m, and 0–80m ranges):

• Efficiency: JustDepth achieves an inference time of 14.8 ms per frame on an NVIDIA RTX 4070 Ti. This is approximately 39.7× faster than the state-of-the-art GET-UP method and significantly faster than other competing approaches, enabling real-time deployment.
• Accuracy: Despite using only a single LiDAR sweep and no auxiliary annotations, JustDepth maintains competitive accuracy. It achieves an MAE of 1324.2 mm (0–50m) and 1674.0 mm (0–70m), outperforming Li et al. [10] and achieving the best AbsRel (0.074) among compared methods.
• Artifact Reduction: JustDepth achieves the lowest VHGR (0.1767) among all compared methods, indicating a significant reduction in horizontal stripe artifacts compared to methods like GET-UP (0.5210) and Li et al. (0.1868).
• Ablation Studies: Removing the GNN, the Height Fusion Block's self-attention, or the LDL mitigation strategies (rotation/upsampling) consistently degrades performance. Notably, while the LDL mitigation strategies slightly increase standard error metrics (MAE/RMSE) by preventing the model from "cheating" by matching the sparse LiDAR pattern, they drastically reduce VHGR and produce visually cleaner, more physically plausible depth maps.

Significance and Claims

The paper claims that JustDepth represents a significant step toward practical, real-time radar-camera perception. By demonstrating that high-quality, dense, metric-scale depth can be achieved using only single-scan LiDAR supervision without auxiliary labels or complex multi-stage pipelines, the method enhances portability across datasets and sensors. The authors emphasize that their approach successfully balances the trade-off between latency and accuracy, achieving real-time performance while mitigating the specific artifact issues (striping) inherent to single-scan supervision. The introduction of VHGR provides a necessary tool for evaluating the visual quality of depth maps beyond standard error metrics, highlighting that lower MAE does not always equate to better structural fidelity in the presence of LDL.