Technical Summary: RbFT-Net

Problem Statement

Dense metric depth prediction is critical for autonomous systems, yet relying solely on cameras lacks metric scale, while LiDAR is costly. Millimeter-wave radar offers direct metric ranging and robustness to lighting but suffers from extreme sparsity, clutter, multipath reflections, and projection errors. While aggregating multiple radar frames (temporal fusion) can increase measurement density, conventional approaches that directly accumulate or geometrically warp these frames introduce significant issues:

1. Temporal Misalignment: Rigid warping does not account for calibration uncertainty or dynamic objects, leading to inaccurate anchors.
2. Error Propagation: Accumulating noisy returns without verification can corrupt large regions of the depth map, particularly around object boundaries and thin structures.
3. Dependency: Existing high-performance methods often rely on "plug-in" architectures that require auxiliary monocular depth models, increasing pipeline complexity and model size.

The core challenge addressed is how to exploit denser multi-frame radar data without allowing its inherent noise and misalignment to degrade the final depth prediction.

Methodology: RbFT-Net

The authors propose RbFT-Net (Rectify-Before-Fuse Temporal Radar Anchors), an end-to-end framework that treats accumulated multi-frame radar returns not as ground truth, but as noisy temporal anchor candidates. The architecture follows a "rectify-before-fuse" paradigm, consisting of three primary stages:

1. Image-Conditioned Anchor Rectification

Before propagation, the network processes raw temporal anchors ($A_t$) using an image-conditioned module to correct their locations and depths while estimating reliability.

• Radar Query Sampler (RQS): Instead of sampling image features at the potentially inaccurate projected radar location, RQS predicts radar-conditioned sampling offsets. This allows the network to gather visual evidence from the surrounding area to correct the anchor.
• Radar-Centered Attention (RCA): This module models local consistency among radar candidates using self-attention and weighted pooling. It then retrieves compatible visual evidence via cross-attention between the pooled radar features and the sampled image features.
• Multi-Scale Anchor Fusion (MAF): Features from multiple scales are integrated to preserve both local boundaries and global context.
• Prediction Heads: Three lightweight heads predict:
  • Spatial offset ($\Delta p_i$) to correct the image-plane location.
  • Depth residual ($\Delta z_i$) to correct the metric depth.
  • Pointwise reliability ($c_i \in [0, 1]$) to estimate the confidence of the anchor.
    The output is a set of rectified anchors ($\hat{A}_t$) with corrected coordinates, depths, and reliability scores.

2. Reliability-Aware Anchor Propagation

Rather than propagating all nearby anchors, the network selectively transfers metric evidence to dense image regions based on two criteria:

• Target–Anchor Compatibility: A learned compatibility score evaluates how well an anchor's features and depth match the target image location.
• Reliability Modulation: The predicted confidence ($c_i$) acts as a weight.
  The system selects the top-$K_s$ (set to 4) most compatible anchors for each target pixel and aggregates their metric evidence using weights derived from both compatibility and reliability. This suppresses the influence of unreliable or misaligned measurements before they spread across the image.

3. Multi-Modal Fusion and Refinement

The propagated feature map, combined with the current frame's raw radar measurements and RGB features, is fed into a lightweight Multi-Modal Fusion Network (MFN) based on the CSPN++ architecture. This stage performs high-level fusion and dense refinement to produce the final metric depth map ($D_t$).

Key Contributions

1. Rectify-Before-Fuse Framework: An end-to-end pipeline that rectifies noisy temporal radar anchors (correcting location, depth, and estimating reliability) prior to propagation, preventing error accumulation.
2. Image-Conditioned Rectification Module: A novel mechanism using RQS and RCA to jointly correct spatial and metric errors of radar returns using visual context, without assuming the raw accumulation is accurate.
3. Reliability-Aware Propagation Strategy: A selection mechanism that combines learned target-anchor compatibility with pointwise reliability scores to selectively propagate trustworthy evidence, avoiding the contamination of dense regions by noisy returns.
4. Independent Pipeline & New Dataset: The method operates without auxiliary monocular depth models, offering a compact solution. The authors also introduce a new 4D radar–camera–LiDAR dataset collected on a different platform to validate cross-platform generalization.

Experimental Results

The method was evaluated on the ZJU-4DRadarCam benchmark and the newly collected dataset.

• Performance on ZJU-4DRadarCam: RbFT-Net achieved the best overall performance among independent methods. It reduced Mean Absolute Error (MAE) by approximately 10% and inverse MAE (iMAE) by ~35% compared to the strongest independent baselines. Notably, it remained competitive with, and often surpassed, plug-in methods (like TacoDepth) that rely on large auxiliary monocular depth models, despite having a significantly smaller parameter count.
• Cross-Platform and Transferability: In zero-shot transfer (trained on ZJU-4DRadarCam, tested on the new dataset), RbFT-Net outperformed other methods by a significant margin (reducing MAE by ~22.6% relative to JustDepth). With limited-data adaptation (10% of target data), it achieved the best results across all metrics.
• Efficiency: Despite processing five radar frames, RbFT-Net runs at 44.88 FPS on an NVIDIA RTX PRO 5000, using only ~44M parameters (roughly 1/3 of the parameter count of plug-in pipelines including their auxiliary models).
• Ablation Studies:
  • Both spatial and depth rectification contributed to RMSE reductions (258.2 mm and 180.2 mm respectively).
  • Reliability-aware propagation significantly outperformed direct 4-NN aggregation.
  • Increasing the number of radar frames from 1 to 5 yielded substantial accuracy gains for RbFT-Net (RMSE reduction of 246.5 mm), whereas baseline methods showed diminishing returns, highlighting the effectiveness of the rectification strategy in handling temporal noise.

Significance and Claims

The paper claims that RbFT-Net addresses the fundamental limitation of current radar-camera depth completion: the tendency to propagate unverified, noisy temporal data. By treating accumulated radar returns as candidates that require image-conditioned correction and reliability estimation, the method successfully exploits the density of multi-frame data without sacrificing accuracy.

The authors emphasize that RbFT-Net provides a compact, independent solution that rivals complex plug-in pipelines. Its ability to generalize across different sensing platforms (zero-shot) and adapt with limited data suggests robustness for real-world autonomous driving applications where sensor configurations and data availability may vary. The work demonstrates that "rectifying before fusing" is a more effective strategy for temporal radar integration than simple accumulation or geometric warping.