Horizon3D: Sparse Radar-Camera Fusion for Long-Range 3D Perception in Autonomous Driving

Technical Summary: Horizon3D

Problem Statement
Reliable long-range 3D object detection is critical for safe autonomous driving, particularly at highway speeds where braking distances require accurate perception of objects beyond 150 meters. While LiDAR-based methods have dominated this space, their high cost and vulnerability to adverse weather limit large-scale deployment. Radar-camera fusion offers a cost-effective alternative, yet existing methods face significant limitations in long-range scenarios.

• BEV-based methods effectively encode scene-level context but incur rapidly growing computational costs as the detection range increases. Furthermore, their uniform dense grids struggle to preserve fine-grained object-level details.
• Query-based methods provide efficient, object-centric encoding but often lack sufficient scene-level context.
• Temporal Fusion Challenges: Long-range objects produce sparse radar returns and occupy few image pixels, necessitating multi-frame accumulation. Conversely, high-speed motion causes large inter-frame displacements, requiring precise object-level motion modeling. Existing approaches either rely on dense BEV aggregation (computationally expensive) or propagate object queries (lacking scene context), failing to jointly address per-frame sparsity and object-level dynamics efficiently.

Methodology: Horizon3D
The authors propose Horizon3D, a sparse radar-camera fusion framework that employs a hybrid representation combining Gaussian primitives with sparse Bird's Eye View (BEV) features. The architecture consists of three primary modules:

1. Keypoint-Guided Gaussian Initialization (KGGI):
   Instead of maintaining a large set of Gaussians across the entire scene, KGGI estimates object keypoints from both radar and camera features to initialize a compact set of Gaussian primitives.

   • Radar Branch: Uses VoxelNeXt to generate sparse radar BEV features, from which top-scoring cells are selected as radar keypoints.
   • Camera Branch: Utilizes a 2D keypoint head and depth head to lift top-scoring candidates to 3D, projecting them as camera keypoints.
   • Initialization: Gaussian primitives are anchored at these estimated keypoints (plus a small number of random and history Gaussians). Each Gaussian is parameterized by mean, scale, rotation, opacity, velocity, and a feature vector. This approach ensures object-centric encoding without redundant spatial coverage.

2. Object-Centric Sparse Fusion (OCSF):
   This module aggregates cross-modal features around the initialized Gaussian primitives and iteratively refines them.

   • Multimodal Gaussian Encoder: Over multiple layers, the encoder uses deformable cross-attention to gather modality-specific contexts (radar BEV and multi-view image features) around each Gaussian's mean.
   • Refinement: The Gaussian parameters (mean, scale, rotation, etc.) are updated based on the aggregated features to better align with 3D object geometry.
   • Object-Centric Splatting: The refined Gaussians are splatted onto the BEV plane. A class-agnostic occupancy map is generated, and "Active Gaussian Sets" are pooled via Gaussian-weighted aggregation to create a sparse, object-centric BEV representation. This is fused with sparse radar BEV features to combine fine-grained object details with scene-level context, avoiding dense BEV construction.

3. Dual-Path Temporal Fusion (DPTF):
   To handle long-range sparsity and high-speed motion, DPTF integrates temporal cues through two complementary paths, both utilizing predicted velocities for motion compensation:

   • BEV Path: Accumulates foreground BEV features from multiple frames. It predicts occupancy and planar velocity for non-empty cells, storing only these features in a memory bank. Historical frames are warped to the current frame using ego-motion and velocity-guided warping before being stacked and fused via sparse convolutions.
   • Gaussian Path: Propagates past Gaussians to the current frame. A spatially diverse subset of history Gaussians (selected via Farthest Point Sampling) is warped using predicted velocities and ego-motion. These are merged with current-frame Gaussians and processed through the OCSF pipeline, with additional temporal self-attention to handle residual misalignment.

Key Contributions

• Hybrid Representation: The introduction of a sparse framework that jointly captures object-level detail and scene-level context using Gaussian primitives and sparse BEV features.
• KGGI and OCSF: A mechanism to initialize Gaussians at sensor-estimated keypoints and iteratively refine them, enabling efficient object-centric encoding without dense BEV grids.
• Dual-Path Temporal Fusion (DPTF): A novel temporal fusion strategy that simultaneously addresses per-frame sparsity (via BEV accumulation) and object-level motion (via Gaussian propagation), utilizing explicit velocity compensation in both paths.
• State-of-the-Art Performance: Demonstrated effectiveness on the TruckScenes benchmark for long-range, high-speed scenarios.

Experimental Results
Evaluated on the TruckScenes dataset (featuring up to 150m detection ranges and high speeds), Horizon3D achieves state-of-the-art performance among radar-camera fusion methods:

• Performance: On the validation set, Horizon3D outperforms the previous best radar-camera method (SpaRC) by +3.0 NDS and +1.6 mAP. It also surpasses the previous best BEV-based method (CRT-Fusion) by a significant margin.
• LiDAR Comparison: Notably, Horizon3D exceeds the performance of CenterPoint-V (a LiDAR-based method) by +0.8 NDS and +1.0 mAP on the test split, despite using lower-resolution image inputs.
• Efficiency: The method maintains a sparse representation, achieving competitive inference speeds (8.5 FPS) and moderate memory usage (3.2 GB), significantly outperforming dense BEV methods in efficiency while avoiding the accuracy drop of query-based methods.
• Ablation Studies: Experiments confirm that KGGI initialization, bounding box enlargement for occupancy supervision, and the combination of both BEV and Gaussian paths in DPTF are critical for performance. Distance-wise analysis shows consistent gains across all ranges, with particularly notable improvements at 25–50m and 100–150m.

Significance
The paper claims that Horizon3D addresses the fundamental trade-off in long-range perception between computational efficiency and the need for both scene-level context and object-level detail. By avoiding dense BEV construction while still capturing temporal dynamics and scene context through a hybrid Gaussian-sparse BEV approach, the framework enables robust, high-speed autonomous driving perception using cost-effective radar-camera sensor suites. The results suggest that this hybrid approach is a viable and superior alternative to both dense BEV and pure query-based paradigms for extended-range detection.