Geometry-Aware Fisheye-LiDAR Fusion for Robust 3D Object Detection in Low-Overlap Setups

Technical Summary: Geometry-Aware Fisheye-LiDAR Fusion for Robust 3D Object Detection

Problem Statement
As autonomous systems shift from capital-intensive robotaxis to cost-sensitive logistics (e.g., mining, last-mile delivery), sensor configurations are increasingly optimized for coverage-per-cost. A prevalent sparse-view setup utilizes a single roof-mounted LiDAR paired with dual lateral fisheye cameras. This configuration introduces severe geometric challenges: extreme radial distortion, minimal overlap between cameras, and a fundamental misalignment between the spherical projections of fisheye lenses and the rectilinear grids used in standard Bird's-Eye View (BEV) fusion algorithms.

Standard fusion pipelines force image and point cloud modalities into unified Cartesian grids early in the process. This approach causes significant feature distortion and information loss for wide-view fisheye cameras because the non-uniform distribution of image information (dense near the center, compressed at the periphery) mismatches the uniform spatial resolution of Cartesian grids. While polar BEV grids better align with the radial distribution of fisheye data, they introduce intrinsic challenges for object detection: rigid rectangular objects become severely distorted and rotation-variant in polar coordinates, complicating bounding box regression.

Methodology: Geometry-Aware Hybrid Fusion (GA-HF)
To resolve the trade-off between preserving fisheye geometry and maintaining metric accuracy for detection, the authors propose the Geometry-Aware Hybrid Fusion (GA-HF) framework. This approach decouples feature extraction from object detection by leveraging the strengths of both coordinate representations.

1. Distortion-Aware Polar Lift-Splat-Shoot (LSS):
   Instead of using a standard Cartesian frustum, the framework lifts fisheye image features into a polar BEV grid ($R \times \Theta \times Z$). This module utilizes a generic unprojection interface $\Pi^{-1}$ that adapts to specific fisheye models (e.g., MEI for KITTI-360, Polynomial/OCam for Fisheye3DOD and Dur360BEV). By projecting features directly into a polar grid, the method preserves the native angular density and radial resolution of the fisheye lens, avoiding the interpolation artifacts associated with Cartesian resampling.

2. Hybrid Coordinate Architecture:
   The framework employs a dual-stream design:

   • Polar Stream: Processes fisheye features in the polar domain to maximize angular fidelity.
   • Cartesian Stream: Processes LiDAR point clouds using a sparse voxel encoder (VoxelNet) to generate features in Cartesian space, ensuring metric precision essential for bounding box regression.
   • Alignment: A differentiable coordinate warping step aligns the polar camera features with the Cartesian LiDAR grid via bilinear interpolation.

3. Dual-Attention Warping Correction Module (DA-WCM):
   To bridge the heterogeneous streams and address the quality imbalance caused by warping (which induces stretching artifacts in low-overlap or high-distortion peripheral regions), the authors introduce a Dual-Attention Warping Correction Module based on CBAM.

   • Spatial Attention: Learns a reliability map to adaptively downweight unreliable regions (e.g., blind sectors, severe peripheral distortion) before fusion.
   • Channel Attention: Refines feature channels to enhance high-quality semantic cues.
     This module ensures that the dense but potentially noisy fisheye features do not degrade the high-precision LiDAR representation, allowing the camera stream to contribute semantic context only where deemed reliable.

Key Contributions

1. Distortion-Aware Polar LSS: A novel lifting scheme that projects fisheye features into a polar BEV grid using specific camera models, preserving native angular resolution.
2. Hybrid Coordinate Architecture: A dual-stream framework that decouples polar feature extraction from Cartesian object regression, avoiding the geometric instabilities of pure-polar detection heads while maintaining metric consistency.
3. Dual-Attention Warping Correction: A mechanism that explicitly refines warped camera features using spatial and channel attention to suppress artifacts in low-quality regions prior to fusion.
4. Cross-Dataset Validation: Comprehensive evaluation on three benchmarks (KITTI-360, Dur360BEV, and the synthetic Fisheye3DOD), demonstrating generalizability across real-world and synthetic domains.

Experimental Results
The GA-HF framework was evaluated on three datasets, achieving state-of-the-art performance in fisheye-LiDAR fusion:

• KITTI-360: GA-HF improved the NuScenes Detection Score (NDS) by 4.2% over Cartesian baselines. It significantly outperformed LiDAR-only methods and existing fusion baselines (like BEVFusion and DAL), particularly in orientation error (mAOE), where it reduced errors from >1.0 (random orientation in some baselines) to ~0.78.
• Dur360BEV: Despite imperfect sensor calibration parameters, GA-HF surpassed both LiDAR-only and BEVFusion methods. It achieved the best mAP (0.706) and NDS (0.751), while BEVFusion failed catastrophically in orientation estimation (mAOE: 1.022).
• Fisheye3DOD: GA-HF attained the highest detection score among all fusion methods (NDS: 0.925), significantly reducing orientation error compared to other fusion approaches.

Significance and Claims
The paper claims to be the first approach to explore LiDAR–fisheye camera fusion. It argues that for sparse sensor setups, standard Cartesian fusion is insufficient due to geometric distortion, while pure-polar fusion is unstable for detection. The GA-HF framework validates the necessity of a geometry-aware approach that respects the native sensor geometry during feature extraction but reverts to a metric-consistent Cartesian space for final detection. The results demonstrate that camera features are indispensable for semantic and orientation estimation but require targeted geometric refinement to maintain regression stability. The authors conclude that this hybrid design enables robust perception in cost-effective autonomous systems where sensor redundancy is limited.