Technical Summary: BEVCALIB

Problem Statement

Accurate extrinsic calibration between LiDAR and cameras is a prerequisite for fusing multi-modal perception in autonomous driving and robotic systems. Even minor misalignments (a few degrees in rotation or centimeters in translation) can compound over distance, significantly degrading downstream task performance. Traditional calibration methods rely on controlled environments with specialized targets (e.g., checkerboards), which are tedious and fail to account for transformation changes during vehicle movement. While target-less, data-driven approaches have emerged, many rely on feature matching or latent space alignment that does not explicitly enforce geometric constraints. Furthermore, existing learning-based methods often struggle with robustness under significant noise or require iterative refinement, limiting their efficiency and generalizability in real-world "in-the-wild" scenarios.

Methodology

The paper proposes BEVCALIB, the first target-less LiDAR-camera calibration model that utilizes Bird's-Eye View (BEV) representations to perform calibration directly from raw data. The architecture operates in three primary stages:

1. BEV Feature Extraction:

   • LiDAR Branch: Processes the input point cloud using a sparse convolutional backbone to generate 3D voxel features, which are then flattened into 2D BEV features ($B^2D_L$).
   • Camera Branch: Extracts 2D image features using a 2D backbone (Swin-Transformer) and lifts them into 3D frustum features using the Lift-Splat-Shoot (LSS) module. These are projected into the world coordinate system using an initial extrinsic guess ($T_{init}$) and pooled into BEV features ($B^2D_C$).
   • Fusion: The modality-specific BEV features are fused via a $1 \times 1$ convolution into a shared BEV feature space ($F_B$). A Feature Pyramid Network (FPN) is applied to enhance multi-scale geometric information.

2. Geometry-Guided BEV Decoder (GGBD):

   • Feature Selector: A novel component that filters BEV features based on geometric relevance. It projects 3D camera feature positions into the BEV grid to identify overlapping regions between the camera and LiDAR fields of view. This acts as an implicit geometric matcher, selecting only spatially meaningful features and eliminating redundancy.
   • Refinement Module: The selected features are processed using vanilla self-attention to capture cross-modal relationships. An average-pooling operation aggregates high-level information, which is then passed through separate Multi-Layer Perceptrons (MLPs) to predict the translation vector and rotation quaternion.

3. Optimization:

   • The model predicts a correction matrix ($T_{pred}$) to compensate for noise injected into the initial extrinsic guess ($T_{init}$).
   • Training utilizes a combined loss function consisting of:
     • Rotation Loss: Geodesic loss based on quaternion distance and a normalization loss.
     • Translation Loss: Smooth-L1 loss.
     • Reprojection Loss: Point cloud reprojection loss to supervise the alignment of transformed point clouds.

Key Contributions

• First BEV-based Calibration: BEVCALIB is the first model to leverage BEV features for LiDAR-camera extrinsic calibration, moving beyond depth-image or bounding-box priors to utilize raw sensor data.
• Geometry-Guided Feature Selection: The introduction of a feature selector that filters BEV features based on 3D positional anchors ensures the model focuses on overlapping, geometrically consistent regions, reducing memory consumption and improving training efficiency.
• Robustness to Noise: The method is designed to handle significant initial misalignments (up to $\pm 1.5$m translation and $\pm 20^\circ$ rotation) in a single-pass inference, avoiding the need for iterative refinement.
• Open-Source Advancement: The authors provide code and a new dataset (CALIBDB) with heterogeneous extrinsics, addressing the scarcity of high-performing, reproducible open-source calibration tools.

Experimental Results

The authors evaluated BEVCALIB on the KITTI, NuScenes, and their proprietary CALIBDB datasets under various noise conditions.

• Performance vs. Literature Baselines:
  • On KITTI, BEVCALIB outperformed the best literature baselines by an average of 47.08% in translation and 82.32% in rotation.
  • On NuScenes, it improved upon the best baselines by 78.17% in translation and 68.29% in rotation.
  • The model achieved near-perfect rotation accuracy (errors $< 0.2^\circ$) across Roll, Pitch, and Yaw axes.
• Performance vs. Reproducible Open-Source Baselines:
  • BEVCALIB improved the best reproducible open-source results by approximately one order of magnitude on KITTI and NuScenes.
  • On the challenging CALIBDB dataset (featuring dynamic and heterogeneous extrinsics), it outperformed the best open-source baselines by 60.21% in translation and 24.99% in rotation.
• Qualitative Analysis: Visual overlays of LiDAR point clouds onto camera images demonstrated that BEVCALIB achieves fine-grained alignment, whereas baseline methods exhibited significant misalignment, particularly with vertical structures and distant objects.

Significance and Claims

The paper claims that BEVCALIB establishes a new state-of-the-art for learning-based LiDAR-camera calibration. By explicitly enforcing geometric constraints through BEV representations and a geometry-guided selection mechanism, the model overcomes the limitations of previous methods that rely on feature matching or lack geometric rigor. The authors emphasize that BEVCALIB fills a critical gap in the open-source community, providing a robust, single-stage calibration tool that generalizes well to diverse and noisy real-world scenarios without requiring specialized targets or iterative inference.