Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement

Technical Summary: Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement

Problem Statement
Modern perception systems increasingly rely on multi-camera platforms coupled with a single LiDAR sensor to enhance field of view and robustness. While learning-based methods for single camera–LiDAR calibration have advanced, they typically treat each camera–LiDAR pair independently. This approach ignores the rigid geometric coupling inherent in multi-camera rigs, where all cameras share a fixed relationship through the common LiDAR coordinate frame. Consequently, individually accurate per-camera estimates may result in system-level inconsistencies, leading to misalignment artifacts across views. The central question addressed is whether learned pairwise calibration can be improved by jointly optimizing all cameras under the rigid geometric constraints of a shared platform.

Methodology
The authors propose a two-stage framework for joint multi-camera LiDAR extrinsic calibration:

1. Stage 1: Learned Single-Camera Initialization
   The pipeline utilizes CMRNext, a state-of-the-art learning-based method, applied independently to each camera. CMRNext predicts dense optical flow between a rendered LiDAR depth map and the camera image to establish dense 2D–3D correspondences. From these, it estimates an initial extrinsic transformation ($\hat{T}_{L \to C_i}$) for each camera. To ensure robustness, per-frame predictions across a sequence are aggregated (via median) to produce a single initialization estimate per camera.

2. Stage 2: Joint Multi-Frame Bundle Adjustment (BA)
   The initial estimates are refined jointly using a multi-frame bundle adjustment that enforces inter-camera consistency. The optimization minimizes a three-term objective function over all accumulated frames:

   • Reprojection Term: Minimizes the error between projected 3D LiDAR points and their observed 2D correspondences across all cameras.
   • Per-Camera Prior: A regularization term that keeps the optimized extrinsic close to the Stage 1 estimate ($\hat{T}_i$), stabilizing the solution when correspondences are sparse or noisy.
   • Relative-Pose Prior: A novel term that enforces geometric consistency between cameras by penalizing deviations from the relative pose implied by the Stage 1 estimates ($\hat{T}_{12} = \hat{T}_2 \hat{T}_1^{-1}$). This allows information to propagate across cameras, ensuring the final solution respects the rigid rig structure.

   The optimization employs a Cauchy robust loss to down-weight outliers and a two-pass solver strategy (fit-gate-refit) to iteratively remove gross outliers before final refinement.

Key Contributions

• Joint Formulation: A method that uses learned single-camera predictions as initialization but refines all camera extrinsics simultaneously rather than independently, leveraging the rigid structure of multi-camera rigs.
• Explicit Relative-Pose Prior: An optimization constraint that enforces inter-camera geometric consistency, enabling the system to correct individual camera errors using information from other cameras in the rig.
• Empirical Analysis: A comprehensive study comparing in-domain (KITTI) and out-of-domain (Walkley) datasets to determine when joint refinement provides significant gains over standard reprojection-only refinement.

Experimental Results
The method was evaluated on the KITTI dataset (in-domain for CMRNext) and the Walkley dataset (out-of-domain with different sensor configurations).

• KITTI (In-Domain): Since Stage 1 initialization is already strong, the joint refinement offers modest but consistent improvements. At $N=100$ frames, the full method achieved 0.89 cm translation error and 0.038° rotation error on the primary camera, outperforming prior methods (e.g., LCCNet, RGGNet, CMRNet) and the Stage 1 baseline. The joint approach notably improved inter-camera consistency.
• Walkley (Out-of-Domain): Here, Stage 1 predictions were significantly less reliable (e.g., 108.6 cm translation error on the secondary camera). The joint refinement provided substantial recovery:
  • Multi-frame reprojection alone reduced the secondary camera error to 3.57 cm.
  • The full method (including priors) further refined this to 3.1 cm and significantly improved the primary camera's accuracy (from 87.2 cm to 22.6 cm).
  • Crucially, the "priors-only" configuration (without reprojection) failed to recover from large errors, confirming that the gains stem from fitting the learned correspondences while the priors act as stabilizers and consistency enforcers.

Significance and Claims
The paper claims that explicit multi-camera coupling is beneficial, particularly when single-camera predictions are less reliable (out-of-domain scenarios). The proposed framework demonstrates that joint optimization does not trade per-camera accuracy for system consistency; rather, it enhances both. The relative-pose prior is identified as the key mechanism that allows the optimizer to use information from well-constrained views to correct weaker views without allowing arbitrary drift. The authors note that while the method offers limited improvement when initialization is already near-optimal and data is scarce (e.g., KITTI with $N=10$), it provides significant robustness and accuracy gains in challenging or out-of-domain conditions. Future work is suggested for extending the framework to rigs with more than two cameras and exploring end-to-end training.