A Pose-only Geometric Constraint for Multi-Camera Pose Adjustment

1. Problem Statement

Multi-camera systems offer superior robustness and wider fields of view for visual navigation and 3D reconstruction compared to monocular or stereo systems. However, these advantages introduce significant feature redundancy.

• Computational Bottleneck: In traditional Bundle Adjustment (BA), the optimization process jointly refines camera poses and 3D scene points. As the number of cameras and observations increases, the parameter space grows rapidly (due to the inclusion of 3D point coordinates), leading to high computational overhead and memory consumption.
• Redundancy: In multi-camera setups, a single 3D point is often observed by multiple cameras simultaneously or across many frames, creating a massive number of redundant variables that do not necessarily improve accuracy but drastically slow down optimization.
• Core Question: How can one optimize multi-camera poses efficiently without sacrificing accuracy, by eliminating the need to explicitly optimize 3D scene points?

2. Methodology

The authors propose a Pose-Only Adjustment framework that decouples pose optimization from 3D point reconstruction. The methodology consists of four main components:

A. Generalized Camera Model Representation

The multi-camera system is modeled as a single Generalized Camera.

• Each 2D image pixel observation is mapped to a unique 6D vector $(f, v)$, where $f$ is the normalized direction vector and $v$ is the vertex of the observation ray.
• This unified representation handles arbitrary camera configurations (e.g., overlapping fields of view, fisheye, or pinhole cameras) and extrinsic calibrations within a single mathematical framework.

B. Multi-Camera Pose-Only Constraint

The core innovation is the derivation of a geometric constraint that implicitly represents 3D points using only poses and observations.

• Base Observations: For any 3D point $X_W$, two specific observations (a "left-base" and a "right-base") are selected.
• Analytical Elimination: The 3D point $X_W$ is analytically expressed as a function of the base observations and their corresponding poses.
• Constraint Formulation: All other observations of the same point are expressed as functions of the base pair and the poses. This creates a Pose-Only Constraint Set ($S^{(l,r)}_{X_W}$) where the 3D point coordinates are eliminated from the optimization variables.
• Result: The cost function depends solely on the poses ($R, t$), effectively removing 3D points from the parameter space.

C. Base Observation Selection Strategy

Selecting the optimal pair of base observations is critical for accuracy. The authors propose a selection algorithm based on Uncertainty Ellipsoid Roundness:

• Instead of simply choosing the maximum disparity or theta angle, the algorithm calculates the covariance matrix of the reconstructed 3D point for potential observation pairs.
• It selects the pair that yields the most "round" (isotropic) uncertainty ellipsoid, which corresponds to the most stable and accurate geometric intersection.
• This is a two-stage process: first selecting the observation with minimum variance, then iterating to find the partner that maximizes roundness.

D. Statistically Optimal Reconstruction

Once poses are optimized, a Statistically Optimal Triangulation method is used to reconstruct the 3D scene:

• Unlike standard midpoint methods, this approach incorporates the full covariance of the 2D observations.
• It projects the 3D residuals into the tangent space of the unit sphere (using the null space of the direction vector) to handle the singularity of spherical normalization.
• The solution minimizes the Mahalanobis distance, providing a closed-form, statistically optimal solution for the generalized camera model.

3. Key Contributions

1. Pose-Only Geometric Constraint: A novel formulation that analytically eliminates 3D points from the optimization loop for multi-camera systems, reducing the dimensionality of the parameter space.
2. Pose Adjustment Algorithm: A complete algorithm (MCPA/MCPALR) that optimizes poses without 3D variables, significantly improving computational efficiency.
3. Uncertainty-Based Selection: A base observation selection algorithm based on uncertainty ellipsoid roundness, which outperforms traditional disparity-based selection methods in noisy conditions.
4. Statistically Optimal Reconstruction: A closed-form triangulation method for generalized cameras that accounts for observation covariance, ensuring high-quality 3D reconstruction after pose optimization.

4. Experimental Results

The authors evaluated their method on both synthetic and real-world datasets (ETH3D and KITTI Odometry), comparing against standard Bundle Adjustment (MultiCol, BACS) and RigBA.

• Computational Efficiency:
  • The proposed methods (MCPA and MCPALR) significantly outperformed baselines in runtime.
  • On large-scale datasets (e.g., 200 poses, 10,000 points), optimization speed improved by 2.5x, and memory usage was reduced by 19x compared to MultiCol.
  • On the KITTI dataset, MCPA achieved up to 3.82x speedup over MultiCol.
• Pose Accuracy:
  • The proposed methods maintained or improved pose estimation accuracy compared to traditional BA.
  • Under high noise conditions, the pose-only methods demonstrated superior robustness because they are immune to errors introduced by inaccurate 3D point triangulation during the optimization loop.
  • MCPALR (using both left and right base constraints) generally offered the highest accuracy, while MCPA offered the best speed/accuracy trade-off.
• Reconstruction Quality:
  • The proposed statistically optimal triangulation method achieved the lowest reconstruction errors across all noise levels, outperforming standard methods like MidPoint, DLT, and NonLin.

5. Significance

This paper addresses a critical bottleneck in multi-camera visual systems: the computational cost of handling redundant observations.

• Scalability: By removing 3D points from the optimization variables, the method scales much better with the number of cameras and frames, making it highly suitable for real-time applications like autonomous driving and large-scale SLAM.
• Robustness: The decoupling of pose and point optimization prevents error propagation from noisy 3D points to the camera poses, leading to more stable trajectories in challenging environments.
• Generalizability: The use of the Generalized Camera Model ensures the method is applicable to diverse hardware configurations (e.g., multi-fisheye, stereo, or arbitrary rigid arrays) without requiring system-specific adaptations.

In summary, the authors successfully demonstrate that pose-only optimization is not only computationally superior but also capable of achieving state-of-the-art accuracy in multi-camera systems, offering a new paradigm for efficient 3D vision pipelines.