Technical Summary: SFVO – Decoupled Confidence-Guided Stereo-Flow Visual Odometry with Bidirectional PnP

Problem Statement

While deep learning-based Visual Odometry (VO) has advanced significantly, most existing methods rely on monocular inputs, which inherently suffer from scale ambiguity. Although stereo vision naturally provides metric depth, learning-based stereo VO remains underexplored due to high computational costs and the complexity of jointly modeling stereo and temporal relationships. Furthermore, while recent advances in stereo matching and optical flow have produced accurate dense correspondences, their complementary geometric information is not fully exploited for VO. Existing learning-based approaches often attempt to learn pose directly from images or rely on complex end-to-end designs that struggle to balance the distinct geometric contributions of different correspondences (e.g., near vs. far points) to rotation and translation.

Methodology

The authors propose SFVO, a correspondence-driven stereo VO framework that leverages pretrained stereo matching and optical flow models to establish geometric constraints directly, rather than learning pose regression from scratch.

1. System Overview

SFVO takes two consecutive rectified stereo frames ($t$ and $t+1$) as input. It utilizes a pretrained UniMatch model to simultaneously estimate:

• Stereo Disparity: Converted into metric depth ($Z$) using the known baseline and focal length.
• Optical Flow: Estimating forward ($t \to t+1$) and backward ($t+1 \to t$) temporal correspondences.

These outputs are combined to form bidirectional 3D–2D correspondences. Instead of learning pose directly, SFVO maps these correspondences into geometric constraints and predicts their reliability.

2. Decoupled Confidence Prediction

A core innovation is the rotation–translation decoupled confidence map. The authors observe that:

• Nearby points provide strong constraints for translation (translational motion induces larger image motion for close objects).
• Distant points provide consistent constraints for rotation (rotational motion is depth-independent).
• Stereo depth uncertainty grows quadratically with distance, making distant points unreliable for translation but still useful for rotation.

Consequently, SFVO predicts separate confidence weights ($W_r$ for rotation, $W_t$ for translation) for both forward and backward temporal directions. This allows the system to retain distant correspondences for rotational estimation while prioritizing nearby points for translational estimation.

3. Differentiable Bidirectional PnP Solver

The framework integrates these decoupled weights into a differentiable bidirectional Perspective-n-Point (PnP) solver. The solver alternates between updating rotation and translation:

• Rotation Estimation: Formulated as a weighted Wahba's problem, aligning unit bearing vectors from the 3D points to the 2D observations using rotation-specific confidence weights.
• Translation Estimation: Formulated by minimizing weighted bearing-perpendicular residuals, using translation-specific confidence weights.
• The solver is implemented using standard batched linear algebra operations in PyTorch, allowing for end-to-end gradient backpropagation.

4. Training Strategy

The stereo and optical flow networks are initialized with pretrained UniMatch weights. Since VO datasets lack ground-truth flow and disparity, correspondence estimation is supervised via image reconstruction loss (SSIM between warped and reference images). Pose estimation is supervised by ground-truth transformations using $\ell_1$ loss for translation and angular error for rotation. A regularization loss prevents confidence maps from collapsing to zero.

Key Contributions

1. SFVO Framework: A deep learning stereo VO framework built upon pretrained stereo matching and optical flow models, enabling efficient metric pose estimation with reduced training effort by avoiding low-level correspondence learning from scratch.
2. Bidirectional Correspondence-to-Geometry Formulation: A novel approach converting disparity and optical flow into forward and backward 3D–2D constraints, integrated with a rotation–translation decoupled confidence weighting scheme.
3. Differentiable Bidirectional PnP: The integration of decoupled confidence maps into a differentiable solver that alternates between rotation and translation updates, improving robustness against outliers and depth uncertainty.
4. Generalization: Demonstrated strong performance and generalization across diverse datasets (outdoor KITTI, indoor EuRoC, and unseen UGV sequences) without retraining.

Experimental Results

The authors evaluated SFVO on the KITTI Odometry (outdoor) and EuRoC MAV (indoor) datasets, comparing it against monocular learning-based methods (DPVO, BotVIO) and a conventional stereo SLAM system (ORB-SLAM3).

• EuRoC MAV: SFVO achieved the best result in seven of nine metrics (specifically, 3 ATE and 4 RPE metrics), consistently obtaining the lowest Relative Pose Error (RPE) for both translation and rotation across all three sequences. It outperformed monocular baselines significantly and remained competitive with ORB-SLAM3 despite lacking backend optimization (loop closure/keyframe management).
• KITTI: SFVO achieved the lowest translational RPE on sequences 09 and 10 and the second-best rotational RPE and Absolute Trajectory Error (ATE), substantially outperforming monocular baselines in global trajectory accuracy.
• Cross-Dataset Generalization: Tested on an unseen UGV campus sequence (Fusion Portable V2 dataset) with different camera hardware and motion patterns. SFVO reduced ATE by approximately 60% compared to the best baseline, demonstrating robustness to new platforms and environments.
• Ablation Studies:
  • Decoupled Confidence: Removing the decoupling (using a shared confidence map) led to higher errors and required significantly more training epochs (40+ vs. 5) to converge. Qualitative analysis showed coupled confidence tended to suppress distant regions, losing valuable rotational constraints.
  • Solver Efficiency: The PnP solver converges stably within ~12 iterations (from identity initialization) and ~7 iterations with warm-starting, requiring only ~1.1 ms per iteration on an RTX 4060 Ti.

Significance and Claims

The paper claims that SFVO offers a simpler and more efficient perspective on stereo VO. By leveraging existing, high-quality pretrained correspondence models, the framework shifts the learning focus to VO-specific components: confidence estimation and geometric pose refinement.

The authors emphasize that their decoupled confidence design better aligns with the physical characteristics of visual correspondences, acknowledging that different points contribute differently to rotation and translation. This design allows the system to handle the inherent trade-offs in stereo depth estimation without complex engineering. The results suggest that SFVO achieves robust and accurate pose estimation with strong generalization capabilities, making it a viable alternative to both monocular learning-based methods and conventional feature-based stereo SLAM systems, particularly in scenarios requiring metric scale without inertial sensors.