ViBA: Implicit Bundle Adjustment with Geometric and Temporal Consistency for Robust Visual Matching

1. Problem Statement

Existing visual odometry (VO) and visual-inertial odometry (VIO) systems rely heavily on feature detection and matching. Current approaches face three primary limitations:

• Dependency on Supervised Data: Most learning-based methods (e.g., SuperPoint, SuperGlue) are trained on datasets with ground-truth pose and depth annotations. This limits scalability and generalization to unconstrained, real-world video streams.
• Disconnect from Geometric Optimization: Learned features are typically optimized using surrogate objectives (e.g., homography, photometric loss) rather than the true geometric objectives of VO/VIO pipelines (minimizing reprojection error). Consequently, features may be visually discriminative but geometrically inconsistent.
• Inability to Integrate Global Optimization: Traditional Bundle Adjustment (BA) is an iterative, non-differentiable process. Integrating it directly into end-to-end learning frameworks is difficult, preventing learned features from being refined by global geometric constraints.

2. Methodology: The ViBA Framework

The authors propose ViBA, a sustainable learning framework that integrates geometric optimization with feature learning for continuous online training. The system operates within a standard VO pipeline and consists of three main stages:

A. Architecture Overview

The framework includes a Point Extraction Network and a Point Matching Network.

• Pre-training: The matching network is pre-trained on the MegaDepth dataset using a D2-Net-style formulation (shared encoder for detection and description) to establish initial geometric reasoning capabilities.
• Warm-up: A "warm-up" phase initializes the extraction network using the pre-trained matching network to ensure stable convergence before self-supervised training begins.

B. Implicit Differentiable Bundle Adjustment (BA)

The core innovation is an implicitly differentiable geometric residual framework that allows gradients to flow from a global BA module back to the feature networks without unrolling the iterative solver.

1. Initialization: An initial tracking network establishes sparse inter-frame correspondences. These are used to estimate initial camera poses and 3D points via triangulation and depth filtering.
2. Implicit Optimization: A global BA module minimizes the global reprojection error ($E_{reproj}$) to refine camera poses and 3D points.
3. Implicit Differentiation: Instead of unrolling the Levenberg-Marquardt (LM) iterations (which is memory-intensive), ViBA treats the BA solver as an implicit mapping. It computes the gradient of the optimal solution ($X^*$) with respect to network parameters ($\theta$) using the first-order optimality condition:
   $$\nabla_X E(X^*, \theta) = 0 \implies \frac{dX^*}{d\theta} = -H^{-1} \nabla^2_{X\theta} E(X^*, \theta)$$
   This allows the system to backpropagate the geometric residuals directly to the feature extraction and matching networks, optimizing them for true geometric consistency.

C. Multi-Frame Temporal Consistency

To prevent drift and ensure stability over long sequences, ViBA enforces temporal consistency at two levels:

1. Trajectory Consistency: Compares recursive frame-to-frame tracking paths against long-baseline direct predictions to detect and penalize temporal drift.
2. Descriptor Consistency: Enforces coherence in the descriptor space by aligning similarity maps across different propagation paths and applying a unimodality constraint to ensure matches are well-localized.

The total loss function combines the implicit geometric loss (reprojection error) and the temporal consistency losses, enabling self-supervised learning directly from raw video streams without ground-truth annotations.

3. Key Contributions

1. Implicit Differentiable BA: The paper introduces a principled method to integrate global bundle adjustment into end-to-end feature learning via implicit differentiation, allowing features to be optimized directly against geometric reprojection objectives.
2. Spatio-Temporal Consistency: A novel multi-level consistency formulation that jointly optimizes spatial geometry (via BA) and temporal stability (via trajectory and descriptor constraints), ensuring robust long-term tracking.
3. Self-Supervised Online Learning: The framework supports continuous online training from unconstrained video streams, eliminating the need for pre-collected datasets with pose/depth annotations.
4. Robust Generalization: The method demonstrates the ability to adapt to diverse environments and motion patterns without overfitting to specific dataset characteristics.

4. Experimental Results

ViBA was evaluated on the EuRoC and UMA datasets for VIO trajectory estimation, as well as HPatches (homography) and 7-Scenes (indoor pose).

• VIO Performance (EuRoC & UMA):
  • Compared to state-of-the-art methods (SuperPoint+SuperGlue, ALIKED, LightGlue), ViBA reduced the Mean Absolute Translation Error (ATE) by 12–18% and Absolute Rotation Error (ARE) by 5–10%.
  • On average, ViBA reduced translation error by 25–35% compared to SuperGlue and 30–40% compared to LightGlue.
  • Generalization: While end-to-end models like ALIKED degraded significantly when transferred from EuRoC to the unseen UMA dataset, ViBA retained over 90% localization accuracy, demonstrating superior robustness.
• Efficiency:
  • ViBA maintains real-time inference speeds, ranging from 36 to 91 FPS depending on input resolution.
  • It achieves a better accuracy-efficiency trade-off than dense end-to-end methods, particularly benefiting from higher-resolution inputs where it exploits finer local details.
• Homography & Pose Estimation:
  • On HPatches, ViBA achieved top precision and high recall, enabling accurate homography estimation even with non-robust solvers (DLT).
  • On 7-Scenes, it outperformed other learning-based matchers in pose estimation accuracy (AUC).

5. Significance

The significance of ViBA lies in its ability to bridge the gap between deep learning and classical geometric optimization.

• Paradigm Shift: It moves away from training features on static, annotated datasets toward a dynamic, self-supervised approach where features are continuously refined by the geometric consistency of the video stream itself.
• Robustness: By explicitly optimizing for the true system-level objective (minimizing reprojection error) rather than proxy tasks, ViBA produces features that are inherently more robust to challenging conditions like rapid motion, illumination changes, and weak textures.
• Scalability: The implicit differentiation approach allows the integration of complex global optimization (BA) into learning pipelines without prohibitive memory costs, paving the way for more robust and adaptive autonomous navigation systems in real-world scenarios.