FVO: Fast Visual Odometry with Transformers

Technical Summary: Fast Visual Odometry with Transformers (FVO)

Problem Statement
Visual odometry (VO) aims to estimate a camera's trajectory from video sequences. While hybrid pipelines combining deep learning with classical optimization (e.g., bundle adjustment) have achieved high accuracy, they suffer from significant limitations. These methods often rely on massive, pre-trained 3D backbones that are frozen and scale-ambiguous, inheriting the inability to estimate absolute scale. Furthermore, the reliance on post-processing steps like bundle adjustment and feature matching creates bottlenecks in inference speed and requires known camera calibration parameters, which are often unavailable in real-world deployments. Pure end-to-end approaches, while faster, have historically lagged behind in accuracy and robustness. The challenge lies in developing a monocular VO system that is fast, scalable, does not require camera intrinsics or test-time optimization, and can accurately estimate metric scale without heavy post-processing.

Methodology
The authors propose Fast Visual Odometry (FVO), a direct relative pose regression framework that replaces traditional post-processing with a transformer-based architecture.

• Architecture: The core of FVO is a transformer model designed for end-to-end structured prediction. It utilizes a frozen, pre-trained image encoder (based on the CroCo architecture within the DUSt3R framework) to extract visual features from input frames. These features are processed by a time-space decoder consisting of $L$ repeating blocks.
  • Time-Space Attention: The decoder employs a factorized attention mechanism. First, temporal attention aggregates information across frames for the same spatial location. Subsequently, spatial attention allows communication among different spatial locations within a frame.
  • Camera Embeddings: Learnable camera embeddings are introduced to aggregate information specifically for pose estimation. Crucially, these embeddings are integrated only within the spatial attention layers, as injecting them into temporal attention was found to degrade performance.
• Relative Pose Regression: The model directly predicts relative camera transformations ($SE(3)$) between consecutive frames. The output includes a rotation matrix, a translation vector, and confidence scalars for both.
  • Rotation Handling: To ensure valid rotations, the raw predicted rotation matrix is projected onto the $SO(3)$ manifold using the special orthogonal Procrustes problem.
  • Uncertainty-Aware Learning: Instead of using per-pixel confidence labels, FVO employs a heteroscedastic uncertainty formulation. The network learns to predict confidence scores ($c_R, c_t$) for rotation and translation in a self-supervised manner. The loss function penalizes overconfident low-accuracy predictions and down-weights uncertain residuals, allowing the model to learn confidence without explicit labels.
• Inference Module: To address the issue of error accumulation in monocular VO, FVO utilizes an overlapping-window scheme. The input video is decomposed into overlapping windows, and the model predicts relative poses for each. A confidence-aware inference module aggregates these overlapping predictions. Predictions are weighted by their learned confidence scores (converted to weights via softmax normalization) and averaged. Rotations are averaged using the weighted Fréchet mean on $SO(3)$, while translations are averaged linearly. This process mitigates the impact of outliers and produces a smooth, globally consistent trajectory without global optimization.

Key Contributions

1. FVO Pipeline: An efficient, end-to-end pipeline for monocular visual odometry that achieves competitive performance across multiple benchmarks while operating nearly $2\times$ faster than the fastest baselines.
2. Transformer-Based Direct Regression: A novel network architecture that directly predicts relative camera poses and confidence estimates from image sequences. The confidence is learned self-supervised, requiring only pose annotations.
3. Confidence-Aware Inference: A robust trajectory recovery module that aggregates overlapping pose predictions using learned confidence weights, effectively mitigating spurious predictions without requiring bundle adjustment or camera calibration.

Experimental Results
The method was evaluated on diverse indoor and outdoor datasets, including ARKitScenes, ScanNet, KITTI, and TUM_RGBD (zero-shot).

• Accuracy: FVO achieves state-of-the-art or competitive performance in Absolute Translation Error (ATE), both aligned and unaligned. Notably, it outperforms large-scale 3D models (like VGGT and CUT3R) and optimization-based methods (like DPVO) in unaligned metrics, demonstrating superior scale estimation capabilities.
• Speed: On commodity hardware (NVIDIA RTX 3090), FVO achieves a runtime almost $2\times$ faster than existing baselines. This efficiency is attributed to the absence of expensive post-processing steps and the streamlined transformer architecture.
• Scaling: Ablation studies confirm that the model scales robustly with both increased training data (e.g., adding ARKitScenes to ScanNet) and increased model capacity (more decoder layers).
• Design Choices: Experiments validate that the specific time-space attention mechanism with camera tokens in spatial layers, the $SO(3)$ projection for rotations, and the heteroscedastic uncertainty loss are critical for performance.

Significance and Claims
The paper claims that FVO represents a significant step forward by demonstrating that streamlined, end-to-end architectures, when paired with sufficient data, can replace complex hybrid pipelines. By eliminating the need for test-time optimization, camera intrinsics, and hand-crafted components, FVO enables real-time, large-scale deployment in scenarios where traditional methods fail or are too slow. The authors emphasize that their approach successfully leverages diverse data to resolve monocular scale ambiguity directly from visual cues, offering a robust path for real-time visual odometry in practical applications like autonomous driving and robotics. However, they modestly note that the model is primarily trained in static environments and may face limitations in highly dynamic settings, suggesting future work in data curation and model scaling.