Technical Summary: NoDrift3R

Problem Statement

Feed-forward 3D Gaussian Splatting (3DGS) has emerged as a paradigm for rapid scene reconstruction, yet pose-free variants face a critical bottleneck when processing long image sequences. The primary issue is cumulative camera pose estimation drift. In existing pose-free frameworks, errors in pose estimation propagate into geometric modeling, distorting the 3D representation, while inaccuracies in scene structure further degrade pose predictions. This feedback loop leads to accumulated drift, severely limiting rendering fidelity and geometric consistency over time.

Furthermore, the paper identifies a training instability inherent to pose-free paradigms. While Structure-from-Motion (SfM) based pseudo-ground-truth poses introduce sensor noise, purely rendering-based supervision often results in optimization instability and local minima due to the entangled optimization of geometry and pose. Existing methods attempting to stabilize this (e.g., via teacher-forcing or curriculum learning) struggle to balance stability with fine-grained precision, particularly when extending to wide temporal ranges or low-overlap viewpoints.

Methodology

The authors propose NoDrift3R, a synergistic pose-free framework designed to explicitly couple geometry and appearance to suppress drift. The methodology consists of three core components:

1. Raymap-Guided Coupling (RGC) Module

Unlike prior approaches that either predict Gaussian representations directly from images (ignoring pose) or condition them on low-dimensional pose parameters (creating an information bottleneck), NoDrift3R introduces a Raymap-Guided Coupling Module.

• Mechanism: The model predicts a dense raymap ($R \in \mathbb{R}^{H \times W \times 6}$) for each view, encoding per-pixel ray directions ($r$) and origins ($o$).
• Geometric Anchoring: Gaussian centers ($p_j$) are explicitly lifted from 2D pixels to 3D space using the predicted raymap and a depth map ($D$): $p_j = o_j + D_j \cdot r_j$.
• Synergistic Optimization: This design creates a bidirectional feedback loop:
  • Appearance $\to$ Geometry: RGB reconstruction loss backpropagates through the rendering process to refine the raymaps and depth, sharpening the geometry.
  • Geometry $\to$ Appearance: The raymap loss constrains ray directions, ensuring the spatial distribution of Gaussians is geometrically consistent, which in turn improves rendering quality.
• Objective: The total loss combines RGB reconstruction (MSE + LPIPS), camera regularization (Huber loss), and raymap consistency (L1 loss), forming a unified objective where geometry and appearance mutually reinforce each other.

2. Dual-Frequency Viewpoint Scheduling

To address training instability across wide temporal ranges, the authors introduce a curriculum-based sampling strategy that balances "easy" (high overlap) and "hard" (low overlap) view pairs.

• Bounded Overlap-Aware Curriculum: The model is gradually exposed to larger view intervals (from high to low overlap) by annealing a target overlap threshold. A sparsity upper-bound ($g_{max}=15$) is applied to prevent overly sparse supervision.
• Stochastic Replay Regularization: To prevent performance degradation on small-interval (high-overlap) samples—a common failure point in linear extension strategies—the scheduler includes a replay mechanism. During the latter half of training, there is a 50% probability of sampling small-interval pairs ($g \in [6, 10]$). This ensures the model maintains local geometric consistency while learning long-range dependencies.

3. Architecture

The framework utilizes a single multi-view transformer backbone (based on DINOv2) to process view tokens. It employs a Dual-DPT head to jointly predict the raymap, depth map, and camera parameters, followed by a Gaussian head to predict attributes (opacity, color, scale, rotation) anchored to the lifted 3D positions.

Key Contributions

1. Identification of the Long-Sequence Bottleneck: The paper establishes that cumulative pose drift is the primary factor restricting reconstruction quality in long sequences and demonstrates that purely rendering-based supervision leads to instability.
2. Dual-Frequency Viewpoint Scheduling: A novel training schedule combining easy-to-hard interval expansion with a replay mechanism for small-interval samples, stabilizing optimization across arbitrary frame intervals.
3. Synergistic "Rendering-to-Geometry Gain": A framework that tightly couples 3DGS rendering supervision with explicit raymap constraints. This bidirectional coupling resolves parameter ambiguity and suppresses drift without complex auxiliary losses, achieving state-of-the-art (SOTA) performance in both rendering fidelity and camera pose accuracy.

Experimental Results

Extensive experiments were conducted on the DL3DV, RealEstate10K (RE10K), and ScanNet++ datasets.

• Novel View Synthesis (NVS): On DL3DV, NoDrift3R consistently outperformed SOTA pose-dependent (MVSplat, DepthSplat) and pose-free (NoPoSplat, AnySplat, YoNoSplat) methods. In the 24-view setting, it improved PSNR by 1.6 dB over the previous best pose-free method (YoNoSplat).
• Pose Estimation: The method achieved superior camera pose estimation accuracy (measured by AUC@5°, 10°, 20°) across all view settings. The improvement was most pronounced in long sequences (12v/24v), verifying effective drift suppression.
• Generalization:
  • Cross-Dataset: Trained on DL3DV, the model achieved SOTA performance on ScanNet++ without fine-tuning.
  • Zero-Shot: When trained on DL3DV and tested on RE10K, the model achieved the best results for 5° and 10° pose error thresholds, demonstrating strong transferability.
• Ablation Studies:
  • Removing the raymap-guided geometry or the raymap loss significantly degraded pose accuracy and rendering quality, confirming the necessity of the bidirectional coupling.
  • Removing the replay mechanism in the scheduler caused a trade-off where long-range performance improved at the cost of short-range consistency; the replay mechanism successfully balanced both.

Significance and Claims

The paper claims that explicitly designed geometry-appearance synergy is the key to scalable and drift-robust pose-free feed-forward 3D reconstruction. By anchoring Gaussian centers to raymap-induced geometry, the framework creates a self-reinforcing training dynamic that stabilizes optimization across long sequences and wide view intervals. The authors assert that this approach resolves the fundamental trade-off between stability and precision in pose-free 3DGS, enabling high-fidelity reconstruction in scenarios where camera poses are unavailable.

The work is presented as a significant step forward in making feed-forward 3D reconstruction practical for long, unposed sequences, moving beyond the limitations of current methods that suffer from accumulated drift and optimization instability.