LongStream: Long-Sequence Streaming Autoregressive Visual Geometry

1. Problem Statement

Long-sequence streaming 3D reconstruction is a critical challenge for applications like autonomous driving and embodied robotics, which require real-time, metric-scale scene understanding over thousands of frames. Existing approaches suffer from two main failure modes when processing long sequences:

• Catastrophic Extrapolation Failure: Current streaming autoregressive models (e.g., Stream3R, StreamVGGT) often "collapse" within tens of meters. They anchor poses to the first frame, leading to attention decay, scale drift, and extrapolation errors as the sequence length increases.
• Train-Test Mismatch: Existing models are trained on short sequences but tested on long ones ("train-short, test-long" bias). They rely on a fixed first-frame coordinate system, forcing the model to learn a position-fixed mapping that becomes increasingly difficult to extrapolate as the frame index grows.
• Memory and Attention Issues: Streaming Transformers often suffer from attention sinks (over-reliance on the first token) and KV-cache contamination (saturation of long-term memory), causing geometric instability and drift.

2. Methodology: LongStream

The authors propose LongStream, a gauge-decoupled streaming autoregressive framework designed to maintain metric accuracy and geometric stability over kilometer-scale sequences at 18 FPS. The methodology consists of three core pillars:

A. Gauge-Decoupled Formulation

The framework theoretically decouples the gauge freedoms of global SE(3) coordinates and metric scale (Sim(3)).

• Keyframe-Relative Poses (SE(3) Decoupling): Instead of regressing absolute poses anchored to the first frame, LongStream predicts keyframe-relative poses ($T_{i \leftarrow k}$). This transforms the ill-posed long-range extrapolation problem into a constant-difficulty local estimation task. The prediction is invariant to the global coordinate choice, eliminating the "fixed-anchor" bias.
• Orthogonal Scale Learning (Sim(3) Decoupling): To prevent scale drift, the model separates geometry learning from scale estimation.
  • A Geometry Branch optimizes 3D shape in a scale-invariant space (using normalized pointmaps).
  • A dedicated Scale Head independently predicts the global metric scale factor ($s$).
  • This ensures that the geometry backbone learns shape without being entangled with scale, while the scale head learns the metric factor separately.

B. Cache-Consistent Training (CCT)

The authors identify that attention sinks (biased dependence on the first frame) and KV-cache saturation are primary causes of temporal degradation.

• CCT Scheme: The training process explicitly mimics the inference pipeline. It passes and trims the KV cache between chunks during training, ensuring that the model sees the same "sliding window" context during training as it does during inference.
• Effect: This forces the model to operate in a pure sliding window without a persistent anchor, suppressing the attention sink and reducing the gap between training and inference.

C. Periodic Cache Refresh

For ultra-long sequences, even with CCT, accumulated KV caches can lead to memory saturation and drift.

• Mechanism: The system performs a periodic cache refresh (resetting the sink frame and KV cache every $N$ keyframes).
• Benefit: This acts similarly to state marginalization in SLAM, clearing degraded features and stale context while maintaining geometric continuity because the model operates in a keyframe-relative coordinate system.

3. Key Contributions

1. Gauge-Decoupled Architecture: A novel framework that predicts keyframe-relative poses and employs orthogonal scale decoupling, systematically eliminating first-frame anchor dependence and mitigating long-sequence extrapolation failures.
2. Cache-Consistent Training (CCT): A training strategy that aligns training and inference contexts by managing KV-cache visibility, effectively suppressing attention sinks and temporal drift.
3. State-of-the-Art Performance: LongStream achieves stable, metric-scale reconstruction over kilometer-scale sequences (e.g., 2.45 km) at 18 FPS, outperforming existing streaming models and offline baselines that fail or run out of memory on such lengths.

4. Experimental Results

The model was evaluated on diverse indoor (TUM-RGBD, 7Scenes, Oxford Spires) and outdoor (KITTI, vKITTI, Waymo) datasets.

• Pose Estimation (ATE):
  • On KITTI, LongStream achieved an Average ATE of 51.90, significantly outperforming baselines like Stream3R (227.77) and StreamVGGT (226.15).
  • On vKITTI, it achieved an average ATE of 1.610, compared to 83.916 for StreamVGGT.
  • It demonstrated robustness on long sequences (up to 561m) where other methods exhibited non-linear error growth or complete tracking failure.
• 3D Reconstruction:
  • On 7Scenes and TUM, LongStream achieved competitive Chamfer Distance (CD) and F1 scores, rivaling offline methods while operating in a strictly online, future-invisible setting.
• Scale Accuracy:
  • The model recovered metric scale with a ratio of 0.9905 relative to ground truth on vKITTI, demonstrating effective scale decoupling.
• Efficiency:
  • Maintains stable memory usage and latency (18 FPS) on a single GPU, whereas baselines like VGGT-SLAM suffer from Out-Of-Memory (OOM) errors on long sequences.

5. Significance

LongStream represents a significant breakthrough in online 3D vision. By addressing the fundamental "gauge-coupled" design flaws in previous streaming models, it enables metric-scale, kilometer-long 3D reconstruction in real-time.

• Practical Impact: It solves the "catastrophic collapse" problem, making autoregressive visual geometry viable for real-world applications like autonomous driving and large-scale AR/VR where long-term stability is required.
• Theoretical Insight: It provides a principled approach to handling gauge invariance and attention mechanisms in long-sequence Transformers, offering a blueprint for future streaming geometry models.

In summary, LongStream shifts the paradigm from "anchored absolute prediction" to "decoupled relative prediction," enabling stable, real-time, metric-accurate 3D reconstruction over sequences that were previously impossible for streaming models to handle.