💻 computer science

HorizonStream: Long-Horizon Attention for Streaming 3D Reconstruction

HorizonStream is a novel long-horizon Transformer that achieves state-of-the-art, stable online 3D reconstruction on sequences exceeding 10,000 frames with constant memory and linear time by factorizing geometric propagation into multi-timescale decay and localized spatiotemporal matching to overcome the drift and memory limitations of existing methods.

Original authors: Chong Cheng, Peilin Tao, Nanjie Yao, Guanzhi Ding, Xianda Chen, Yuansen Du, Xiaoyang Guo, Wei Yin, Weiqiang Ren, Qian Zhang, Zhengqing Chen, Hao Wang

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chong Cheng, Peilin Tao, Nanjie Yao, Guanzhi Ding, Xianda Chen, Yuansen Du, Xiaoyang Guo, Wei Yin, Weiqiang Ren, Qian Zhang, Zhengqing Chen, Hao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Bad Memory" of 3D Cameras

Imagine you are walking through a massive, endless city while wearing a pair of smart glasses that try to build a 3D map of the world around you in real-time.

The Challenge: As you walk, the glasses need to remember what the city looked like 10 seconds ago, 10 minutes ago, and even 10 hours ago to know where you are.
The Failure: Existing methods are like people with bad memory habits.
- The "Sliding Window" approach: This is like only remembering the last 5 seconds. If you walk past a landmark and then come back to it 5 minutes later, the glasses say, "I've never seen this before!" and the map breaks.
- The "Full Recall" approach: This is like trying to remember every single detail of your entire life. Eventually, your brain gets so full of old, useless information that it starts to glitch, hallucinate, or crash.
- The "Drift": Over long walks, these systems slowly get confused. They think you are walking in a circle when you aren't, or they think the street is getting wider or narrower. This is called "drift" or "jitter."

The Solution: HorizonStream

The authors created HorizonStream, a new system that acts like a smart librarian for your 3D map. Instead of treating all memories the same, it understands that different types of information have different "lifespans."

1. The "Evidence Influence Kernel" (The Librarian's Rulebook)

The paper argues that not all memories are created equal.

Short-lived evidence: A specific red car passing by or a person walking across the street. These change fast. You only need to remember them for a few seconds.
Long-lived evidence: The shape of a building, the layout of a street, or the overall size of the room. These stay the same for hours or days.

Old systems treated the red car and the building the same way, leading to confusion. HorizonStream uses a special "rulebook" (called an Evidence Influence Kernel) to decide how long to keep each piece of information.

2. The Two-Part Memory System

HorizonStream splits its brain into two specialized teams:

The "Local Team" (Geometric Local Attention):
- Analogy: This team is like a detective looking at the immediate room you are in.
- Job: They match tiny details (like a crack in the wall or a specific window) to figure out exactly where you are right now. They are very picky and ignore anything that looks noisy or unreliable.
- Special Trick: They use a "reliability gate" to ignore bad clues, preventing the system from getting confused by shaky camera movements.
The "Global Team" (Geometric Linear Attention):
- Analogy: This team is like a historian keeping a summary of the whole city.
- Job: They don't remember every single pixel. Instead, they maintain a bounded, shrinking memory.
- The Magic: They use a "decay rate." Imagine a bucket of water where the water slowly leaks out.
  - If the water represents a red car, the hole in the bucket is big, and it leaks out fast (you forget the car quickly).
  - If the water represents a skyscraper, the hole is tiny, and it stays for a long time (you remember the building for hours).
- This ensures the system never gets "full" (no memory crashes) and never forgets the big picture (no drift).

3. The "Metric Readout" (The Ruler)

Sometimes, a camera can tell you where you are, but not how big things are. You might think a door is 1 inch tall or 100 feet tall.

HorizonStream has a special "Ruler Token" that constantly checks the global memory to make sure the size of the world stays consistent. It prevents the map from accidentally shrinking or growing as you walk.

Why It's a Big Deal

The paper claims that HorizonStream is a breakthrough because:

It's Efficient: It uses a constant amount of memory, no matter if you walk for 100 steps or 10,000 steps. It doesn't get slower or heavier.
It's Stable: It was trained on very short clips (only 48 frames long), yet it works perfectly on sequences longer than 10,000 frames (over 2 hours of video).
It Fixes the "Drift": While other systems start to wobble and lose their way after a few minutes, HorizonStream keeps a straight, stable line for hours.

Summary Analogy

Imagine trying to draw a map of a city while walking.

Old methods either forget the city after a few blocks or get so overwhelmed by remembering every leaf on every tree that they stop drawing entirely.
HorizonStream is like a master cartographer who knows: "I only need to remember the leaf for a second, but I must remember the mountain forever." By filtering what to keep and what to let go, it can draw a perfect map of an entire continent without ever running out of paper or getting tired.

The result is a system that can navigate endless streams of video, building a stable, accurate 3D world that doesn't collapse, jitter, or get lost.

Technical Summary: HorizonStream

Problem Statement

Online 3D reconstruction from streaming video requires estimating camera pose and scene geometry under strict causal and bounded-memory constraints. While classical methods rely on iterative optimization with limited throughput, and recent offline feed-forward methods achieve high accuracy by accessing future frames, strictly causal streaming methods struggle with long sequences.

Existing streaming architectures often suffer from drift, jitter, or collapse as sequence length increases. The authors identify the root cause as a fundamental mismatch between the nature of streaming geometry and current architectural designs:

Temporal Heterogeneity: Streaming geometry contains evidence with vastly different lifetimes. Local 2D-3D correspondences are short-lived and quickly become invalid, whereas global scale and scene structures are persistent and must remain reliable over long horizons.
Pathological Influence Patterns: Current methods impose uniform propagation rules on all evidence.
- Sliding windows enforce hard cutoffs, discarding useful past evidence prematurely.
- Ungated recurrence leads to heavy-tailed kernels with unbounded error accumulation.
- Causal softmax attention degenerates into "attention sinks," focusing on irrelevant early tokens, while KV cache refresh mechanisms create discontinuities.
- These pathological kernels cause cache saturation and severe geometric drift as sequences grow.

Methodology: HorizonStream

The authors propose HorizonStream, a long-horizon Transformer that explicitly factorizes the geometric evidence influence kernel to address the temporal heterogeneity of streaming geometry. The framework processes the most recent $W$ frames causally while maintaining an $O(1)$ geometric state for cross-window structure and scale.

1. Geometric Evidence Influence Kernel

The reconstruction process is formalized as aggregating diverse geometric evidence via a spatio-temporal kernel $K(t, i)$ , factorized as:
$K(t, i) = K_{spatial}(t, i) \cdot K_{time}(t, i)$
This factorization maps specific computational components to the three core requirements of streaming reconstruction: selecting reliable local correspondences, ensuring bounded multi-timescale propagation, and preserving metric scale/pose.

2. Geometric Linear Attention (Long-Range Temporal Factor)

To model $K_{time}$ , the authors introduce Geometric Linear Attention, which maintains a bounded, recurrent geometric state $S_t \in \mathbb{R}^{d \times d}$ .

Discounted State Estimation: The state is updated via a discounted objective where the influence of past evidence decays exponentially. This prevents unbounded accumulation of stale information.
Channel-Wise Retention: Instead of a scalar retention rate, the model learns a channel-wise retention vector $\gamma_t \in (0, 1)^d$ $γ_{t} \in (0, 1)^{d}$ .
- Low- $\gamma$ channels rapidly revise transient evidence (e.g., local correspondences).
- High- $\gamma$ channels preserve long-lived evidence (e.g., scene structure, metric scale).
- This creates a continuous spectrum of effective memory lifetimes ( $\tau = -1/\log \bar{\gamma}$ ), enabling stable propagation across windows without periodic resets.

3. Geometric Local Attention (Short-Range Spatial Factor)

To model $K_{spatial}$ , the authors employ Geometric Local Attention within the local window ( $W$ frames).

Head-wise Gating: Each attention head is assigned a reliability gate to filter noisy correspondences and suppress attention sinks.
Spatiotemporal RoPE: Rotary Position Embeddings are extended to three axes (time, height, width) to encode relative 3D space-time layout, ensuring attention depends on relative geometric offsets.

4. Metric Readout Tokens (MRT)

To satisfy metric invariance constraints, Metric Readout Tokens are introduced. These tokens participate in the Geometric Linear Attention and read metric scale directly from the high-retention channels of the recurrent state.

A scale head predicts a global scale factor $\hat{s}$ to rescale translation and depth.
Relative pose fusion over pose tokens in the local window estimates a consensus pose, avoiding error accumulation from sequential keyframe chaining.

5. Architecture and Training

Backbone: A ViT-L initialized from VGGT and DINOv2, alternating frame blocks and global blocks. Geometric Linear Attention layers are interleaved at specific depths (4, 11, 17, 23).
Training: The model is trained on 48-frame clips using a causal window. It is supervised with pose, depth, and scale losses.
Optional Loop Closure: A module retrieves revisited frame pairs to estimate local geometric corrections, optimizing the global trajectory via pose graph optimization.

Key Contributions

Formalization of Streaming Reconstruction: The paper unifies long-sequence failures (drift, jitter, collapse) as pathological kernel shapes (hard cutoffs, discontinuities, attention sinks) and proposes the geometric evidence influence kernel as a unifying framework.
Constrained Kernel-Decomposition Architecture (HorizonStream):
- Geometric Linear Attention: Provides bounded, multi-timescale propagation across windows via learned channel-wise retention.
- Geometric Local Attention: Enables content-aware 3D matching within windows using head-wise gating and Spatiotemporal RoPE.
- Metric Readout Tokens: Preserves metric scale and rigid pose by reading from high-retention subspaces.
Generalization and Efficiency: Trained only on 48-frame clips, HorizonStream generalizes stably to sequences exceeding 10,000 frames with constant memory and linear time complexity, outperforming existing streaming methods.

Experimental Results

Extensive experiments were conducted on multiple datasets (KITTI, VBR, Oxford Spires, ScanNet++, Waymo, etc.), evaluating both camera trajectory estimation (ATE) and dense reconstruction (CD, F1).

Long-Sequence Stability: On the VBR dataset (sequences up to 18,846 frames), HorizonStream maintains stable trajectories with an average ATE of 25.30m (18.84m with loop closure), significantly outperforming other online methods (e.g., Lingbot-map at 27.53m, LongStream at 77.93m) which suffer from severe jitter or collapse.
KITTI Performance: On KITTI sequences (up to 4,661 frames), HorizonStream achieves an average ATE of 19.75m (16.44m with loop closure), approaching or surpassing offline methods and optimization-based SLAM systems.
Dense Reconstruction: HorizonStream achieves state-of-the-art online reconstruction quality (lowest CD and highest F1) on ETH3D, Oxford Spires, 7Scenes, and TUM RGB-D benchmarks.
Memory and Time: Unlike competing methods that exhibit super-linear memory growth or require periodic cache resets, HorizonStream maintains nearly constant peak GPU memory and scales smoothly to 10K frames.

Significance

The paper claims that HorizonStream resolves the fundamental mismatch in current streaming 3D reconstruction architectures by explicitly modeling the temporal heterogeneity of geometric evidence. By learning a bounded, multi-timescale retention spectrum rather than relying on fixed windows or unbounded recurrence, the method achieves stable, metric-consistent reconstruction over ultra-long sequences without the need for periodic state resets or access to future frames. This enables robust online 3D perception for robotics and autonomous driving in long-horizon scenarios.