Technical Summary: GHOST (Geometry-Hierarchical Online Streaming Token Eviction)

Problem Statement

Streaming 3D reconstruction from long monocular video sequences faces a critical memory bottleneck. Transformer-based models, such as VGGT, achieve state-of-the-art (SOTA) results by attending jointly over all input frames; however, this joint attention scales quadratically with sequence length, rendering it infeasible for long videos. Streaming approaches like StreamVGGT and InfiniteVGGT attempt to mitigate this by maintaining a growing Key-Value (KV) cache and evicting tokens when a fixed budget is exceeded.

Existing eviction strategies, particularly those relying on key-similarity heuristics (e.g., discarding tokens with keys least similar to the current query), suffer from a fundamental limitation: attention scores from the current query frame do not faithfully reflect the long-term geometric value of historical tokens. As demonstrated in the paper, key-similarity scores show negligible correlation with camera pose changes ($\rho = -0.07$) and only moderate correlation with depth gradient variance ($\rho = +0.31$). Consequently, these methods fail to preserve tokens that are geometrically distinctive but visually similar to the current frame, leading to reconstruction quality degradation in long sequences.

Methodology

The authors propose GHOST (Geometry-Hierarchical Online Streaming Token Eviction), a training-free framework designed to manage KV caches for streaming 3D reconstruction. GHOST leverages the model's own 3D geometry outputs (depth, point maps, and camera poses) to determine token importance, rather than relying solely on attention patterns.

1. Hierarchical Dual-Level Importance Scoring

GHOST decomposes token importance into two mutually reinforcing levels, updated incrementally online without extra forward passes:

• Frame-Level Importance ($s_{frame}$): Determines the geometric value of an entire viewpoint. It integrates three signals:
  • Camera Pose Change: Penalizes translational proximity and rotational similarity to ensure frames with significant viewpoint shifts are retained.
  • Depth Gradient Variance: Prioritizes frames with rich structural details (edges, boundaries) over uniform surfaces.
  • Temporal Recency: Assigns higher weight to recent frames to maintain contextual overlap, while still preserving older frames with unique viewpoints.
• Token-Level Importance ($s_{token}$): Identifies specific informative patches within a frame. It aggregates:
  • Visual Saliency: Measured via the spatial gradient magnitude of feature maps to anchor geometric correspondence.
  • Depth Confidence: The model's epistemic certainty regarding depth estimates.
  • Point Confidence: The model's certainty regarding 3D point map estimates.

The final per-patch score $\phi(t, p)$ is a weighted sum of the normalized frame-level and token-level scores.

2. Special-Token Privilege Mechanism

Standard importance scoring may rank special tokens (camera tokens $c_t$ and register tokens $\{r^i_t\}$) lower than informative patch tokens due to their lack of visual saliency. GHOST introduces a deterministic boost for these tokens. By adding a fixed offset ($\Delta_{boost}$) and a rank-based tie-breaker, the mechanism ensures that tokens encoding global scene geometry and structural priors are never evicted, preventing the corruption of pose estimation and scene globalisation.

3. Cosine-Similarity-Guided Layer-Wise Budget Allocation

Recognizing that transformer layers vary in their "transformation strength," GHOST avoids uniform budget distribution.

• Offline Profiling: The system computes the mean cosine similarity ($\bar{\rho}_\ell$) between the input and output of each layer across representative sequences.
• Allocation: Layers with high similarity (near-identity transformations) are allocated smaller budgets, while layers with low similarity (strong transformations) receive larger budgets. This concentrates the token budget where it is most needed for representation preservation.

Key Contributions

1. Geometry-Aware Scoring: A novel, training-free scoring scheme that utilizes the model's own 3D outputs (depth, points, pose) to identify geometrically valuable tokens, addressing the disconnect between attention scores and 3D structure.
2. Special-Token Protection: A deterministic mechanism that guarantees the retention of camera and register tokens, safeguarding global pose and structure encodings from eviction.
3. Adaptive Layer Budgeting: An offline profiling strategy that allocates KV cache budgets based on layer-wise transformation strength, optimizing the trade-off between memory usage and reconstruction fidelity.
4. Comprehensive Evaluation: Extensive experiments demonstrating that GHOST outperforms SOTA baselines across multiple benchmarks and input lengths while significantly reducing memory footprint.

Experimental Results

The authors evaluated GHOST on four benchmarks: Bonn, 7-Scenes, NRGBD, and Long3D, across various input lengths (up to 9,545 frames).

• Performance: GHOST consistently achieves the best performance across nearly all metrics (Accuracy, Completeness, Normal Consistency, and Depth Abs Rel) compared to baselines like VGGT, StreamVGGT, CUT3R, Point3R, TTT3R, and InfiniteVGGT.
  • On 7-Scenes (300 frames), GHOST reduced mean accuracy error by 42.5% compared to InfiniteVGGT.
  • On Long3D (up to 9,545 frames), GHOST showed significant gains, with accuracy improvements scaling with sequence length (e.g., 33.5% improvement on the Library scene).
• Efficiency:
  • Memory Reduction: GHOST cuts the KV cache size by nearly half (e.g., 3.51 GB vs. 6.94 GB at a 600k budget) while maintaining competitive accuracy.
  • Inference Speed: The method delivers up to 1.75× faster inference compared to state-of-the-art methods.
• Ablation Studies:
  • Removing the special-token boost degraded accuracy and normal consistency, confirming the critical nature of protecting global structural tokens.
  • Cosine-similarity-guided budget allocation outperformed uniform allocation and other profiling strategies (Fisher information, gradient magnitude).
  • The system demonstrated robustness to perturbations in importance weights.

Significance

The paper claims that GHOST addresses a fundamental limitation in current streaming 3D reconstruction: the inability of attention-based heuristics to distinguish between historically redundant tokens and those that are geometrically valuable. By shifting the eviction criterion from attention patterns to 3D geometry signals, GHOST provides a more principled approach to token management. The framework enables high-quality, long-sequence 3D reconstruction with significantly reduced memory overhead and improved inference speed, making it a viable solution for real-world applications requiring continuous, unbounded video processing. The authors emphasize that this direction—exploiting model outputs for KV cache compression in 3D tasks—has not been explored in prior work.