FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction

1. Problem Statement

Streaming 3D reconstruction aims to predict geometry and camera motion from an image stream using constant memory and low latency. Recent recurrent architectures (e.g., CUT3R) maintain a compact set of persistent latent tokens to distill scene context. However, these systems face a critical failure mode when the sequence length exceeds the training horizon:

• Drift and Catastrophic Forgetting: Aggressive update rules (e.g., uniform overwrites) trust new frames too much, causing the system to forget useful history and accumulate errors.
• Instability: Conservative updates fail to track new evidence (e.g., scene changes), leading to lag and degraded geometry.
• Heuristic Limitations: Existing training-free methods (e.g., TTT3R) use heuristic gating based on attention statistics. While they improve stability, they lack an explicit representation of uncertainty and do not adapt principledly as confidence accumulates.

The core challenge is designing a state update rule that dynamically balances memory retention (long-term consistency) with adaptation (fast response to new evidence) without retraining the backbone model.

2. Methodology: FILT3R

The authors propose FILT3R, a training-free, plug-and-play latent filtering layer that casts recurrent state updates as stochastic state estimation in token space using an Adaptive Kalman Filter (AKF) framework.

Core Formulation

The system models the recurrent latent state $s_t$ as a belief state and the decoder's candidate output $\tilde{s}_t$ as a noisy measurement. The dynamics are modeled as:

1. Process Model: $s_t = s_{t-1} + w_t$, where $w_t \sim \mathcal{N}(0, Q_t)$ represents the uncertainty in how much the scene latent changes between frames (Process Noise).
2. Measurement Model: $\tilde{s}_t = s_t + v_t$, where $v_t \sim \mathcal{N}(0, R)$ represents the uncertainty in the decoder's prediction (Measurement Noise).

Key Design Choices

• Fixed Measurement Noise ($R = rI$): The authors assume the decoder is a frozen, pretrained network. Therefore, the reliability of its prediction is determined by its architecture, not the current scene dynamics. They fix $r$ as a scalar hyperparameter shared across all tokens. This prevents the instability seen in methods that adaptively estimate both process and measurement noise simultaneously (which can lead to correlated noise spikes).
• Adaptive Process Noise ($Q_t$): The process noise is estimated online from the temporal drift of candidate tokens.
  • Drift Calculation: The system computes the $L_2$ norm of the difference between consecutive candidate states ($\Delta_{t,i} = \|\tilde{s}_{t,i} - \tilde{s}_{t-1,i}\|_2$).
  • EMA Baseline: A running Exponential Moving Average (EMA) of the mean drift ($\hat{\Delta}_t$) is maintained to normalize the current drift.
  • Sigmoid Gate: The normalized drift score is mapped to a process noise value $q_{t,i}$ via a sigmoid function.
    • Stable Regime: Low drift $\rightarrow$ Low $q_{t,i}$ $\rightarrow$ Low Kalman Gain $\rightarrow$ High memory retention.
    • Scene Change: High drift $\rightarrow$ High $q_{t,i}$ $\rightarrow$ High Kalman Gain $\rightarrow$ Aggressive adaptation.

Update Rule

The state update follows the standard Kalman filter equations but operates element-wise (per-token):

1. Variance Prediction: $p^-_t = p_{t-1} + q_t$
2. Kalman Gain: $k_t = \frac{p^-_t}{p^-_t + r}$
3. State Update: $s_t = s_{t-1} + k_t \odot (\tilde{s}_t - s_{t-1})$
4. Variance Update (Joseph Form): $p_t = (1-k_t)^2 \odot p^-_t + r k_t^2$

This recursive uncertainty propagation allows the gain $k_t$ to naturally decay as evidence accumulates in stable scenes (extending the effective memory horizon) and spike only when genuine process noise (scene change) is detected.

3. Key Contributions

1. Reframing as State Estimation: The authors recast streaming 3D reconstruction as a stochastic state estimation problem in latent token space, introducing FILT3R as an adaptive filtering layer.
2. Training-Free & Plug-and-Play: FILT3R requires no retraining of the backbone. It acts as a lightweight, interpretable update rule that generalizes existing policies (uniform overwrite, fixed gating) as special cases.
3. Principled Uncertainty Propagation: Unlike heuristic gates, FILT3R propagates a per-token variance state. This enables the system to "learn" confidence over time, shrinking gains in stable regimes to prevent drift while remaining responsive to transitions.
4. Robust Process Noise Estimation: By estimating process noise solely from temporal drift (while keeping measurement noise fixed), the method avoids the instability of correlated noise estimates during scene transitions.

4. Experimental Results

The authors evaluated FILT3R on video depth estimation, camera pose estimation, and 3D reconstruction across multiple datasets (Sintel, Bonn, KITTI, TUM-RGBD, 7-Scenes, NRGBD).

• Long-Horizon Stability:

  • 3D Reconstruction: On 7-Scenes and NRGBD, FILT3R significantly outperforms CUT3R (uniform overwrite) and TTT3R (heuristic gating) as sequence length increases. At length 1000 (far beyond training horizon), FILT3R maintains coherent geometry while others degrade or run out of memory (Point3R).
  • Pose Estimation: On TUM-RGBD, FILT3R drastically reduces origin-aligned ATE (accumulated drift). At 800 frames, ATE$_{orig}$ drops from 0.214 (TTT3R) and 0.493 (CUT3R) to 0.107 with FILT3R.
  • Depth Estimation: On Bonn sequences, FILT3R achieves the lowest Abs Rel and highest $\delta < 1.25$ metrics, demonstrating superior robustness to outliers and drift.

• Short-Horizon Performance:

  • FILT3R matches or slightly improves upon baselines in short-horizon settings, proving it does not sacrifice immediate accuracy for long-term stability.

• Efficiency:

  • FILT3R maintains a constant memory footprint (similar to CUT3R) and does not require caching attention maps (unlike TTT3R), making it highly efficient for long rollouts.

• Ablation Studies:

  • Removing variance propagation (resetting uncertainty every step) causes severe drift, confirming the necessity of recursive confidence accumulation.
  • Using a fixed process noise (Fixed-Q) fails to adapt to scene changes, leading to higher drift.
  • Making measurement noise adaptive hurts performance, validating the design choice of a fixed $r$.

5. Significance

FILT3R addresses a fundamental limitation in recurrent 3D reconstruction: the inability to balance memory retention with adaptation over long horizons. By introducing a principled, uncertainty-aware filtering mechanism that is training-free and computationally lightweight, the paper enables:

• Continuous, Stable Inference: Systems can now process video streams indefinitely without catastrophic forgetting or scale drift.
• Generalization: The method generalizes well beyond the training horizon, a critical requirement for real-world robotics and AR/VR applications.
• Interpretability: The update rule is mathematically grounded in Kalman filtering, providing clear insights into how the system balances trust between history and new observations.

In summary, FILT3R provides a robust, universal solution for stabilizing streaming 3D reconstruction, paving the way for reliable, long-form online scene understanding.