Technical Summary: Online Neural Space Time Memory for Dynamic Novel View Synthesis

Problem Statement

Online Novel View Synthesis (NVS) from multi-view streaming videos faces a fundamental trade-off between maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions and operating under strict real-time constraints. Existing methods struggle to retain historical context (e.g., a person's back when only frontal views are currently visible) without sacrificing speed.

Standard Test-Time Training (TTT) approaches, which offer expressive memory via gradient-based updates, are computationally prohibitive for real-time streaming. They typically require per-frame gradient updates to adapt to dynamic motion, leading to high latency (e.g., 58ms per frame at 256×256 resolution) and instability over long sequences due to catastrophic drift. Conversely, feed-forward models with full self-attention scale quadratically with sequence length, making them unsuitable for minute-long contexts. Furthermore, naively training memory modules on sparse occlusion signals often leads to underconstrained problems where the network bypasses the memory, relying instead on current incomplete observations.

Methodology: Neural Space-Time Memory (NSTM)

The authors propose NSTM, an online framework that decouples the frequencies of memory updates and view synthesis to achieve amortized real-time performance while retaining minute-long context.

1. Decoupled Memorization and Synthesis

To overcome the computational bottleneck of TTT, NSTM separates the inference process into two frequency-asymmetric steps:

• Periodic Memorization Step: Heavy gradient-based memory updates occur only periodically (e.g., every few frames) to compress novel states into fast weights. This avoids the latency of updating at every timestep.
• Per-Frame Synthesis Step: At every timestep, the model performs a lightweight query against the persistent memory state.
• Cross-View Attention: Because the memory state (updated periodically) and the current observation (dynamic) are inherently misaligned, NSTM employs cross-view attention. This mechanism allows target views to attend directly to current input views, fusing ongoing motion with the historical memory context to resolve temporal deformations.

2. Disentangled Memory Supervision

To prevent the network from over-relying on current inputs and bypassing the memory module, the authors introduce an alternating training regime with an auxiliary Memory Loss:

• Memory Supervision (Even Timesteps): The model is forced to reconstruct a target view using only isolated memory tokens (representing target camera rays) without attending to current input views. This "blind retrieval" forces the network to internalize the scene context entirely within the memory weights.
• Synthesis Supervision (Odd Timesteps): The model operates in standard NVS mode, allowing cross-attention between inputs and targets to capture motion, while querying the previously updated memory.
  This alternating objective ensures the memory module learns to persistently internalize scene geometry rather than relying on sparse occlusion signals in the current frame.

3. Stability Mechanisms

To ensure stability over long horizons (minutes), NSTM incorporates two critical strategies:

• L2 Inner Loss: The authors replace the standard unbounded dot-product inner loss used in prior TTT works with an L2 objective. This prevents magnitude inflation of the fast weights during repeated gradient steps, a common cause of drift.
• Memory Caching: To mitigate RNN-like drift, the model aggregates historical memory checkpoints. It maintains an average of the current active memory and past memory states (every $K$ steps). Crucially, this caching strategy is only effective because the decoupled architecture uses cross-view attention to explicitly handle the spatial deformations between the averaged historical pose and the current observation.

Key Contributions

1. Real-Time, Long-Context Dynamic NVS: NSTM is the first online framework for dynamic NVS from multi-view videos that sustains minute-long persistent memory while operating at amortized real-time speeds (30 FPS synthesis, 1 FPS memorization updates).
2. Decoupled Architecture: The paper structurally decouples periodic, heavy memory updates from continuous, transformer-driven per-frame synthesis, bypassing the quadratic scaling of self-attention and the per-frame cost of standard TTT.
3. Disentangled Supervision: The introduction of an alternating training regime and an auxiliary memory loss explicitly forces the network to internalize persistent scene context, overcoming the underconstrained nature of sparse occlusion signals.
4. Stability via Caching and L2: The combination of memory caching and an L2 update rule enables stable, minute-long view synthesis, preventing the performance collapse seen in prior stateful baselines.

Experimental Results

The framework was evaluated on the MVHumanNet++ dataset, featuring 60,000 multi-view sequences with complex human motion and garments.

• Memory Stress Test: In a forced-occlusion setup where the model observes a subject's back at $T=0$ and only frontal views thereafter, NSTM maintains high-fidelity recall of occluded details (e.g., logos, hairstyles) at $T=60$ frames.
  • Quantitative Performance: NSTM achieves a masked PSNR (mPSNR) of 20.71 dB at $T=60$, significantly outperforming baselines.
  • Baseline Comparison:
    • LVSM (Stateless): Fails to reconstruct occluded regions, maintaining a flat but low performance (~19.34 dB).
    • Token-Mem: Quickly forgets context, dropping to ~18.84 dB.
    • LaCT-NVS (Stateful): Suffers from severe drift due to continuous updates, dropping 5.47 dB from $T=4$ to $T=60$ (25.85 dB to 16.25 dB).
• Ablation Studies:
  • Replacing the dot-product loss with L2 improved PSNR by +5.45 dB.
  • Applying memory caching to a coupled architecture (LaCT-NVS) caused a performance drop, confirming that decoupled training with cross-view attention is essential for effective caching.
  • Removing the auxiliary memory loss ($L_{mem}$) reduced performance, validating the need for explicit internalization supervision.

Significance and Claims

The paper claims that NSTM establishes a new paradigm for online dynamic NVS by resolving the conflict between long-horizon memory retention and real-time inference. By leveraging the computational asymmetry between memory updates and application, and by enforcing strict internalization of scene context, the method achieves state-of-the-art performance in reconstructing occluded regions over minute-long sequences. The authors position this as a foundational step for applications requiring live, free-viewpoint broadcasts and 3D telepresence, where historical context is essential for rendering unseen parts of dynamic subjects.

The authors modestly acknowledge limitations, noting that periodic updates may miss events between steps, that finite memory capacity remains a constraint, and that resolving complex long-range trajectories within compressed memory is an area for future work.