Technical Summary: Stabilizing Streaming Video Geometry via Dynamic Feature Normalization

1. Problem Statement

While modern Monocular Geometry Estimation (MGE) and Monocular Depth Estimation (MDE) foundation models (e.g., MoGe, Depth Anything) achieve high accuracy on single images, they exhibit severe temporal inconsistency when applied to continuous video streams. The primary manifestation of this instability is scale-shift drifting, where the predicted global depth scale and offset fluctuate frame-to-frame.

This drift causes significant artifacts in 3D reconstruction, including:

• Non-rigid warping: The reconstructed scene distorts over time.
• Layering breaks: Disjointed point clouds where surfaces appear to separate.
• Positional jitter: Unstable object positions in the global coordinate system.

Existing solutions attempt to fix this via temporal attention or recurrent memory modules but often require full-network fine-tuning on large annotated video datasets. This approach is computationally expensive, data-intensive, and frequently degrades the zero-shot generalization and per-frame accuracy of the pre-trained backbone.

2. Empirical Investigation

The authors conducted an empirical study to identify the root cause of this instability using the state-of-the-art MoGe model.

• Observation: When individual frames are aligned to the ground truth using independent affine transformations (scale and shift), the geometric accuracy is extremely high ($\delta < 1.25$ reaches 99.8%). However, when a single consistent scale/shift is applied to the entire sequence, accuracy drops significantly (to 62.5%), resulting in severe warping.
• Conclusion: The underlying geometric understanding of the backbone is robust; the instability is not due to poor per-frame geometry but rather frame-to-frame fluctuations in the predicted global scale and shift.
• Root Cause Analysis: The study revealed a tight coupling between the mean and variance of latent features extracted by the encoder and the predicted scale/shift. Fluctuations in these latent feature statistics across a video stream directly drive the observed scale-shift drift.

3. Methodology: Dynamic Feature Normalization (DyFN)

To address this, the paper proposes Dynamic Feature Normalization (DyFN), a lightweight, causal, recurrent module designed to stabilize feature statistics over time without retraining the entire network.

Architecture

• Frozen Backbone: The pre-trained MGE encoder and decoder remain frozen to preserve single-image accuracy and zero-shot generalization.
• Recurrent Module: DyFN is inserted between the encoder and decoder. It utilizes a Convolutional GRU (ConvGRU) to maintain a hidden state ($h_t$) that aggregates historical context from previous frames.
• Normalization Process:
  1. Standardization: The incoming latent feature $F_t$ is normalized using its channel-wise mean ($\mu_t$) and standard deviation ($\sigma_t$).
  2. Prediction: The ConvGRU hidden state $h_t$ is projected via two lightweight $1\times1$ convolutional heads to predict new, temporally consistent statistics: a mean $\hat{\mu}_t$ and a standard deviation $\hat{\sigma}_t$.
  3. Modulation: The normalized features are re-scaled and shifted using the predicted statistics: $F^{consistent}_t = \hat{\sigma}_t \cdot F^{norm}_t + \hat{\mu}_t$.
  4. Decoding: The stabilized features are passed to the frozen decoder to generate the final depth map.

Training Strategy

• Parameter Efficiency: Only the DyFN module is fine-tuned, constituting merely 2% of the total parameters (approx. 5M parameters vs. billions in the backbone).
• Loss Functions:
  • Base Loss ($L_{MoGe}$): Preserves the original single-frame geometric fidelity.
  • Global Alignment Loss ($L_{align}$): Enforces a single, consistent scale and shift across the entire sequence by penalizing deviations from a global optimal affine transformation.
  • Temporal Loss ($L_{temp}$): Minimizes the discrepancy between predicted and ground-truth inter-frame deltas over multiple window sizes to prevent long-horizon drift.

4. Key Contributions

1. Root Cause Identification: The paper establishes that temporal instability in foundation MGE models stems from fluctuations in latent feature statistics (mean and variance), which directly dictate scale and shift predictions.
2. DyFN Module: A novel, lightweight stabilization module that dynamically modulates these statistics using a recurrent mechanism, ensuring consistent geometry without compromising the backbone's generalization.
3. Efficient Adaptation: The method demonstrates that freezing the backbone and fine-tuning only the DyFN module achieves state-of-the-art temporal stability while preserving per-frame accuracy, outperforming methods that require full-network retraining.

5. Experimental Results

The method was evaluated on four benchmarks: Sintel, ScanNet, KITTI, and Bonn.

• Temporal Stability (Video Depth Evaluation):

  • DyFN achieves state-of-the-art performance across all datasets.
  • On ScanNet, it improves $\delta < 1.25$ from 84.7% (MoGe v1) to 96.6%, a 11.9% absolute improvement.
  • It outperforms strong video-based baselines like DepthCrafter and Video Depth Anything (VDA), as well as streaming methods like FlashDepth.
  • It significantly reduces non-rigid warping and positional jitter in qualitative 3D reconstructions.

• Single-Frame Accuracy:

  • Unlike other fine-tuning approaches that often degrade per-frame performance, DyFN perfectly inherits the single-frame accuracy of the frozen backbone (MoGe v1).
  • Results on single-frame metrics (Table 2) are identical to the base MoGe v1 model, avoiding the accuracy trade-off seen in methods like FlashDepth.

• Long-Sequence Robustness:

  • In experiments with sequences up to 500 frames, DyFN demonstrates minimal error accumulation compared to baselines, maintaining superior scale consistency over time.

• Generalization:

  • The approach was successfully adapted to the DepthAnythingV2 (DAv2) backbone, showing significant improvements in temporal stability, confirming the method's architecture-agnostic nature.

6. Significance and Claims

The paper claims that Dynamic Feature Normalization offers a simple, efficient, and highly effective path for adapting static foundation models to continuous video streams. By directly regulating the latent feature statistics that cause scale-shift drift, the method achieves:

• State-of-the-art temporal stability without the computational cost of training large video models.
• Preservation of zero-shot generalization and single-image accuracy by keeping the backbone frozen.
• Superior performance over both offline video methods (which suffer from high latency and memory redundancy) and other streaming methods (which often fail to maintain long-term consistency).

The authors conclude that their work bridges the gap between high-accuracy single-image geometry estimation and the requirements of real-world streaming applications (e.g., autonomous driving, embodied AI) by solving the specific problem of scale-shift inconsistency through feature-level normalization.