💻 computer science

RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstruction

RayMap3R is a training-free, streaming framework for dynamic 3D reconstruction that leverages the static-scene bias of RayMap predictions to identify and suppress moving objects, thereby achieving state-of-the-art performance in real-time geometry and pose estimation.

Original authors: Feiran Wang, Zezhou Shang, Gaowen Liu, Yan Yan

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Feiran Wang, Zezhou Shang, Gaowen Liu, Yan Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a busy city street using only a video camera. You want the model to be perfect, but there's a problem: people are walking by, cars are driving past, and leaves are blowing in the wind.

Most computer programs trying to do this get confused. They think the moving people are part of the building. They try to "glue" the moving people onto the walls. As the camera moves, the program gets dizzy, the building starts to wobble, and the whole 3D world falls apart. This is called camera drift.

Enter RayMap3R, a new method that acts like a smart, invisible filter to solve this problem. Here is how it works, using simple analogies:

1. The "Ghost Vision" Trick (The Core Idea)

Imagine you are looking at a painting of a park.

Normal Vision: You see the trees, the grass, and a dog running across the path.
RayMap Vision (The Trick): Now, imagine you put on special "ghost glasses." These glasses only show you the geometry of the scene based on where the camera is pointing, but they ignore the "color" and "texture" of moving things.

The researchers noticed something fascinating: When the AI uses these "ghost glasses" (which they call RayMap), it naturally ignores the moving dog and only "sees" the static trees and buildings. It has a built-in bias toward things that don't move.

2. The Two-Brain System (Dual-Branch Inference)

RayMap3R gives the AI two "brains" that work at the same time:

Brain A (The Realist): Looks at the full video. It sees the dog, the cars, and the trees. It tries to build the 3D world based on everything it sees.
Brain B (The Skeptic): Uses the "ghost glasses" (RayMap only). It ignores the moving dog and only sees the trees and buildings.

The Magic: The system compares what Brain A sees with what Brain B sees.

If both brains agree on a spot (e.g., a brick wall), the system says, "Great, this is real. Let's save this to our memory."
If Brain A sees a dog but Brain B sees nothing (or just empty air), the system says, "Ah, that's a moving object. It's not part of the permanent world. Let's ignore it."

This allows the AI to build a clean 3D map of the street without accidentally gluing the moving people onto the buildings.

3. The "Reset Button" (Reset Metric Alignment)

Even with the two brains, if you watch a video for a very long time, the AI's memory can get a bit fuzzy, like a rubber band that has been stretched too far. The scale might get wrong (a car might look like a toy), or the position might drift.

RayMap3R has a clever safety net. Every so often, it looks at a frame it has seen before (a "repeated frame"). It checks: "Does the world look the same size and shape as it did last time?" If not, it performs a metric alignment—essentially a "reset button" that snaps the 3D model back to the correct size and position, fixing the drift before it gets worse.

4. The "Smooth Operator" (State-Aware Smoothing)

Sometimes, the camera shakes, or the AI gets a little jittery and thinks the camera moved when it didn't. This causes the 3D model to shake.

RayMap3R acts like a shock absorber on a car. It looks at how much the AI's "confidence" is changing.

If the AI is confident and moving steadily, it keeps the movement smooth.
If the AI is confused or the camera is shaking wildly, it applies a "brake" to smooth out the jitter, ensuring the final 3D path looks like a calm, steady drive rather than a bumpy ride.

Why This Matters

Before this, fixing moving objects in 3D reconstruction required extra, heavy software modules (like adding a separate engine to a car just to fix the tires). RayMap3R is training-free, meaning it doesn't need to be re-taught or given extra data. It simply uses the "ghost vision" trick that was already hiding inside the AI's existing brain.

In summary: RayMap3R is like a construction crew that builds a 3D city. Instead of getting distracted by the crowd of people walking by, they have a special tool that lets them see through the moving people to the buildings behind them. They build a perfect, stable city while the crowd keeps moving, and they never get dizzy or lost.

1. Problem Statement

Streaming feed-forward 3D reconstruction aims to estimate scene geometry (depth, point clouds) and camera poses in real-time from a continuous stream of RGB images. While recent models have achieved constant memory usage and real-time speeds, they face a critical limitation: dynamic interference.

The Challenge: Existing streaming models lack explicit mechanisms to identify moving objects. When dynamic objects (e.g., people, vehicles) are present, they corrupt the memory state used for reconstruction, leading to camera drift, geometric artifacts, and degraded trajectory estimation.
Limitations of Current Solutions: Traditional approaches often require external modules (e.g., optical flow estimators, segmentation networks) or additional training with dynamic annotations, which adds computational overhead and reduces generalization.

2. Core Observation: The Static Bias of RayMap

The authors identify a unique property in models trained with the RayMap representation (a per-pixel encoding of camera ray origins and directions).

The Phenomenon: When a model trained with RayMap is queried using only RayMap tokens (without image appearance features), it exhibits a strong static-scene bias.
Mechanism: RayMap tokens encode only camera geometry. Without visual appearance cues, the model relies on the latent memory state, which prioritizes temporally consistent structures (static background) and suppresses transient elements (dynamic foreground).
Insight: This creates a natural discrepancy between a prediction based on Image + RayMap (which sees the dynamic object) and a prediction based on RayMap-only (which ignores the dynamic object). This discrepancy serves as a training-free signal to identify dynamic regions.

3. Methodology: RayMap3R

RayMap3R is a training-free framework that leverages this static bias to perform dynamic-aware reconstruction. It consists of four main components:

A. Dual-Branch Inference Scheme

At each timestep $t$ , the system runs two parallel branches decoding from the same frozen latent state $s_{t-1}$ :

Main Branch: Processes both image tokens ( $f_t$ ) and RayMap tokens ( $r_t$ ) to predict the full scene (including dynamic objects).
RayMap Branch: Processes only RayMap tokens ( $r'_t$ ), constructed from the main branch's predicted pose. This branch produces a "static-biased" prediction, effectively hallucinating the scene without the moving objects.

B. Dynamic Identification & Memory Gating

Discrepancy Calculation: The system computes the per-pixel depth difference between the Main and RayMap branches. Large differences indicate dynamic regions.
Staticness Weights ( $\alpha_t$ ): These pixel-level differences are aggregated and projected onto the latent state tokens to generate weights $\alpha_t$ .
Gated Update: The state update is modulated by these weights: $s_t = s_{t-1} + \alpha_t \odot \Delta s_t$ $s_{t} = s_{t - 1} + α_{t} ⊙ Δ s_{t}$ .
- High $\alpha_t$ (static regions): Full state update.
- Low $\alpha_t$ (dynamic regions): Suppressed update, preventing moving objects from corrupting the long-term memory.

C. Reset Metric Alignment

Streaming models often suffer from metric drift when memory is periodically reset to manage long sequences.

Solution: The method detects "repeated frames" used for memory reset. It estimates a Sim(3) transformation to align the point cloud reconstruction of the repeated frame before and after the reset.
Effect: This corrects scale mismatches and pose offsets, ensuring metric consistency across the entire sequence.

D. State-Aware Smoothing

To stabilize camera trajectories and reduce noise accumulation:

Signal: The method calculates a smoothing coefficient $\beta_t$ based on two factors: trajectory acceleration and the magnitude of internal state changes ( $\Delta s_t$ ).
Adaptive Filtering: If the state change is large (indicating uncertainty or noise), the filter relies more on the accumulated history. If the motion is steady, it preserves the raw displacement. This operates fully online without storing explicit history.

4. Key Contributions

Discovery of Static Bias: The first work to exploit the inherent static-scene bias of RayMap-based models for dynamic identification without retraining or external supervision.
Training-Free Dynamic Filtering: A dual-branch inference scheme that identifies and suppresses dynamic regions during memory updates, effectively solving the "moving object" problem in streaming reconstruction.
Stabilization Techniques: Introduction of Reset Metric Alignment and State-Aware Smoothing to ensure metric consistency and trajectory stability over long sequences.
State-of-the-Art Performance: Achieves leading results among streaming methods on dynamic scene reconstruction benchmarks.

5. Experimental Results

The authors evaluated RayMap3R on diverse datasets including Sintel, TUM-Dynamics, KITTI, Bonn, and 7-Scenes.

Video Depth Estimation: Outperformed other streaming methods (CUT3R, TTT3R, StreamVGGT) in both per-sequence scale alignment and metric-scale absolute depth accuracy, particularly in dynamic outdoor scenes (KITTI).
Camera Pose Estimation: Achieved the lowest Absolute Translation Error (ATE) and Relative Pose Error (RPE) across all datasets. On Sintel (highly dynamic), it significantly reduced drift compared to baselines.
3D Reconstruction: On the 7-Scenes dataset, RayMap3R achieved the lowest accuracy error and best completion rates among streaming approaches, producing clearer and more geometrically faithful reconstructions than CUT3R and TTT3R.
Efficiency: Maintains constant memory usage (~9.4 GB) regardless of sequence length (tested up to 1000+ frames) and runs at ~13.8 FPS, making it suitable for real-time applications.

6. Significance

RayMap3R represents a paradigm shift in dynamic 3D reconstruction by demonstrating that implicit geometric biases in foundation models can be leveraged for complex tasks like dynamic scene understanding.

No Retraining Required: It works "out-of-the-box" with existing RayMap-trained models, eliminating the need for expensive dynamic data collection and retraining.
Robustness: It solves the critical issue of camera drift in dynamic environments, a major bottleneck for real-time SLAM and AR/VR applications.
Generalizability: The findings suggest that similar biases may exist in other feed-forward architectures, opening new avenues for training-free dynamic reasoning in computer vision.