💻 computer science

StreetForward: Perceiving Dynamic Street with Feedforward Causal Attention

StreetForward is a pose-free and tracker-free feedforward framework that leverages a temporal mask attention mechanism and 3D Gaussian Splatting to jointly reconstruct static and dynamic street scenes, enabling high-fidelity novel view synthesis and per-pixel velocity estimation without per-scene optimization.

Original authors: Zhongrui Yu, Zhao Wang, Yijia Xie, Yida Wang, Xueyang Zhang, Yifei Zhan, Kun Zhan

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhongrui Yu, Zhao Wang, Yijia Xie, Yida Wang, Xueyang Zhang, Yifei Zhan, Kun Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a busy city street through a window. Cars zoom by, pedestrians cross the road, and the sun casts moving shadows. Now, imagine you want to build a perfect, 3D digital twin of that street so you can:

Walk around it (change your viewpoint).
Rewind or fast-forward time (see the street at a different moment).
Do all this instantly, without spending days or weeks calculating the math for every single scene.

That is exactly what the paper StreetForward is trying to solve.

Here is the breakdown using simple analogies:

The Problem: The "Slow Cooker" vs. The "Microwave"

Most current 3D reconstruction methods are like slow cookers. To create a 3D model of a street, they have to spend hours or days "cooking" (optimizing) the data for that specific scene. They need to track every car, every person, and every tree individually to figure out how they move. If you want to simulate a new street, you have to start the slow cooking process all over again. This is too slow for self-driving cars, which need to process thousands of miles of road data instantly.

StreetForward is the microwave. It uses a "feedforward" approach, meaning it looks at the video, processes it in one quick pass, and instantly spits out a 3D model. It doesn't need to "think" about the specific street beforehand; it just knows how to do it based on what it learned from millions of other streets.

The Secret Sauce: "Causal Attention" (The Time-Traveling Detective)

The biggest challenge in making a 3D street is separating the static stuff (buildings, roads, trees) from the dynamic stuff (cars, people).

Old methods often get confused. They might think a parked car is moving, or they might lose track of a pedestrian and make them disappear.
StreetForward uses a special trick called Causal Masked Attention.

The Analogy: Imagine you are a detective watching a security camera.

Old AI: Looks at the whole video at once, getting confused about who moved when. It might mix up a car from 5 seconds ago with a car right now.
StreetForward: It acts like a detective who only looks at the cause and effect. It asks: "If I look at this car right now, where was it 1 second ago? Where will it be 1 second later?"

By forcing the AI to look strictly in a "cause-and-effect" order (past $\to$ present $\to$ future), it becomes incredibly good at predicting velocity (speed and direction). It learns that "cars move forward," "people walk," and "buildings stay still," without needing a human to tell it which is which.

The Magic Trick: 3D "Splats" (The Digital Confetti)

The paper uses a technique called 3D Gaussian Splatting.

The Analogy: Imagine throwing a handful of digital confetti into the air. Each piece of confetti is a tiny, fuzzy 3D dot with a color, a size, and a position.
The Innovation: In the past, these dots were just static. StreetForward gives these dots superpowers.
- Static dots (buildings) stay put.
- Dynamic dots (cars) are given a "velocity vector" (an arrow showing speed and direction).
- The AI learns to attach these arrows to the dots. So, when you want to see the street 2 seconds in the future, the AI just slides the "car dots" along their arrows to their new positions and renders the image instantly.

Why is this a Big Deal?

No "Training Wheels" Needed: You don't need to give the AI special labels telling it "this is a car" or "this is a person." It figures out the motion on its own just by watching the video.
Time Travel: Because it understands the speed of every object, you can pause the video, move the camera to a new spot, and then play the video forward or backward, and the objects will still be in the right place.
The "Zero-Shot" Superpower: The team trained the AI on real-world data from Waymo (a huge dataset of real driving). Then, they tested it on CARLA (a video game simulation). The AI had never seen the game before, yet it worked perfectly. It's like teaching someone to drive on a real highway, and then handing them the keys to a video game car—they know how to drive immediately.

Summary

StreetForward is a new AI system that turns a video of a busy street into a fully interactive, 3D time-traveling world in a split second. It does this by teaching the AI to understand cause and effect in motion, allowing it to predict where every car and person will be without needing to track them individually. This makes it a perfect tool for training self-driving cars to handle the real world safely and efficiently.

1. Problem Statement

Autonomous driving simulation and closed-loop testing require rapid, high-fidelity reconstruction of dynamic street scenes (containing both static infrastructure and moving agents like vehicles and pedestrians).

Limitations of Existing Methods:
- Iterative Optimization: Traditional methods (SfM, NeRF, standard 3DGS) rely on time-consuming per-scene optimization, making them impractical for large-scale, diverse datasets.
- Feedforward Gaps: Recent feedforward models (e.g., DUSt3R, VGGT) excel at static scenes but struggle with 4D dynamic reconstruction. They often suffer from geometry degradation under motion and lack robust motion modeling.
- Dependencies: Current 4D solutions often require explicit 4D supervision, external object trackers, or segmentation masks, which introduce failure modes and limit generalization.

Goal: Develop a pose-free, tracker-free, and segmentation-free feedforward framework capable of reconstructing dynamic street scenes with high-fidelity novel view synthesis and depth estimation at arbitrary poses and timestamps.

2. Methodology: StreetForward

The proposed framework builds upon the Visual Geometry Grounded Transformer (VGGT) architecture but introduces specific modifications to handle dynamic motion without explicit supervision.

A. Core Architecture & Tokenization

Input: Video sequences are encoded into patchified tokens using a frozen DINOv2 encoder.
Backbone: Uses VGGT's Alternating Attention (AA) mechanism, interleaving cross-frame (global) and intra-frame attention to aggregate scene-aware features.
Representation: Both static and dynamic elements are represented uniformly using 3D Gaussian Splatting (3DGS).
- Static Gaussians: Persist across the entire sequence.
- Dynamic Gaussians: Aggregated across frames using predicted velocities.

B. Key Innovations

Causal Masked Attention (Motion Head):
- Standard AA treats all frames equally, weakening directed motion cues. StreetForward introduces a temporal mask attention module that explicitly models source $\to$ target frame relationships (causal attention).
- This strengthens motion-aware features by restricting attention to specific source-target pairs (e.g., $f_t \to f_{t+1}$ ), enabling the model to infer per-pixel velocities without explicit 4D labels.
Joint Optimization of Geometry and Motion:
- The model predicts camera poses, depth, Gaussian attributes (position, covariance, color, opacity), and a dense velocity field simultaneously.
- No External Tracking: Dynamic regions are separated from static ones via a predicted dynamic probability mask, eliminating the need for external instance trackers or segmentation models.
Regularization Strategies:
- Local Rigidity ( $L_{rigid}$ ): Enforces piecewise-rigid motion in both 2D image space and 3D space to prevent velocity drift and structural floaters.
- Forward-Backward Consistency ( $L_{fb}$ ): Penalizes deviations between forward and backward velocity predictions to ensure temporal symmetry and smooth trajectories.
- Opacity Stabilization ( $L_{\alpha}$ ): Prevents 3D Gaussians from collapsing opacity to zero (a common failure mode in multi-frame fusion) by biasing them to retain non-trivial opacity, forcing the model to correct geometry rather than hide errors.
Rendering Strategy:
- To render a frame at time $t$ , the model uses static Gaussians from the whole sequence and warped dynamic Gaussians from adjacent frames ( $t-1, t+1$ ).
- Crucially, it excludes dynamic Gaussians parameterized at the current frame $t$ during rendering. This forces the model to explain the current view using motion-propagated data from neighbors, preventing degenerate solutions.

3. Key Contributions

Pose/Tracker/Segmentation-Free Framework: A fully feedforward 4D reconstruction method that requires no camera poses, object tracking, or semantic segmentation at inference time.
Causal Dynamics Modeling: A novel causal masked attention mechanism that enhances VGGT's ability to model directed motion without explicit 4D supervision.
Unified 3DGS Representation: A joint optimization strategy that handles static and dynamic content uniformly, enabling high-fidelity rendering at novel poses and novel times.
Strong Generalization: Demonstrated zero-shot transfer capabilities to unseen domains (e.g., training on Waymo, testing on CARLA).

4. Experimental Results

The model was evaluated on the Waymo Open Dataset and the CARLA simulator.

Novel View Synthesis (Waymo):
- Achieved 24.30 PSNR and 0.827 SSIM on dynamic-only regions, outperforming baselines like DGGT (20.99 PSNR) and STORM (22.10 PSNR).
- Successfully synthesized coherent motion for deformable objects (pedestrians) and rigid objects (vehicles) without ghosting or drifting artifacts common in other methods.
Depth Estimation:
- Achieved the lowest RMSE on both static (3.09) and dynamic (3.45) regions compared to VGGT, Pi3, and DGGT.
- Demonstrated superior reconstruction of thin structures (poles, traffic lights) and dynamic vehicle geometry.
Zero-Shot Generalization (CARLA):
- Trained solely on Waymo, the model achieved 22.37 PSNR on CARLA dynamic regions without fine-tuning, significantly outperforming DGGT (21.32 PSNR) and other baselines.
Ablation Studies:
- Removing causal attention led to significant drops in dynamic region quality (ghosting, temporal blur).
- Removing rigidity regularization caused structural floaters.
- Removing opacity stabilization resulted in "black holes" (low-opacity artifacts) in novel views.

5. Significance

StreetForward represents a significant leap forward in autonomous driving simulation and scene understanding:

Efficiency: It eliminates the computational bottleneck of per-scene optimization, enabling the rapid processing of large-scale driving datasets.
Robustness: By removing dependencies on external trackers and segmentation, it avoids the failure modes associated with those components (e.g., tracking loss, segmentation errors).
Versatility: It supports spatial extrapolation (new camera poses) and temporal interpolation/extrapolation (new time steps), which are critical for closed-loop simulation where the agent must predict future states or view the scene from hypothetical angles.
Scalability: The feedforward nature allows the model to scale to massive datasets, facilitating the creation of high-fidelity digital twins for autonomous vehicle development.