SceneStreamer: Continuous Scenario Generation as Next Token Group Prediction

SceneStreamer is a unified autoregressive transformer framework that generates continuous, long-horizon traffic scenarios by predicting sequences of tokens representing dynamic elements like agents and traffic signals, thereby enabling the creation of realistic, diverse, and adaptive environments that significantly improve the robustness and generalization of autonomous driving policies.

The Gist

Imagine you are trying to teach a self-driving car how to navigate a busy city. The best way to do this is to let it practice in a video game simulator. But here's the problem: most current simulators are like old home movies. They just play back a recording of traffic that happened in the past.

If you (the self-driving car) try to brake suddenly in that movie, the other cars in the recording don't react. They just keep driving straight, likely crashing into you. This is bad for training because real traffic is interactive; if you brake, the car behind you should brake too.

SceneStreamer is a new kind of simulator that solves this. Instead of playing back a movie, it acts like a creative storyteller or a live director who improvises the scene in real-time.

Here is how it works, broken down into simple concepts:

1. The "Lego" Approach (Tokenization)

Imagine the entire traffic scene isn't a complex 3D video, but a long sentence made of Lego bricks.

• Some bricks represent the road (static).
• Some bricks represent traffic lights (changing colors).
• Some bricks represent cars, pedestrians, and bikes (moving agents).
• Some bricks represent how they move (accelerating, turning).

SceneStreamer treats the whole world as a single, long sentence. It doesn't try to draw a perfect picture all at once. Instead, it builds the scene one brick at a time, step by step, just like a human writes a story one word after another.

2. The "Infinite Party" (Continuous Generation)

In old simulators, the number of people at the "party" (the traffic scene) is fixed at the start. If a car leaves the road, the seat stays empty. If a new car wants to join, it can't.

SceneStreamer is different. It's like a live concert where the band can add new instruments or stop playing instruments mid-song.

• New Agents: If a car turns into a side street, SceneStreamer can "spawn" a new car entering the main road at that exact moment.
• Retiring Agents: If a pedestrian walks off the screen, the model knows to stop tracking them.
• The Result: The simulation can run forever (an "unbounded horizon"), creating realistic, long-duration traffic jams or free-flowing streets without getting stuck.

3. The "Causal Chain" (How it Thinks)

The paper mentions "autoregressive generation." Think of this as a domino effect.

• To know where a car goes next, the model first decides: What kind of car is it? (A truck or a bike?)
• Then: Where is it on the map? (On a highway or a sidewalk?)
• Then: What is it doing right now? (Speeding or stopped?)
• Finally: Where will it be in the next second?

Because it decides these things in a specific order (like a logical story), it avoids silly mistakes. For example, it won't accidentally put a pedestrian on a highway or make a car drive sideways. It understands the "rules of the road" because it builds the scene logically, step-by-step.

4. The "Training Gym" (Why it Matters)

The authors used this new simulator to train self-driving cars using Reinforcement Learning (a method where the AI learns by trial and error, like a dog learning tricks for treats).

• The Old Way: The AI practiced against "ghosts" (recorded data) that didn't react. It learned to drive perfectly only in those specific, static situations.
• The SceneStreamer Way: The AI practiced against a reactive, living world. If the AI made a risky move, the simulated traffic reacted realistically (e.g., other cars swerved to avoid it).

The Result: The self-driving cars trained in SceneStreamer became much tougher and smarter. They learned to handle surprises and generalize better to real-world driving, just like a student who practices with a live sparring partner instead of a punching bag.

Summary Analogy

• Old Simulators: Like watching a scripted TV show. The actors follow a script and ignore you. If you try to change the plot, the show breaks.
• SceneStreamer: Like a live improv comedy show. The actors (traffic agents) react to your moves instantly. The story evolves naturally, new characters can join the stage, and the plot can go on forever.

By turning traffic simulation into a "storytelling" task, SceneStreamer creates a much more realistic, flexible, and safe environment for teaching self-driving cars how to survive on the road.

Technical Summary

1. Problem Statement

Autonomous driving (AD) systems require realistic, interactive, and long-horizon traffic simulations for training and evaluation. Existing approaches suffer from several critical limitations:

• Static Initialization & Log-Replay: Most methods rely on replaying logged trajectories or static initializations. These lack interactivity; background agents do not react to the ego vehicle, limiting closed-loop evaluation.
• Fixed Agent Populations: Traditional motion prediction models assume a fixed set of agents. They cannot dynamically introduce new traffic participants (e.g., cars entering from side streets) or retire existing ones during a simulation, failing to model traffic as an open system.
• Covariate Shift & Error Accumulation: One-shot prediction models often fail when unrolled over time. Small prediction errors compound, leading the simulator into out-of-distribution states and producing unrealistic outcomes.
• Disjoint Pipelines: Many generative models separate "initial state generation" from "motion prediction." This prevents the model from sharing context between initialization and motion, leading to inconsistencies and inflexibility in editing scenarios.

2. Methodology: SceneStreamer

The authors propose SceneStreamer, a unified autoregressive framework that treats the entire traffic scenario (static maps, dynamic agents, and traffic signals) as a single sequence of discrete tokens.

Core Concept: Token Group Prediction

Instead of predicting continuous coordinates, SceneStreamer casts scenario generation as a next-token prediction task (similar to Large Language Models). The scenario is represented as a sequence of token groups:
$$x_{1:T} = [\langle MAP \rangle; (\langle TL \rangle, \langle AS \rangle, \langle MO \rangle)_1; (\langle TL \rangle, \langle AS \rangle, \langle MO \rangle)_2; \dots]$$
Where:

• $\langle MAP \rangle$: Static map context (vectorized lanes, crosswalks).
• $\langle TL \rangle$: Traffic light states (Green, Yellow, Red, Unknown).
• $\langle AS \rangle$: Agent State tokens (Type, Map Location, Relative State).
• $\langle MO \rangle$: Agent Motion tokens (Acceleration, Yaw Rate).

Key Architectural Components

1. Unified Tokenization:

   • Map Tokens: Encoded via a PointNet-like encoder, assigned unique IDs, and serve as static cross-attention keys/values.
   • Agent State Tokens ($\langle AS \rangle$): Each agent is represented by a sequence of 4 tokens:
     1. Start-of-Agent (SOS): Flag.
     2. Type: Vehicle, Pedestrian, or Cyclist.
     3. Map ID: The specific map segment (lane) where the agent resides.
     4. Relative State (RS): An 8D vector encoding physical dimensions (L, W, H), position offset, heading residual, and velocity relative to the selected map segment.
   • Motion Tokens ($\langle MO \rangle$): Discretized control inputs $(a, \omega)$ (acceleration and yaw rate) mapped to a 2D grid (1,089 classes).

2. Autoregressive Generation Scheme:

   • Step-by-Step Rollout: The model generates tokens step-by-step. At each time step $t$, it predicts traffic lights, then agent states, then motions.
   • Dynamic Agent Injection: New agents are introduced by sampling new $\langle AS \rangle$ token groups. Existing agents are handled via State-Forcing: the model bypasses the generative head for existing agents and directly feeds reconstructed state tokens (derived from previous motion predictions) back into the sequence. This allows the agent population size to vary dynamically over an unbounded horizon.
   • Relative State Head: A dedicated small Transformer decoder generates the 8D relative state vector autoregressively, conditioned on the selected Map ID. This ensures agents are placed physically consistent with the map geometry.

3. Attention Mechanism:

   • Group Causal Attention: Tokens within the same group (e.g., all motion tokens at step $t$) can attend to each other. Tokens can attend to logically preceding groups (e.g., Motion attends to State) and historical contexts.
   • Relative Attention: Uses geometric biases ($\Delta x, \Delta y, \Delta \psi, \Delta t$) to modulate attention weights, making the model invariant to global coordinates and improving generalization.

3. Key Contributions

1. Unified State & Trajectory Tokenization: A single autoregressive model generates both initial agent states and their future trajectories as one continuous token sequence. This eliminates the inconsistency between initialization and motion prediction found in two-stage models.
2. Agent State Autoregressive Generation: A novel scheme that predicts agent type, map location, and detailed kinematic states sequentially. This allows the model to place agents accurately on specific map segments and generate realistic state details in a compact representation.
3. Versatile Capabilities via State-Forcing: By dynamically choosing which tokens to sample (for new agents) and which to state-force (for existing agents), SceneStreamer supports multiple tasks without model modification:
   • Motion Prediction.
   • Full Scenario Generation (from scratch).
   • Scenario Densification (injecting new agents).
   • Closed-loop Simulation for RL training.

4. Experimental Results

Experiments were conducted on the Waymo Open Motion Dataset (WOMD).

• Initial State Quality: SceneStreamer achieved competitive Maximum Mean Discrepancy (MMD) scores compared to state-of-the-art methods (TrafficGen, UniGen, MotionCLIP). Crucially, it demonstrated superior performance when using autoregressive decoding, avoiding invalid combinations (e.g., pedestrians on highways) that flat decoding often produces.
• Motion Prediction: While the full generative model showed slightly higher displacement errors (ADE/FDE) than a motion-only baseline (due to the complexity of joint generation), it achieved significantly higher diversity (ADD/FDD), indicating a broader coverage of plausible behaviors.
• Reinforcement Learning (RL) Training:
  • RL planners trained on SceneStreamer-generated scenarios significantly outperformed those trained on standard log-replay data.
  • Metrics: Improved Success Rate (+2.3%), Route Completion (+6.2%), and reduced Collision Rate (-39%) compared to the log-replay baseline.
  • Adaptive Training: The best results were achieved using "Adaptive" training, where the ego vehicle's trajectory was updated based on the planner's policy, creating a true closed-loop interaction.
• WOSAC Benchmark: SceneStreamer achieved competitive realism and behavioral likelihood scores on the Waymo Sim Agents Challenge, validating its efficacy as a general-purpose simulator.

5. Significance and Impact

• Bridging the Sim-to-Real Gap: By enabling dynamic, reactive, and diverse traffic populations, SceneStreamer provides a high-fidelity environment for training robust AD policies that generalize better to novel, unseen scenarios.
• Efficiency and Flexibility: The token-based approach unifies initialization and motion prediction, removing the need for complex, disjointed pipelines. It supports "scene editing" (adding/removing agents) mid-simulation, which is critical for stress-testing AD systems.
• Foundation for Closed-Loop Simulation: The ability to run long-horizon, autoregressive simulations where agents react to the ego vehicle makes SceneStreamer a powerful tool for training reinforcement learning agents, moving beyond the limitations of passive log-replay simulators.

In summary, SceneStreamer represents a paradigm shift from static, fixed-population simulation to a dynamic, token-based generative framework that treats traffic scenarios as a language-like sequence, enabling realistic, interactive, and scalable autonomous driving research.