Technical Summary: Training-Free Norm Injection for Driving World Models

Problem Statement

Driving world models based on large video-diffusion backbones are capable of generating realistic sensor streams but suffer from a critical lack of controllability. Current state-of-the-art strategies to enforce traffic norms (e.g., adhering to speed limits, avoiding collisions) typically require either fine-tuning the backbone for every new constraint or conditioning the model on hand-built layouts (maps, bounding boxes) baked in during training. These approaches are costly to scale and prevent a deployed model from adding or re-prioritizing rules at sampling time. The authors pose the central question: Does controllability require training at all? Specifically, can a frozen world model be steered to satisfy driving norms solely through sampling-time interventions?

Methodology

The paper proposes a training-free energy guidance approach applied to a rectified-flow driving world model.

1. Model Architecture

The backbone is a frozen Open-Sora 2.0 MM-DiT (Multi-Modal Diffusion Transformer) model, post-trained with LoRA to jointly denoise:

• Future video frames (image stream).
• A planned ego trajectory $x_{ego} \in \mathbb{R}^{T \times d}$ (text stream, where $T=40$ frames and $d=4$ components: $x, y, \theta, v$).
  Crucially, the model is trained without agent or map information, relying only on observed past frames.

2. Training-Free Guidance Mechanism

The authors adapt the concept of energy-based guidance to rectified-flow models. Instead of retraining, driving norms are injected as differentiable energy functions $E(\cdot)$ evaluated on the predicted clean ego trajectory $\hat{x}_0$ at each sampling step.

The process involves:

1. One-Step Estimation: At time step $t$, the model estimates the clean sample $\hat{x}_0$ using the learned velocity field $v_\theta$:
   $$\hat{x}_0(x_t, t) = x_t - t \cdot v_\theta(x_t, t)$$
2. Energy Descent: A gradient step is taken in the clean-trajectory space to minimize the driving norm energy:
   $$\hat{x}'_0 = \hat{x}_0 - \lambda(t) \nabla_{\hat{x}_0} E(\hat{x}_0)$$
   Here, $E(\hat{x}_0)$ is defined as a Mean Squared Error (MSE) attraction toward a counterfactual target (e.g., a braking target simulating a slow lead vehicle).
3. Velocity Correction: The updated estimate is converted back to a velocity correction for the sampling ODE:
   $$v' = \frac{x_t - \hat{x}'_0}{\max(t, t_{min})}$$
4. Stabilization: To prevent the trajectory from drifting off the data manifold (a known issue with training-free guidance when $\hat{x}_0$ is inaccurate early in sampling), the per-step guidance correction is capped at a fixed multiple of the backbone's own velocity. An asymmetric schedule $\lambda(t)$ is used, setting guidance to zero near $t=0$ and $t=1$ to avoid numerical instability and allow the prior to establish global structure.

Key Contributions

1. Formulation: The authors frame driving-norm compliance as a training-free energy guidance problem on a frozen rectified-flow world model, demonstrating that knowledge can be added entirely at sampling time.
2. Empirical Validation: They demonstrate that this method successfully steers the ego trajectory of a world model built on Open-Sora 2.0 to a counterfactual braking target (simulating a rear-end collision avoidance scenario) without retraining the backbone.
3. Architectural Insight: The paper identifies a critical limitation in current joint-generation architectures: while the trajectory is successfully steered, the generated video does not follow the steered trajectory. The authors attribute this failure to the specific cross-stream coupling mechanism (joint self-attention) in the current model, where the video stream fails to propagate the changes induced in the trajectory stream.

Experimental Results

The authors evaluated the method on 35 sampled clips using a counterfactual braking target:

• Trajectory Control: The guidance was highly effective for the trajectory. The guided trajectory ended a mean of 1.3 meters from the brake target, compared to 13.6 meters for the unguided (factual) trajectory.
• Video Consistency: Despite the significant change in the ego trajectory, the generated video did not reflect this change. The per-frame optical flow difference between the guided and unguided videos was statistically indistinguishable (a change of $\lesssim 2\%$, within the noise of across-clip variation).
• Conclusion on Results: The "steered plan does not move the pixels." The video stream remains decoupled from the trajectory stream under the current architecture.

Significance and Claims

The paper claims that driving-norm compliance does not require retraining. A frozen world model can have its planned ego trajectory steered at sampling time using differentiable energy functions. This offers a pathway for deployed models to adapt to new rules or prioritize safety constraints dynamically.

However, the authors are modest about the current state of end-to-end controllability. They explicitly state that the video generation does not yet follow the steered trajectory. They identify the cross-stream coupling (specifically the bypass of single-stream blocks in their modified model) as the central obstacle. The paper concludes that achieving true end-to-end controllable rollouts requires a deeper joint denoising of the video and trajectory streams, proposing that routing both streams through the single-stream blocks of the backbone is the necessary next step to align the video with the guided trajectory.

The work positions itself as a bridge between training-free guidance (previously applied only to trajectory-only models) and controllable world models (which previously required training-time conditioning), highlighting the architectural gap that remains to be closed for full semantic control.