Optimization-Guided Diffusion for Interactive Scene Generation

1. Problem Statement

Generating realistic, diverse, and controllable multi-agent driving scenes is critical for evaluating autonomous vehicles (AVs). However, current approaches face two significant limitations:

• Lack of Controllability and Physical Consistency: Existing diffusion-based generators often produce samples that violate kinematic constraints (e.g., impossible vehicle dynamics), social rules (e.g., driving off-road), or interaction logic (e.g., collisions). While they learn data distributions, the denoising process during inference can accumulate small inconsistencies, leading to physically invalid trajectories.
• Inability to Generate Safety-Critical Scenarios: Safety-critical events (long-tail scenarios like near-collisions) are rare in real-world datasets. Standard diffusion models tend to gravitate toward dominant, safe motion patterns, making it difficult to synthesize rare but necessary adversarial scenarios without retraining or manual scripting.

2. Methodology: OMEGA

The authors propose OMEGA, a training-free, optimization-guided framework that enhances existing diffusion-based scene generators. It operates by re-anchoring the reverse diffusion steps via constrained optimization rather than modifying the model weights.

Core Components:

A. Optimization-Guided Re-anchoring
Instead of blindly following the predicted clean sample ($\tilde{x}_0$) from the diffusion model, OMEGA treats each reverse diffusion step as a conditional sampling process. It solves a constrained optimization problem to find an optimized anchor ($\hat{x}_0$) that balances:

1. Guidance Objective ($R(x)$): Encodes structural and behavioral priors (e.g., kinematic feasibility, collision avoidance, smoothness).
2. KL-Divergence Constraint: Limits the deviation from the original model's learned distribution to prevent "mode collapse" or unrealistic drift.

• Mechanism: The method formulates the update as maximizing a reward function within a trust region defined by the KL divergence. This effectively translates the KL constraint into an adaptive Euclidean trust region, where the radius scales with the diffusion noise level ($\sigma_t$). Early steps allow broader exploration, while later steps enforce strict adherence to the manifold.

B. Phase-Aligned Guidance Schedule
To handle the complexity of multi-agent interactions, OMEGA employs a two-phase denoising schedule:

1. Warm-up Phase: Performs full-sequence denoising with agent-wise structural guidance. This establishes a globally plausible motion layout (kinematic consistency, road adherence) without worrying about fine-grained interactions.
2. Rolling-Zero Phase: Operates in a low-noise regime, progressively refining the scene frame-by-frame. It activates inter-agent interaction guidance, allowing agents to react to each other dynamically while preserving the global structure established in the Warm-up phase.

C. OMEGA-Adv: Sensitivity-Enhanced Adversarial Generation
To generate safety-critical scenarios, the authors introduce OMEGA-Adv, which models the interaction between an ego vehicle and an attacker as a distributional game:

• Game Formulation: The ego seeks to maintain realistic behavior, while the attacker seeks to maximize perturbation (risk) within the learned data distribution.
• Sensitivity-Enhanced Iterative Best Response (SE-IBR): To approximate the Nash equilibrium efficiently, the attacker uses a sensitivity term derived from the ego's optimal response. Specifically, the attacker predicts how the ego will react to collision-avoidance constraints (using KKT multipliers) and adjusts its trajectory to exploit these reactions, generating realistic "cut-in" or "sudden braking" scenarios without explicit trajectory scripting.

3. Key Contributions

1. Training-Free Framework: OMEGA improves existing diffusion models without retraining, making it a drop-in replacement for inference.
2. Structural Consistency via Optimization: By re-anchoring reverse steps via constrained optimization, it enforces physical and social constraints, significantly reducing invalid samples.
3. Phase-Aligned Scheduling: The separation of macro-level layout formation (Warm-up) and micro-level interaction refinement (Rolling-Zero) improves both global plausibility and local reactivity.
4. Adversarial Generation via Game Theory: OMEGA-Adventures generates challenging, safety-critical scenarios by modeling ego-attacker interactions as a sensitivity-enhanced optimization game, avoiding the need for manual adversarial trajectory design.

4. Experimental Results

The method was evaluated on nuPlan (training data) and Waymo (zero-shot generalization).

• Free Exploration (Realism & Validity):

  • On nuPlan, the ratio of physically and behaviorally valid scenes increased from 32.35% (baseline Nexus) to 72.27%.
  • On Waymo (zero-shot), validity improved from 61.71% to 81.95%.
  • Significant reductions in collision rates and off-road events were observed across all baselines.

• Controllability (Goal-Conditioned):

  • When forcing agents to reach specific goal points, OMEGA achieved a validity rate of 80%, compared to 11% for the baseline. This demonstrates precise control without sacrificing physical feasibility.

• Adversarial Generation:

  • OMEGA-Adv generated 5$\times$ more near-collision frames (Time-to-Collision < 3s) compared to baselines.
  • It successfully increased the "Ego-at-fault" collision rate (indicating realistic adversarial pressure) while keeping "Non-responsible" collisions low, proving the scenarios are meaningful and not just random crashes.

5. Significance

• Bridging the Gap: OMEGA addresses the "rarity" problem in driving datasets by synthetically generating safety-critical long-tail scenarios that are essential for robust AV testing.
• Practical Integration: Being training-free, it can be immediately applied to existing state-of-the-art diffusion models, accelerating the deployment of high-fidelity simulators.
• Safety Evaluation: By generating diverse, realistic, and controllable adversarial scenarios, OMEGA provides a powerful tool for stress-testing autonomous driving systems in a safe, virtual environment, potentially reducing the need for dangerous real-world testing.

In summary, OMEGA represents a shift from purely generative diffusion models to guided, optimization-based sampling, ensuring that generated driving scenes are not only statistically probable but also physically valid, socially compliant, and capable of simulating critical safety events.