Technical Summary: SCOPE – Simulating Cross-game Operations in Playable Environments for FPS World Models

1. Problem Statement

Interactive world models based on video diffusion have shown promise in simulating environments for games ranging from Atari to Minecraft. However, these models face a critical failure mode when applied to First-Person Shooter (FPS) games. FPS gameplay generates exceptionally dense, high-frequency control signals (e.g., camera sweeps exceeding 180°/s, simultaneous firing and movement, and rapid discrete events).

Existing world models typically employ global conditioning, where a single action embedding is broadcast uniformly across all spatial positions in the frame. While sufficient for sparse, low-frequency controls (like open-world navigation), this approach collapses under FPS conditions. A firing command intended for a localized region (the weapon's scope) inadvertently perturbs every pixel, causing structural distortions and artifacts. The core challenge is the inability of global conditioning to distinguish between in-scope regions (requiring localized, discrete visual changes) and out-of-scope regions (requiring stable, continuous generation driven by ego-motion).

2. Methodology: SCOPE

The authors propose SCOPE (Simulating Cross-game Operations in Playable Environments), a conditioning module designed to be inserted into each transformer block of a pretrained video diffusion model. The method relies on the observation that FPS actions are spatially selective: discrete events affect only localized areas, while continuous controls govern the broader scene.

2.1 Architecture

SCOPE is integrated into a pretrained Video Diffusion Transformer (DiT). It reshapes the latent token sequence into per-pixel temporal sequences, allowing each spatial position to compute its action response independently based on local visual content. The module processes two distinct action pathways:

• Discrete Events (In-Scope): Actions such as firing, reloading, or switching weapons are processed via visually-queried cross-attention. The per-pixel features serve as queries, while action embeddings serve as keys and values. This mechanism naturally confines effects to relevant spatial regions (e.g., muzzle flashes) without requiring explicit segmentation labels. Pixels outside the scope produce near-zero attention weights.
• Continuous Controls (Out-of-Scope): Camera and movement signals are processed via temporal self-attention combined with a fusion MLP. This pathway models smooth ego-motion and scene flow, ensuring that the background remains stable and is not contaminated by localized discrete effects.

The outputs of both pathways are combined via residual connections and added to the original features. All SCOPE output projections are zero-initialized, allowing the model to start as an unmodified video generator and progressively learn scope separation during end-to-end training.

2.2 Training and Inference

• Training: The model is trained end-to-end on the CrossFPS dataset using a flow matching objective. To prevent single-game dominance, training uses balanced sampling across seven different FPS titles.
• Action-CFG: During inference, the model supports tunable action intensity via Action Classifier-Free Guidance (Action-CFG). This is achieved through stochastic action dropout during training, allowing the model to interpolate between conditional and unconditional velocity predictions.

3. Key Contributions

3.1 The SCOPE Architecture

The paper introduces a novel conditioning mechanism that decomposes action effects into in-scope discrete responses and out-of-scope continuous generation. By utilizing per-pixel conditioning, SCOPE achieves spatial selectivity implicitly through visual content, eliminating the need for segmentation supervision.

3.2 The CrossFPS Dataset

The authors introduce CrossFPS, the first multi-game FPS dataset with frame-aligned action telemetry.

• Scale: Comprises 69,000 clips from 7 distinct FPS titles.
• Data: Includes 10-dimensional controller telemetry (4 continuous axes for movement/camera, 6 discrete buttons for combat/utility).
• Curation: The dataset undergoes rigorous processing to remove gameplay bias, including action distribution balancing, visual-action de-biasing (retaining "inefficient" actions to prevent learning strategies), and kinetic normalization to align action-to-pixel displacement ratios across different game engines.

3.3 Zero-Shot Generalization

The model is designed to learn general visual-to-action mappings rather than game-specific patterns. This enables zero-shot transfer to unseen scenes and game aesthetics without retraining.

4. Experimental Results

4.1 Quantitative Performance

Evaluated on the CrossFPS test set, SCOPE outperforms state-of-the-art baselines (Matrix-Game 3.0, LingBot-World, HY-World 1.5) on 7 of 8 metrics.

• Action Responsiveness: SCOPE achieves a Dynamic Degree of 0.910 and a Flow Score of 18.24, significantly outperforming baselines that often collapse to near-static outputs or suppress motion to maintain smoothness.
• Spatial Stability: With a Photometric Smoothness of 0.198, SCOPE demonstrates 3.2× better stability than the best baseline, confirming effective separation of in-scope and out-of-scope regions.
• Visual Quality: The model maintains high JEPA Similarity (0.806) and low FVD (690.3), indicating that the backbone's generative capabilities are preserved in out-of-scope regions.

4.2 Ablation Studies

• Spatial Selectivity: Removing the per-pixel conditioning (reverting to global injection) causes a severe degradation in Photometric Smoothness (0.198 → 0.745) and Dynamic Degree (0.910 → 0.521), reproducing the failure modes of existing methods.
• Training Strategy: End-to-end training yields superior results compared to frozen backbone or two-stage approaches, confirming the benefit of co-adapting the backbone with the action pathways.
• Scalability: Performance scales monotonically with data volume and diversity, showing no signs of saturation up to the full 65K clip dataset.

4.3 Zero-Shot Generalization

When tested on unseen scenes synthesized via GPT-image-2 (spanning stylized open-world, mythological, and sci-fi aesthetics), SCOPE maintains robust controllability.

• Controllability: The model achieves a 71.5% completion rate on unseen scenes for complex tasks (multi-action compositions and environment interactions), significantly outperforming baselines (e.g., LingBot-World at 38.3%).
• Limitations: The model handles appearance-level responses (fire, smoke, lighting) better than geometric transformations (structural deformation), reflecting the texture bias of the diffusion backbone.

5. Significance and Claims

The paper claims that SCOPE establishes a highly scalable open-world simulation framework capable of handling the dense, high-frequency control signals inherent to FPS gameplay. By shifting from global embeddings to per-pixel conditioning, the model resolves the fundamental conflict between localized action effects and global scene stability.

The authors emphasize that their approach enables zero-shot cross-game generalization using a compact dataset (69K clips), contrasting with prior works that require massive proprietary datasets or fail when transferred to unseen domains. The work suggests that per-pixel conditioning is a viable path toward more robust interactive world models, extending beyond FPS to broader egocentric interactive scenarios, though long-horizon stateful simulation remains a future direction.