Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models

1. Problem Statement

Current video generation models, often termed "world models," synthesize visual worlds by generating 2D frames. While they excel at creating visually coherent sequences under full observation, a critical question remains: Can these models decouple state evolution from observation?

In the real world, physical processes (e.g., water pouring, ice melting, objects falling) continue to evolve regardless of whether they are being observed, occluded, or viewed from a different angle. However, existing benchmarks primarily evaluate models under full observability or focus on static consistency. There is a lack of rigorous evaluation to determine if video world models possess an internal representation of "world state" that persists and evolves correctly when the camera turns away or when objects are occluded. If models cannot handle "out of sight" scenarios, they cannot support long-horizon interactions or generate truly dynamic, large-scale worlds.

2. Methodology: StEvo-Bench

The authors introduce StEvo-Bench, a comprehensive benchmark designed to evaluate whether video world models can maintain and evolve state correctly during periods of interrupted observation.

A. Task Construction

• Scope: The benchmark consists of 225 unique tasks spanning six categories of natural state evolution:
  1. Continuous processes (e.g., water pouring).
  2. Kinematics (e.g., rolling balls).
  3. Relational changes (e.g., stacking).
  4. Causal changes (e.g., switches turning on lights).
  5. State transformations (e.g., melting, burning).
  6. Expected human/animal behavior.
• Input: Each task provides an initial image and a text prompt.
• Controls: The prompt instructs the model to perform two simultaneous controls:
  1. Action Control: Initiates a dynamic process (e.g., "a hand tips a pitcher").
  2. Observation Control: Interrupts the view of the process mid-evolution. This is achieved via:
     • In-scene Occlusion: Inserting an object (curtain, cardboard) or turning off lights.
     • Camera Control: Directing the camera to "look away" (pan left/right) until the subject is out of frame, then return.
• Evaluation: The model is evaluated on whether the state of the main object has evolved correctly during the period it was invisible.

B. Evaluation Protocol & Automatic Verifiers

StEvo-Bench employs a two-stage pipeline with specialist Verifiers built upon Large Vision-Language Models (VLMs, specifically Gemini 3.1 Pro) to automatically detect and disentangle failure modes.

1. Control Success Check:

   • Observation Verifier: Did the scene become fully invisible for a sustained duration?
   • Action Verifier: Did the initiating action occur and trigger the expected physical response?
   • Result: Tasks failing these are excluded.

2. Evolution Success Check (on passing tasks):

   • State Progress Verifier: Did the intended process continue/evolve while the object was occluded? (e.g., Did the water level rise?)
   • Physical Plausibility Verifier: Did the evolution obey physics? (e.g., No floating objects, no gravity violations).
   • Coherence Verifier: Did the scene remain temporally consistent before and after occlusion? (e.g., No teleportation, no abrupt object changes).

3. Key Contributions

1. StEvo-Bench Benchmark: The first benchmark specifically designed to test the decoupling of state evolution from observation, covering diverse physical and causal dynamics.
2. Specialist Verifier Pipeline: A novel evaluation framework using ensemble VLMs to automatically diagnose specific failure modes (progress, physics, coherence) rather than providing a single holistic score.
3. Comprehensive Analysis: A quantitative and qualitative analysis of state-of-the-art (SOTA) video generation models and camera-controlled world models.

4. Key Results & Findings

The authors evaluated both general-purpose video models (e.g., Veo 3, Sora 2 Pro, Wan 2.2) and camera-controlled models (e.g., Genie 3, HunyuanWorld, LingBot).

• Finding 1: Universal Struggle with Decoupling.

  • Both general video models and camera-controlled models fail to evolve states correctly under observation control.
  • Success Rates: General video models achieved <10% overall success. Camera-controlled models performed even worse, with near-zero state progress rates (<5%).
  • Specific Failure: While models like Veo 3 and Sora 2 Pro could occasionally progress the state (17% and 13% respectively), they frequently failed to maintain physical plausibility or coherence.

• Finding 2: Failure Modes of General Video Models.

  • Evolution Stopping: When observation is interrupted (e.g., lights off), the model often "freezes" the state. The process stops evolving until the view is restored.
  • Incoherence: Upon re-emergence, objects often change shape, texture, or identity (e.g., a rectangular sponge becoming round) or disappear entirely.

• Finding 3: Static Bias in Camera-Controlled Models.

  • Camera-controlled models tend to assume the scene is static when the camera moves. When asked to "look away," they fail to evolve the scene, keeping it frozen even if the prompt implies dynamic action.
  • Trade-off: When these models do manage to evolve a state, they often fail to follow the camera control instructions, effectively ignoring the "look away" command to maintain the dynamic simulation.

• Finding 4: Memory Modules Do Not Help.

  • Models incorporating memory architectures (e.g., VMem) were tested to see if they could store state separately from pixels.
  • Result: Memory models reinforced a bias toward static scenes. They could recall the initial frame accurately but failed to evolve the state dynamically. The architecture prioritized memorizing appearance over modeling evolution.

• Finding 5: Data and Architecture Bias.

  • The failures suggest that current training data (rendered 3D scenes, gaming data, static indoor/outdoor videos) associates camera movement with static scenes and lacks examples of natural physics evolving under occlusion.
  • Standard attention mechanisms may be inefficient at handling "blind" periods where no visual feedback is available to guide state updates.

5. Significance

• Redefining World Models: The paper argues that a true "world model" must possess an internal state representation that evolves independently of observation. Current models are essentially "pixel predictors" that fail when visual feedback is removed.
• Benchmarking Standard: StEvo-Bench provides a rigorous, automated protocol to measure this specific capability, filling a gap left by existing benchmarks that focus only on visual quality or static consistency.
• Future Directions: The findings highlight the need for:
  • New training data strategies that include occluded dynamics.
  • Architectural changes to better separate state evolution from pixel generation (moving beyond naive bidirectional attention).
  • Development of models that can reason about physical causality even when the "eyes" are closed.

In conclusion, StEvo-Bench reveals that despite rapid progress in video generation, current world models lack a fundamental understanding of "out of sight, out of mind" dynamics, treating unobserved time as a pause rather than a continuation of physical processes.