Technical Summary: MemoBench

Problem Statement

Video generation models aspire to function as "world generators" capable of simulating dynamic environments where physical processes (e.g., melting ice, flowing traffic) continue regardless of observation. While recent benchmarks evaluate visual quality, temporal coherence, and physical adherence, they predominantly focus on scenes where objects remain continuously visible. A critical gap exists in evaluating visual memory under partial observability: specifically, whether generative models can maintain and update the state of objects that temporarily leave the field of view (occlusion) and correctly recover them upon reappearance. Current benchmarks fail to test if models truly "remember" an object's evolving state (identity, geometry, physical process) during the "disappear-and-reappear" phase, often conflating static consistency with genuine world modeling.

Methodology

1. MemoBench Dataset

The authors introduce MemoBench, a diagnostic benchmark comprising 360 ground-truth video clips (1920×1080 resolution) designed around a disappear-and-reappear paradigm. The dataset is divided into two complementary subsets:

• Synthetic Subset (196 clips): Generated in Unreal Engine 5, featuring 14 scene subdomains across 5 environment categories. These clips utilize scripted camera trajectories (e.g., U-turns, forward motion) to drive the target object out of and back into the field of view while a physical process continues.
• Real-World Subset (164 clips): Captured in controlled indoor settings, covering 30 physical-state-change processes (e.g., dissolving, pouring, burning) across 7 categories. These clips rely on material properties (viscosity, elasticity) that are difficult for game engines to model accurately.

Each clip is annotated with keyframes defining three phases: Visible (V), Disappeared (D), and Reappear (R). The dataset includes ground-truth RGB, metric depth, and camera poses.

2. Evaluation Suite

MemoBench employs a dual-evaluation strategy combining automated metrics and Large Language Model (LLM)-based Visual Question Answering (VQA).

A. Automated Metrics:

• General Video Quality: Visual Quality (AestheticScore, CLIP-IQA+), Motion Smoothness (RAFT optical flow), Object Identity Consistency (DINOv2 patch similarity), and Geo3D Consistency (Depth Anything V2).
• Memory-Specific Metrics:
  • Object Reappearance Score (ORS): Uses SAM-3 to detect if the target object reappears in the R-phase with valid coverage and confidence.
  • Pixel-Level Fidelity: PSNR, SSIM, and LPIPS computed separately for V, D, and R phases to isolate degradation upon reappearance.
  • Camera Controllability: Measures rotation error between estimated and ground-truth camera trajectories using MapAnything.
• Prompt Fidelity: ImageReward Score.

B. VQA-Based Evaluation:
A multi-stage pipeline uses an LLM (Gemini-3.1-Pro) to generate 24 polarity-balanced Yes/No questions per clip, covering four diagnostic dimensions:

1. Instruction Following: Adherence to spatiotemporal instructions.
2. Object & Background Consistency: Detection of morphing or identity switches.
3. Continuity of Memory: Whether the model preserves identity, trajectory, and state after occlusion.
4. Physics Adherence: Plausibility of locomotion, gravity, and lighting.
   The question bank is rigorously filtered via ground-truth verification, failure-case testing, and human cross-validation (achieving 92.9% agreement with human judgments).

3. Benchmarking Protocol

The authors evaluated 10 state-of-the-art world generation models, categorized into:

• Control-Conditioned Text-Image-to-Video (C+TI2V): Models accepting explicit camera poses or actions (e.g., LingBot-World, HunyuanWorldPlay, Stable Virtual Camera).
• Standard Text-Image-to-Video (TI2V): Models without explicit camera conditioning (e.g., Open-SoRA, LTX-Video, CogVideoX).

Key Results

• Memory Failure is Ubiquitous: No model achieved an Object Reappearance Score (ORS) exceeding 0.6, indicating that even top performers fail to reliably regenerate the target object in its updated state after occlusion.
• Camera Conditioning vs. Memory: Explicit camera conditioning improves trajectory precision (Camera Controllability) but does not guarantee object memory. For instance, models with camera conditioning often produced recognizable frames but failed to recall the specific state of the occluded object.
• The "Static" Inflation Artifact: Models without camera conditioning (specifically LTX-Video) achieved deceptively high scores in General Video Quality and ORS. This was attributed to camera inactivity; by generating near-static videos, these models trivially satisfied consistency metrics and avoided the "disappear" phase entirely, masking their inability to handle dynamic viewpoint changes.
• Trade-offs: A tension exists between geometric fidelity and perceptual quality. Stable Virtual Camera achieved high pixel-level fidelity and camera control but suffered from rendering artifacts (blurring, seams), resulting in lower visual quality scores. Conversely, Matrix-Game 2.0 produced sharp frames but drifted significantly from the ground-truth viewpoint.
• VQA Insights: Instruction Following scores were highest for C+TI2V models, confirming that explicit control aids in executing instructions. However, Continuity of Memory remained a bottleneck for all models, with even the best-performing models answering fewer than 50% of memory-related questions correctly.

Significance and Claims

The paper claims that MemoBench is the first benchmark to explicitly evaluate memory consistency in world generation through the disappear-and-reappear paradigm. Its significance lies in:

1. Exposing a Fundamental Gap: It reveals that current generative models struggle to maintain a persistent internal representation of objects once they leave the field of view, treating them as if they are "forgotten" rather than "occluded."
2. Diagnosing Evaluation Artifacts: It highlights how standard metrics can be inflated by camera inactivity, necessitating a multi-dimensional evaluation protocol that jointly assesses trajectory control and object permanence.
3. Guiding Future Research: The results suggest that object permanence is not a byproduct of camera conditioning or increased model scale alone. Instead, it requires explicitly targeted training objectives and persistent state representations in model design.

The authors conclude that closing the gap between geometric consistency, perceptual quality, and genuine memory consistency is a core challenge for future world generation models, and MemoBench serves as a necessary tool for tracking progress in this domain.