WorldPack: Dynamic Frame Compression for Long-context Video World Modeling

Technical Summary: WorldPack: Dynamic Frame Compression for Long-context Video World Modeling

Problem Statement
Video world models, which generate high-fidelity future visual observations conditioned on past observations and navigation actions, face a critical bottleneck in long-horizon consistency. Existing approaches struggle to maintain both temporal and spatial coherence over extended rollouts. Current methods generally fall into two categories, both of which have limitations:

1. Generic Compression: Methods like FramePack compress past frames based on temporal proximity or feature similarity. However, these schedules do not explicitly exploit 3D viewpoint geometry, potentially discarding spatially critical information from the distant past.
2. Spatial Retrieval: Methods like WorldMem and Context-as-Memory select frames based on 3D co-visibility or field-of-view (FoV) overlap. While spatially aware, these operate within a fixed context window, replacing less relevant frames entirely rather than retaining them at reduced resolution. This creates a trade-off where including a spatially relevant distant frame necessitates discarding other potentially useful observations.

Consequently, models either fail to access necessary distant history or suffer from error accumulation and drift when forced to rely on limited context.

Methodology: WorldPack
The authors propose WorldPack, a video world model that unifies frame selection and compression into a single mechanism: spatially-aware compressed memory. The core insight is that compression rates should not be uniform or temporally determined but dynamically allocated based on 3D spatial relevance to the current viewpoint.

WorldPack achieves this through two tightly coupled mechanisms:

1. Geometric Selection:

   • Instead of relying on temporal recency, the model scores historical frames based on their 3D spatial overlap with the current view.
   • Using camera pose information ($p$), the system calculates the Field-of-View (FoV) overlap between a historical frame $i$ and the target frame $t$. The overlap score $o_i$ is estimated via Monte Carlo sampling within the viewing frustum.
   • A final score $s_i$ is computed by combining the overlap score with a mild temporal penalty to break ties, favoring recent frames only when spatial overlap is similar.
   • Frames are sorted by this score. The top-$S$ frames (most spatially relevant) are assigned to "uncompressed slots," while the remaining frames are assigned to discrete compression groups based on their rank.

2. Trajectory Packing:

   • Historical frames are encoded at varying resolutions determined by their assigned priority.
   • High-priority frames (high spatial relevance) are preserved at full resolution ($L_f$). Lower-priority frames are compressed by factors of $\lambda^{d_i}$ (where $\lambda=2$ and $d_i$ is the priority index).
   • This allows the model to pack a significantly larger number of historical frames (e.g., 22 frames) into a fixed-length context budget (equivalent to 4 uncompressed frames) without discarding any historical information.

Architecture
WorldPack is built upon a Conditional Diffusion Transformer (CDiT) backbone with Rotary Position Embeddings (RoPE).

• CDiT: Restricts self-attention to the target frame tokens and uses cross-attention for past frames, reducing computational complexity from quadratic ($O(m^2)$) to linear ($O(m)$) with respect to context length $m$.
• RoPE: Ensures consistent temporal representations regardless of the variable distances of memory frames in the context.
• Implementation: The system uses a discrete set of compression rates ($2^0, 2^2, 2^4$) with independent input projection layers for each rate. In experiments, the context holds 2 uncompressed frames, 4 at $2^2$ compression, and 16 at $2^4$ compression, totaling 22 frames within the token budget of 4.

Key Contributions

1. Unified Mechanism: WorldPack bridges the gap between dynamic compression and spatial retrieval by treating them as a single problem: allocating token budgets based on 3D spatial relevance.
2. Expanded Effective Context: The method expands the effective context from 4 to 22 frames with moderate computational overhead, allowing the model to reason over substantially longer horizons.
3. Spatially Adaptive Compression: It demonstrates that preserving high fidelity for spatially critical frames while aggressively compressing less relevant ones outperforms both uniform compression and fixed-window retrieval.

Experimental Results
The authors evaluated WorldPack on LoopNav (a Minecraft benchmark for long-horizon spatial consistency) and the RECON dataset (real-world navigation).

• LoopNav Performance: WorldPack outperformed strong baselines (Oasis, Mineworld, DIAMOND, NWM) across both Spatial Memory Retrieval (ABA) and Spatial Reasoning (ABCA) tasks.
  • It achieved superior performance in SSIM, LPIPS, PSNR, DreamSim, and FVD metrics.
  • Gains were particularly pronounced in the ABCA task (longer return paths), where the ability to recall distant observations is critical.
• Ablation Studies:
  • Comparing WorldPack against "Nearest Frame Packing" (compression without geometric selection) isolated the benefit of Geometric Selection, showing that 3D-aware importance scoring is superior to temporal proximity.
  • Comparing against "Memory Retrieval" (selection without compression) isolated the benefit of Trajectory Packing, showing that retaining more frames via compression is superior to a fixed context window.
• Real-World Data: On the RECON dataset, WorldPack demonstrated strong generative performance, confirming its utility beyond simulated environments.
• Efficiency: The method increases diffusion-model inference time by approximately 16% and memory usage by 12% to achieve a 5.5x increase in visible historical frames. The geometric selection cost scales linearly with the number of candidates (e.g., 0.26s for 1,600 candidates on an H100).

Significance and Claims
The paper claims that WorldPack addresses the fundamental trade-off between context length and computational cost in video world modeling. The authors argue that simply expanding context length is less effective than intelligently compressing a larger history with spatial guidance. The key contribution is not merely the spatial scoring itself, but the use of that scoring to control compression rates across a larger set of frames. This approach allows the model to maintain high-fidelity access to spatially critical distant observations while retaining the broader context necessary for coherent long-term simulation. The authors note that while the method relies on camera poses (which may be noisy in real-world settings), it represents a significant step toward robust, long-horizon video world modeling.