Technical Summary: PACT-WAM

Problem Statement

Robot manipulation policies require two forms of temporal information: history to provide context for motion and prior interactions, and visual foresight to predict state changes and assess proposed actions. While World-Action Models (WAMs) effectively connect history, actions, and future observations, they face a significant computational bottleneck. Dense representations of past and future observations incur substantial processing costs; for instance, encoding 16 history frames at 64 tokens each requires 1,024 tokens per view, and multi-step forecasts add separate visual-latent sequences. Existing methods often address these challenges in isolation—either compressing history, using modular visual foresight with separate controllers, or omitting future generation during inference. The core challenge is to design compact representations that preserve historical coverage and provide independently decodable states at each transition, while establishing a protocol to use these forecasts to guide execution without prohibitive computational overhead.

Methodology: PACT-WAM

The authors propose PACT-WAM (Predicting Actions and Visual Foresight with Compact Temporal Encoding), a world-action model that jointly generates a 16-step action trajectory and its temporally corresponding multi-view visual forecast through conditional flow sampling. The architecture relies on three key design choices:

1. Hierarchical History Encoding

Instead of uniform encoding, PACT-WAM employs recency-based history allocation. It assigns coarse spatial representations to older observations and finer representations to recent ones.

• Mechanism: Using a frozen DINOv3 ViT-B/16 backbone, the model processes 16 history frames. The oldest 8 frames are compressed to 4 tokens per view, the middle 6 to 16 tokens, and the latest 2 to 64 tokens.
• Efficiency: This retains all 16 observations using only 256 tokens per view, a 75% reduction compared to dense encoding (1,024 tokens) while maintaining high temporal coverage.
• Integration: These compressed features are merged and projected into the Qwen3-VL-4B language model to form a shared context $h_t$ alongside the task instruction.

2. Joint Action-Visual Flow Generation

PACT-WAM utilizes a shared flow transformer with transition-wise causal attention to jointly update continuous action and visual states.

• Visual Latents: Future images are represented by continuous TiTok-VAE latents ($K=32$ positions per image) without temporal downsampling. This ensures every action is paired with an independently decodable subsequent visual state.
• Shared Backbone: The transformer receives noisy action and visual states, a flow-time embedding, and the fixed context. It uses a block-causal mask where queries in block $j$ (representing the $j$-th transition) can attend to all positions in blocks $k \leq j$.
• Dual Heads: Two modality-specific velocity heads ($g_a$ for actions, $g_v$ for visuals) predict velocities in normalized spaces. The modalities exchange information during sampling through their shared dependence on the noisy states within the permitted temporal prefix.
• Training: The model is trained using conditional flow matching with a linear-path velocity regression objective, optimizing the compression branches, context pathway, flow transformer, and output heads while keeping the DINOv3 and TiTok-VAE encoders/decoders frozen.

3. Proposal Review (PR)

To guide execution, PACT-WAM introduces a Proposal Review mechanism using a Vision-Language Model (VLM).

• Process: Four parallel VLM requests assess nested forecast prefixes at 4, 8, 12, and 16 steps. They evaluate task-consistent progress and detect visible failures (e.g., misalignment, collisions).
• Execution Logic: The controller selects the largest consecutively approved prefix below any reported veto. If a proposal is rejected, the system performs joint resampling (up to three retries) within a bounded budget. If all retries fail, the episode terminates.

Key Contributions

1. Recency-Based History Allocation: The paper introduces a strategy that allocates 256 tokens per view by prioritizing recent frames. On the LIBERO benchmark, this outperforms uniform encoding of the same 16 frames (98.6% vs. 87.5% success) and achieves performance comparable to dense encoding (98.0%) with 75% fewer history tokens.
2. Joint Generation in Compact Latent Space: PACT-WAM jointly generates action trajectories and visual states in a compact per-image latent space, providing an independently decodable forecast at each physical transition. This enables the Proposal Review mechanism to validate execution prefixes, building upon unified world-action frameworks by integrating compact history encoding and proposal review.
3. Performance with Test-Time Enhancement: The base policy achieves high success rates across simulation and real-world benchmarks. The addition of Proposal Review provides a significant test-time boost, improving robustness without retraining the core policy.

Experimental Results

The model was evaluated on LIBERO, RoboTwin 2.0, and a real-world AgileX Piper platform.

• LIBERO (Simulation):
  • Without PR: 98.6% average success.
  • With PR: 99.5% average success (reaching 100% on Object and Goal suites).
• RoboTwin 2.0 (Simulation):
  • Without PR: 92.3% average success across 50 bimanual tasks.
  • With PR: 93.4% average success.
• Real-World Piper:
  • Without PR: 78.0% average success across Sorting, Handover, and Cleanup tasks.
  • With PR: 86.7% average success.
• Ablation Studies:
  • History: The 8:6:2 token hierarchy significantly outperforms uniform encoding and dense encoding in terms of efficiency and success.
  • Joint Generation: Jointly training actions and visuals improves control success (98.6%) compared to action-only (93.6%) or visual-auxiliary (96.4%) variants.
  • Visual Capacity: Increasing latent capacity from $K=32$ to $K=64$ improves forecast fidelity (PSNR, SSIM) but slightly reduces control success (97.3% vs. 98.6%), suggesting a trade-off where the default $K=32$ balances performance and computational cost.
  • Proposal Review: PR effectively filters out failed trajectories, with predicted previews reducing false rejection rates compared to using current observations only.

Significance and Claims

The paper claims that PACT-WAM successfully addresses the trade-off between representation cost and the availability of visual foresight for execution decisions. By combining hierarchical history encoding with compact, decodable visual latents, the model enables joint sampling of temporally paired actions and forecasts. This architecture allows for a Proposal Review protocol that guides execution by validating nested prefixes, thereby enhancing robustness in both simulation and real-world settings.

The authors emphasize that their approach preserves the necessary temporal information for action selection while drastically reducing the token count required for history. They note that while the base policy is competitive, the integration of visual foresight with a VLM-based review mechanism offers a distinct pathway for improving reliability in complex manipulation tasks. The work highlights that higher forecast fidelity does not always correlate with higher control success, suggesting that the utility of the forecast for decision-making is more critical than raw reconstruction quality.

Limitations noted by the authors: The current system uses a fixed recency-based allocation that does not adapt to task relevance, a fixed 16-step horizon, and a review process that may miss brief failures between checkpoints. Future work is suggested to explore task-adaptive compression and variable-horizon generation.