Technical Summary: GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation

1. Problem Statement

The evaluation of embodied robot foundation models remains a critical bottleneck. Unlike Large Language Models (LLMs), which can be rapidly assessed via digital benchmarks, robotic policies typically require slow, costly, and hardware-limited real-world rollouts. While classical simulation offers a cheaper alternative, it suffers from the "sim-to-real" gap and prohibitive overhead in constructing digital twins.

World models—specifically video generation models conditioned on actions—have emerged as a potential middle ground, capable of simulating future states to serve as surrogate policy evaluators. However, the field lacks a systematic understanding of which design choices make a world model a reliable evaluator. Current literature often demonstrates that world models can be used for evaluation but fails to answer what properties ensure they predict policy quality accurately relative to real-world execution.

2. Methodology

2.1 WMBench: A Benchmark for Policy Evaluation

To address the lack of standardized evaluation, the authors introduce WMBench, a benchmark constructed from real-robot teleoperation data and matched policy rollouts.

• Data Composition: The benchmark comprises 2,989 paired trajectories across eight task families (including rigid and deformable manipulation), featuring a near 1:1 ratio of teleoperated expert data and policy rollout data (successes and failures).
• Evaluation Protocol: The benchmark employs a four-step closed-loop protocol: (1) collecting real-world policy data, (2) training world models on a strict train-test split, (3) executing closed-loop policy rollouts within the learned world model, and (4) assessing metrics and outcomes.
• Metrics: The authors define a hierarchical metric system separating outcome evaluation (World Model as Evaluator Score, WMES) from rollout diagnostics. WMES is a 0–3 ordinal scale assessing whether the generated rollout predicts the correct task outcome (success/failure) and maintains visual/physical fidelity. Diagnostic metrics include visual fidelity, geometry, semantics, and motion dynamics.
• Scale: The study analyzes over 324,000 simulated rollouts from 7 video world models and 4 action representation schemes, augmented by community submissions from the CVPR 2026 GigaBrain Challenge.

2.2 Empirical Analysis of Evaluator Quality

Through extensive ablation studies on WMBench, the authors derive five key insights regarding what makes a world model a good evaluator:

1. Metric Correlation: Visual and geometric fidelity (Subject Consistency, Perspectivity, Instruction Following) are the strongest predictors of evaluator quality (WMES). Conversely, appearance-stability metrics (e.g., Background Consistency) can be misleading, as static videos score high on stability but fail at policy evaluation.
2. Long-Horizon Necessity: Evaluator quality cannot be judged by short-horizon video generation alone. Small state errors compound in autoregressive rollouts, leading to "viewpoint drift" and "object-identity collapse" in generic models.
3. VLM-Assisted Annotation: To scale evaluation, the authors fine-tune a VLM (Qwen3-VL) with token-type-aware loss weighting to predict WMES. This VLM achieves near-perfect agreement with human ratings (87.8% exact agreement), enabling scalable evaluation.
4. Pretraining Priors vs. Scale: Evaluator quality depends more on the compatibility of pretrained physical priors with robot-conditioned rollout than on raw parameter scale. Models trained on robotics/autonomous driving data (e.g., Cosmos-Predict2.5) outperform larger general-purpose models (e.g., LTX 2.3).
5. Data Composition: A balance between broad physical videos (for general world knowledge) and robot-specific data (for embodiment fidelity) yields the best results. Pure robot data can over-specialize the model, while broad physical data improves robustness without sacrificing task relevance.

2.3 GigaWorld-1 Architecture and Design

Based on these insights, the authors propose GigaWorld-1, a world model optimized specifically for policy evaluation.

• Backbone: Built upon the Wan video diffusion backbone (1.3B and 5B variants).
• Unified Control Injection: Instead of standard cross-attention, GigaWorld-1 uses channel-concatenated control maps. This includes End-Effector (EE) pose maps for static head views and Ray maps for dynamic wrist views, ensuring spatial alignment between robot actions and visual generation.
• Hierarchical Memory: To prevent long-horizon degradation, the model employs a hierarchical history buffer with a persistent first-frame anchor and short/mid/long-range temporal memories. This preserves scene identity and object consistency over time.
• Relative RoPE: Uses Relative Rotary Position Embedding to mitigate temporal position drift during autoregressive generation.
• Prompt Interpolation: Uses Spherical Linear Interpolation (SLERP) for smooth semantic transitions between task phases.
• Training Pipeline: A progressive multi-stage pipeline involving:
  1. Robot world foundation model pretraining (flow-matching).
  2. Autoregressive world modeling (AR).
  3. Optional scene-specific LoRA adaptation.
  4. Few-step distillation (DMD2) for inference acceleration.
• Data Curation: Trained on a heterogeneous corpus of ~12,980 hours, including internet physics videos, open-source robot data, egocentric human data, and curated Giga-collected robot demonstrations.

3. Key Results

• Performance: GigaWorld-1-Plus (5B) achieves a normalized average score of 0.6834 on WMBench, outperforming the strongest baseline, Cosmos-Predict2.5 (0.6123), by 11.6%, and Wan 2.2 5B (0.5948) by 14.9%.
• Metric Specifics: GigaWorld-1 leads in evaluator-critical metrics: JEPA Similarity (0.9337), Semantic Alignment (0.8926), and Trajectory Accuracy (0.3561).
• Long-Horizon Stability: In 40-second autoregressive rollouts, GigaWorld-1 maintains significantly higher PSNR and lower FID/FVD compared to generic baselines, avoiding common failure modes like object identity collapse and spectral concentration.
• Closed-Loop Consistency: GigaWorld-1 demonstrates superior alignment with real-world success rates. While baseline models tend to overestimate policy success (optimistic bias), GigaWorld-1's predictions closely follow the real-world diagonal, indicating better calibration of task difficulty and failure modes.

4. Significance and Claims

The paper claims to move the field from "proof-of-concept" demonstrations toward principled design rules for building world models as policy evaluators. Its primary contributions are:

1. Problem Formulation: It establishes "world model as policy evaluator" as a first-class research problem, identifying that the central challenge is preserving policy-relevant causality under iterative rollout, not just generating photorealistic frames.
2. Benchmarking: The introduction of WMBench provides the first large-scale, controlled environment to compare model families, action encodings, and evaluation metrics specifically for policy assessment.
3. Design Roadmap: The paper derives a practical design map emphasizing that action-faithful rollout consistency and long-horizon stability are more critical than short-term visual realism. It highlights the necessity of spatially aligned control interfaces and persistent memory.
4. Open Science: The authors fully release GigaWorld-1 (code, models, datasets, and toolkits) to facilitate scalable evaluation research for embodied foundation models.

The authors maintain a modest tone, acknowledging limitations such as the current scope of task families (excluding mobile manipulation and safety-critical autonomy) and the reliance on video-centric approaches. They position GigaWorld-1 not as a final solution, but as a step toward reliable, reusable evaluator infrastructure that can accelerate the iteration of robot foundation models.