Technical Summary: "How Should World Models Be Evaluated? A Decision-Making-Centric Position"

1. Problem Statement

The term "world model" has proliferated in modern AI, now referring to a heterogeneous set of objects including action-conditioned environment models, latent imagination models, future-video predictors, interactive neural simulators, latent predictive representations, and synthetic-data engines. While this diversity is scientifically productive, it has led to significant ambiguity in evaluation.

The core problem identified is a recurring claim/evidence mismatch. Papers frequently make strong claims about a model's utility for embodied decision-making (e.g., planning, policy optimization, safety testing) while providing evidence primarily focused on lower-level generative artifacts (e.g., video realism, perceptual similarity, instruction following). The authors argue that metrics suitable for evaluating a video generator (visual plausibility) are often insufficient or misleading when the stated goal is a decision-support tool (reliable counterfactual reasoning). A model may generate visually compelling futures that fail to support reliable policy evaluation or optimization under intervention and distribution shift.

2. Methodology and Framework

The paper employs a comprehensive literature survey and proposes a structured evaluation framework centered on the intended use of the model.

2.1 The L0–L7 Evaluation Ladder

The authors organize the literature and evaluation targets into a hierarchical ladder ranging from visual plausibility to policy optimization utility. This ladder serves as a diagnostic tool to distinguish between different types of world models and the strength of evidence they provide for decision-making claims.

• L0 (Visual Plausibility): Does the output look realistic? (Metrics: FID, FVD, aesthetics).
• L1 (Logged-Future Prediction): Does the prediction match held-out trajectories from the behavior distribution? (Metrics: MSE, PSNR, SSIM, LPIPS).
• L2 (Semantic Alignment): Does the rollout match instructions and scene semantics? (Metrics: CLIPScore, VLM judges).
• L3 (Physical Plausibility): Does the rollout obey intuitive physics and geometric constraints? (Metrics: Physics QA, depth consistency).
• L4 (Action Controllability / Counterfactual Fidelity): Does changing the action produce the correct task-relevant changes? This is identified as the first genuinely interventional test, distinguishing a decision world model from a mere future-video prior.
• L5 (Reward, Value, and Outcome Fidelity): Does the model predict success, reward, or value accurately enough for decision-making?
• L6 (Policy Evaluation and Ranking): Does model-based evaluation agree with real/simulator policy performance? (Metrics: Correlation, ranking accuracy).
• L7 (Policy Optimization / Planning Utility): Does using the model actually improve decisions in the real environment or trusted simulator?

The authors posit that L0–L3 are best interpreted as diagnostics of generated artifacts, L4 marks the threshold for genuine interventional testing, and L5–L7 provide the most direct evidence of decision usefulness.

2.2 Decision-Making-Centric Evaluation Framework

The paper proposes a framework where evaluation is conditional on a declared "decision contract" (Task family, Policy class, Action interface, Horizon, Decision use). Key components of this framework include:

• Counterfactual Action Fidelity: Evaluating the model's ability to distinguish $P(o_{t+1:H} | h_t)$ from $P(o_{t+1:H} | h_t, do(a_{t:t+H-1}))$. This involves measuring interventional prediction error (IPE) and action-effect agreement.
• Policy-Induced Distribution Shift: Testing the model under target policies that differ from the behavior policy used for training data, rather than relying solely on open-loop behavior-policy rollouts.
• Full-Horizon Outcome Fidelity: Prioritizing task-relevant outcomes (rewards, success predicates) over short-horizon pixel reconstruction, as small local errors can compound or be irrelevant to the final objective.
• Closed-Loop Rollout Validity: Evaluating the model when the policy acts on model-generated histories, rather than just teacher-forced ground-truth prefixes.
• Exploitability and Uncertainty: Testing whether an optimizer can find trajectories that look good to the model but fail in reality (exploitability gap) and whether the model's uncertainty estimates are calibrated.

2.3 Benchmark Protocol

The authors propose an operational benchmark protocol (Steps 0–8) that includes:

1. Declaring the decision contract.
2. Constructing policy splits (behavior, anchor, shift, optimization).
3. Reporting L0–L3 as auxiliary diagnostics.
4. Evaluating interventional action fidelity (L4).
5. Evaluating closed-loop fixed-policy rollouts (L5–L6).
6. Evaluating policy optimization under fixed budgets (L7).
7. Testing adversarial exploitability and uncertainty calibration.
8. Reporting a "decision-utility profile" (a vector of scores across levels) rather than a single scalar, potentially using gates (e.g., requiring L4–L6 pass/fail) to prevent lower-level metrics from compensating for higher-level failures.

3. Key Contributions

1. Literature Survey and Diagnosis: A paper-by-paper survey of recent world-model literature, organized by what is actually evaluated, identifying the pervasive issue of claim/evidence mismatch where lower-level evidence is used to support stronger decision-making claims.
2. The L0–L7 Ladder: A conceptual framework organizing evaluation targets from visual plausibility to policy optimization utility, arguing that for decision-making claims, L4 is the minimal interventional requirement and L5–L7 provide the strongest evidence.
3. Decision-Making-Centric Framework: A proposal for a new evaluation paradigm that foregrounds counterfactual action fidelity, closed-loop rollout validity, reward/value prediction, policy-ranking agreement, optimization lift, model exploitability, and uncertainty calibration.
4. Benchmark Protocol: A modular, operational protocol for evaluating world models intended for embodied decision-making, including specific recommendations for policy splits, intervention tests, and reporting standards (e.g., the "World-Model Evaluation Card").

4. Results and Findings

The paper does not present new experimental results or a new model. Instead, its "results" are analytical findings derived from the survey of existing literature:

• Divergent Evaluation Cultures: The literature contains multiple evaluation cultures (benchmark/diagnostic, environment-model/policy-optimization, policy-evaluation/executability, latent/foundation) that are often collapsed under the single term "world-model evaluation."
• Mismatch Prevalence: Many papers evaluate lower-level artifact properties (L0–L3) but discuss them as if they establish stronger decision-making claims (L6–L7). Conversely, some latent models (e.g., V-JEPA) are useful for planning without producing photorealistic pixels, challenging the necessity of high-level visual metrics.
• Insufficiency of Current Metrics: Metrics like FVD, PSNR, and VLM-based physical QA are useful for debugging and interfaces but are often poor proxies for policy utility, ranking, or optimization lift. A model can look strong visually while being a poor environment model for control.
• Necessity of Intervention: For embodied decision-making, observational metrics (L1–L3) are insufficient; interventional metrics (L4+) are required to verify that the model correctly models the consequences of actions.

5. Significance and Claims

The paper's significance lies in its call for a shift in how world models are evaluated, specifically for those claimed to support embodied decision-making.

• Conditional Claim: The authors do not claim that every system called a "world model" must be judged by policy optimization. If the intended use is future-video generation, visual quality is the primary target. However, when a model is presented as a tool for embodied decision-making (planning, policy optimization, safety), the evidence required changes fundamentally.
• Primary Argument: For decision-making claims, action-, outcome-, and policy-level evaluations (L4–L7) provide stronger evidence than artifact quality alone (L0–L3). Lower-level metrics should not be allowed to compensate for failure at higher levels.
• Practical Impact: The proposed framework aims to prevent the optimization of "beautiful but strategically unreliable" models. It encourages the community to declare clear decision contracts and to prioritize evidence of counterfactual reasoning, policy ranking, and optimization lift over visual fidelity.
• Modesty: The authors acknowledge that full interventional evaluation (e.g., exact reset-matched branches) is difficult on real robots and that lower-level metrics may correlate with higher-level utility in some domains. Their proposal is presented as a modular, aspirational protocol to clarify evidence rather than a rigid set of universal axioms.

In conclusion, the paper argues that the field has been evaluating several different objects under one name. The L0–L7 ladder and the proposed decision-making-centric framework serve to separate these objects and define the specific evidence required to validate a model's utility for decision-making.