🤖 AI

WM-Cov: Test Adequacy for Interactive World-Model-Style Autonomous Driving Simulation

This paper introduces WM-Cov, a provider-agnostic evaluation framework that defines and measures testing adequacy for interactive world-model-based autonomous driving simulations by focusing on the convergence of valid, realized, and diverse failure evidence rather than raw failure counts or prompt coverage.

Original authors: Jianxun Cui, Ping Wu, Stanisa Peric, Marko Milojkovic, Vladan Devedzic

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianxun Cui, Ping Wu, Stanisa Peric, Marko Milojkovic, Vladan Devedzic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. In the old days, you would hand the robot a giant library of pre-recorded videos showing every possible traffic situation: a rainy day, a jaywalker, a construction zone. You'd say, "Watch these 10,000 videos, and if you survive them all, you're ready." This is like a video game where the levels are fixed; you know exactly what the enemy will do because the script never changes.

But now, scientists are building "World Models." Think of these not as video libraries, but as living, breathing video game engines. Instead of playing a pre-recorded level, the robot drives in a world that reacts to it. If the robot swerves left, the other cars might swerve right to avoid it. If the robot hesitates, a pedestrian might step out. The world is no longer a fixed script; it's an endless, interactive story where the robot's own choices change the plot. The big question for safety experts is: How do you know when you've tested enough? If the world keeps changing, how do you know you haven't just been lucky, or worse, how do you know the scary crashes you see are real dangers and not just glitches in the game engine?

This is exactly the puzzle tackled by a new paper titled "WM-Cov: Test Adequacy for Interactive World-Model-Style Autonomous Driving Simulation." The authors, a team of researchers from universities in China, Serbia, and industry partners, propose a new way to count and check these tests. They argue that simply generating a million scary-looking videos isn't enough. In fact, in their simulations, they found that many "dangerous" events were actually fake errors or duplicates that shouldn't count toward safety.

To solve this, they created a tool called WM-Cov (World Model Coverage). Imagine you are a food critic reviewing a restaurant that serves an infinite buffet of dishes created by a robot chef. Some dishes look delicious but are actually made of plastic (artifacts). Some are just the same burger you've seen a hundred times (duplicates). Some are real, tasty burgers (valid evidence). If you just count how many plates the chef put out, you might think you've tried everything. But WM-Cov is the critic's notebook that sorts the plates. It separates the "requested" dishes (what you asked for), the "realized" dishes (what actually came out of the kitchen), and the "valid" dishes (the ones that are real, safe to eat, and unique).

The researchers tested this idea using two main methods. First, they looked at a pool of 1,219 simulated driving events. They found that 690 of them were "artifact failures"—basically, the simulator glitched and made a crash look scary, but it wasn't a real driving problem. If you just counted crashes, you'd think the car was terrible. But when WM-Cov filtered them out, it showed that only 230 were real, valid failures. Second, they ran a massive test with a real-world simulation system called DriveArena, involving 360 different driving requests. They discovered that the "reality" of the test depended heavily on who was driving (the planner), how far ahead they were looking (the horizon), and the weather conditions. For example, one driving AI failed to complete 32 out of 90 long-distance requests, while another succeeded every single time.

The paper's main finding is that we can't just look at the number of scary videos a world model generates. Instead, we need a system that tracks valid evidence. The authors suggest that a test campaign is "adequate" (or finished) only when the system has found enough real, unique, and plausible failures to be confident, without wasting time on glitches or repeats. In their tests, their new method, WM-Cov, managed to find 99 valid failures out of 100 selected events while completely ignoring the fake ones, whereas other methods that just looked for "risk" or "coverage" got tricked by the fake data.

Ultimately, the paper suggests that as we move from static video libraries to interactive, living simulations, our safety checks must evolve too. We need to stop counting raw numbers and start auditing the quality of the evidence. It's not about how many times the robot crashes in the simulation; it's about whether those crashes tell us something true and useful about how the robot will behave in the real world. The authors admit this is a new way of thinking that needs more testing across different maps and drivers, but their tool provides a clear, auditable way to stop guessing and start knowing when a self-driving car is truly ready.

Technical Summary: WM-Cov: Test Adequacy for Interactive World-Model-Style Autonomous Driving Simulation

1. Problem Formulation

Traditional autonomous driving (AD) testing relies on scenario libraries and fixed replayed trajectories, where coverage is measured against a curated set of pre-defined cases. However, the emergence of interactive world models (WM) and generative simulators fundamentally shifts the testing substrate. In these systems, the ego planner under test actively influences the future state of the world; background agents, sensor observations, and risk profiles react dynamically to the planner's actions.

Consequently, a "scenario" is no longer a fixed trajectory but an interactive scenario family defined by a map, route, initial state, environment conditions, background-agent policies, prompts, and seed distributions. The realized evolution of the scenario depends on the specific planner being tested.

The core problem addressed is testing adequacy in this interactive setting. The unresolved question is not merely whether dangerous rollouts can be generated, but what constitutes valid closed-loop evidence sufficient to justify stopping a test campaign. A raw count of generated videos, prompts, or collisions is insufficient because:

Generated "failures" may be artifacts (simulator errors, map inconsistencies) rather than valid AD system (ADS) failures.
Generated events may be duplicates or conditioning failures (where the requested prompt was not realized).
High failure counts may concentrate on narrow modes, lacking diversity.

The paper posits that adequacy must be defined by the convergence of valid, interactive evidence under a fixed budget, rather than by raw generation metrics.

2. Methodology: WM-Cov

The authors introduce WM-Cov, a provider-agnostic evaluation layer designed to audit and account for evidence generated by interactive world-model-style simulators. WM-Cov does not modify the generator; instead, it sits above the provider to standardize, audit, and report on the testing process.

A. Core Concepts

WM-Cov distinguishes three levels of coverage:

Requested Coverage ( $C_{req}$ ): The test conditions explicitly asked of the provider (e.g., "rainy night," "unprotected left turn").
Realized Interaction Coverage ( $C_{real}$ ): The conditions actually observed in the closed-loop execution.
Valid Realized Coverage ( $C_{valid}$ ): The subset of realized interactions that are plausible, non-duplicate, and free of artifacts.

The framework separates requested-to-realized tracking (did the prompt work?) from validity accounting (is the result a genuine test case?).

B. The Evaluation Pipeline

Trace Ingestion & Adapter: A standardized adapter schema ingests traces from any provider (neural world models, closed-loop simulators, or log replays). It extracts metadata including the scenario family ID, ego planner, requested conditions, realized interactions, and outcome scores.
Feature Taxonomy: Traces are mapped to feature vectors covering Operational Design Domain (ODD), interaction types, risk bins, behavior classes, failure modes, and validity labels.
Validity Audit: A ledger categorizes traces into:
- Valid ADS Failures: Plausible, replayable failures relevant to the planner.
- Valid Safety-Critical Events: High-risk events without confirmed planner failure.
- Valid Safe Cases: Plausible non-critical cases.
- Conditioning Failures: Requested conditions not realized (e.g., requested rain but got sun).
- Artifact/Invalid Failures: Events caused by generator errors, disconnected map edges, or simulator glitches.
- Duplicates: Repeated failure patterns.
Budgeted Selection: A multi-objective selection policy chooses traces to maximize valid evidence within a budget $B$ $B$ . The selection score $s(\tau|T)$ $s (τ ∣ T)$ balances:
- Marginal gain in valid coverage ( $\Delta C_{valid}$ ).
- Risk exposure ( $R$ ) and realism/validity confidence ( $Q$ ).
- Penalties for redundancy ( $M$ ) and artifact risk ( $A$ ).
Stopping-Oriented Adequacy: The campaign stops when coverage growth, the discovery of new valid failure modes, and the uncertainty of the estimated failure probability converge within a sliding window, provided artifact rates remain low.

C. Adequacy Report

The output is not a single scalar but a report ( $A_B$ ) containing:

Coverage growth metrics ( $C_{req \to real}$ , $C_{valid}$ ).
Counts of Valid Failures ($VF$), Unique Failure Modes ($FM$), and Artifacts ($AI$).
Valid-Evidence Precision: $\Pi_v = VF / (VF + AI)$ .
Mean risk and realism scores.

3. Key Contributions

Problem Formulation: Reframes interactive WM-style testing as an adequacy and stopping problem over scenario families rather than fixed replayed trajectories.
WM-Cov Framework: Introduces a provider-agnostic layer that tracks requested, realized, and valid realized coverage, separating test intent from generated evidence.
Adequacy Metrics: Defines a comprehensive report combining coverage growth, valid failure discovery, failure-mode diversity, artifact suppression, and precision.
Empirical Validation: Evaluates the framework across executed logs (TeraSim/SUMO), mixed trace pools, and a real closed-loop DriveArena matrix involving two planners (UniAD, VAD), two horizons, and six prompt conditions.

4. Experimental Results

The evaluation was conducted in three tiers:

A. Executed TeraSim/SUMO Pool

Dataset: 1,219 event-level traces containing 230 valid ADS failures, 299 valid safety-critical events, and 690 artifact failures.
Finding: A high volume of "dangerous-looking" events were artifacts.
Performance: At a budget of 100, WM-Cov selected 99 valid failures with 0 artifacts, achieving a valid-evidence precision of 1.00. In contrast, "Risk-only" selection found 100 valid failures with 0 artifacts (precision 1.00), while "Coverage-only" selection admitted 58 artifacts for only 17 valid failures (precision 0.23).

B. WM-Like Mixed Trace Pool

Dataset: A synthetic pool emulating world-model outputs with valid failures, duplicates, and artifacts.
Finding: Risk-only selection wasted budget on artifacts.
Performance: WM-Cov selected 76 valid failures with 0 artifacts, covering all 6 observed failure modes. It maintained high mean risk (0.9686) and realism (0.8932). In contrast, "Risk-only" selection found 61 valid failures but admitted 25 artifacts, resulting in a valid-evidence precision of 0.71 ( $61/(61+25)$ ).
Sensitivity: Across 16 weight settings, WM-Cov maintained zero artifact selections and stable coverage (0.4874), indicating robustness.
Stress Test: As artifact contamination increased to 55%, risk-only selection spent 67% of its budget on artifacts, whereas WM-Cov maintained 0% artifact selection.

C. Real DriveArena Provider Matrix

Setup: 360 requested closed-loop attempts (2 planners $\times$ 2 horizons $\times$ 6 prompts $\times$ 15 routes).
Realization Rates:
- VAD: 100% realization at 5.0s horizon; 80% at 7.5s.
- UniAD: 93% realization at 5.0s; 64% at 7.5s.
- Total: 304 fully realized traces, 56 partial realizations, 0 completely unrealized.
Insight: Realization is planner-, horizon-, and prompt-conditioned. For example, UniAD failed to realize the "default cloudy" prompt at the 7.5s horizon. WM-Cov correctly identified these as partial provider evidence rather than planner safety evidence.
Disjoint Slice Check: An 80-request check on a disjoint route slice confirmed that the requested-to-realized accounting behavior persists beyond specific route slices.

D. Stopping Criteria

In a stopping-oriented study on the WM-like pool, WM-Cov reached a stop point at 66 traces with zero artifacts and precision 1.00. Competing methods either stopped early with narrow coverage (Risk-only) or required significantly more traces (109) while admitting artifacts (Coverage-only).

5. Significance and Claims

The paper claims that WM-Cov provides a necessary evaluation layer for the shift from curated scenario libraries to interactive generative testing. Its significance lies in:

Decoupling Generation from Evidence: It asserts that a realistic generator can still produce poor feature coverage or invalid evidence. Therefore, testing adequacy cannot be assumed from the generator's output alone.
Valid Evidence Accounting: It demonstrates that without an audit layer, testing campaigns may overstate safety relevance by counting artifacts and duplicates as failures.
Operational Stopping Rules: It proposes a practical stopping criterion based on the convergence of valid interactive evidence under a budget, rather than arbitrary run counts.
Provider Agnosticism: The framework is designed to work with any provider (neural or rule-based) as long as outputs can be mapped to the trace adapter schema.

The authors explicitly state that the work does not propose a new world model or a universal safety sufficiency claim. Instead, it defines when interactive testing becomes adequate by auditing the transition from "requested" to "realized" to "valid" evidence. The results support the conclusion that dangerous-looking generated events often contain non-evidential artifacts, and that valid closed-loop evidence must be accumulated and verified before a testing campaign can be deemed sufficient.