🤖 machine learning

Deterministic World Models for Verification of Closed-loop Vision-based Systems

This paper proposes a Deterministic World Model (DWM) that eliminates stochastic latent variables to map system states directly to images, enabling more precise verification of closed-loop vision-based control systems through tighter reachable sets and rigorous statistical bounds on trajectory deviations.

Original authors: Yuang Geng, Zhuoyang Zhou, Zhongzheng Zhang, Siyuan Pan, Hoang-Dung Tran, Ivan Ruchkin

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuang Geng, Zhuoyang Zhou, Zhongzheng Zhang, Siyuan Pan, Hoang-Dung Tran, Ivan Ruchkin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to prove that a self-driving car is safe. You can't just drive it around a few times and hope for the best; you need a mathematical guarantee that it will never crash, no matter what happens.

The problem is that these cars "see" the world through cameras, which produce massive, complex images (like a 1000x1000 grid of pixels). Traditional safety tools are great at checking simple numbers (like "speed is 50 mph"), but they get completely overwhelmed by the sheer complexity of an image. They try to guess every possible image the camera could show, and their guesses become so wild and loose that they end up saying, "Well, anything could happen, so we can't prove it's safe."

This paper introduces a clever new way to solve this problem. Here is the breakdown using simple analogies:

1. The Problem: The "Magic Box" vs. The "Crystal Ball"

In the past, researchers tried to use Generative AI (like cGANs) to act as a "camera simulator."

The Analogy: Imagine you want to predict what a photo looks like based on where a car is. Old methods used a "Magic Box" that had a hidden, random dial inside it. Even if you told the box the car was in the exact same spot, you could turn the dial to a different random number, and the box would spit out a totally different photo.
The Issue: For safety checks, this randomness is a nightmare. The safety checker has to assume the dial could be turned to any possible setting. This forces them to imagine a photo where the car is driving off a cliff, even if the car is actually on a straight road. Their safety net becomes so huge it covers the whole world, making the check useless.

2. The Solution: The "Deterministic World Model" (DWM)

The authors built a new kind of simulator called a Deterministic World Model.

The Analogy: Think of this as a perfectly predictable 3D printer. If you feed it the exact same coordinates (e.g., "Car is at position X, moving at speed Y"), it will always print the exact same photo. There is no hidden random dial.
Why it helps: Because the output is predictable, the safety checker doesn't have to guess wildly. They can calculate a tight, precise box around exactly what the camera will see. It's like switching from a foggy crystal ball to a laser-guided ruler.

3. The Secret Sauce: "The Control-Consistent Loss"

Training a 3D printer to look like a camera is hard. If the printer makes a photo that looks almost right but moves a tree slightly to the left, a human might not notice. But the self-driving car's brain (the neural network) might see that tree move and decide to swerve, causing a crash.

The Analogy: The authors added a special rule to the training process. It's like a tough coach who doesn't just care if the drawing looks like the real thing; they care if the drawing makes the driver react correctly.
How it works: If the simulator draws a tree in the wrong spot, the coach checks: "Did the driver swerve?" If yes, the coach yells, "Fix it!" This ensures the simulator isn't just pretty; it's behaviorally accurate.

4. The Safety Net: "Conformal Prediction"

Even with a perfect 3D printer, there's a tiny chance the real camera might see something slightly different due to dust, glare, or sensor noise. How do we guarantee safety for the real car, not just the simulator?

The Analogy: Imagine you are building a bridge. You test it with a perfect model. But you know the real wind might be a little stronger than your model predicts. So, you add a safety margin.
The Method: The authors use a statistical trick called Conformal Prediction. They run the real car and the simulator side-by-side many times. They measure the biggest difference between the two paths. Then, they take their calculated "safe zone" and simply inflate it by that maximum difference.
The Result: They can now say with 95% confidence: "If our simulator says the car is safe, the real car is also safe, because we've already added a buffer for the worst-case difference."

Summary of the Win

Old Way: Used a random, chaotic simulator. The safety check was so loose it flagged safe cars as dangerous (false alarms).
New Way: Uses a predictable, "no-randomness" simulator trained to mimic the driver's reactions.
The Outcome: The safety checks are tighter and more accurate. They can prove safety for complex visual systems without getting lost in the noise.

In short, the authors built a predictable, behavior-aware camera simulator and added a statistical safety buffer to it, allowing us to mathematically prove that vision-based robots won't crash, even when they are looking at the world through a camera lens.

1. Problem Statement

The paper addresses the challenge of formally verifying the safety of end-to-end vision-based controllers in safety-critical systems (e.g., autonomous vehicles, robotics). These controllers map raw camera images directly to control actions.

Two fundamental challenges prevent existing formal verification tools from being applied to these systems:

Lack of a State-to-Image Model: Closed-loop verification requires a closed-form mathematical expression for every component in the loop. While physical dynamics are known, there is no analytical equation mapping a physical state (e.g., position, velocity) to a high-dimensional camera image.
Overapproximation Explosion: Even if a surrogate model exists, propagating sets of high-dimensional images through deep convolutional networks causes rapid error accumulation. Existing methods often use stochastic latent variables (e.g., in Conditional GANs) to generate images. These variables introduce uninterpretable, high-dimensional uncertainty, causing the reachable sets to grow so large that the verifier incorrectly flags safe trajectories as unsafe (high false positives).

2. Methodology

The authors propose a framework consisting of three core components:

A. Deterministic World Model (DWM)

Instead of using stochastic generative models with latent variables, the authors propose a latent-free neural decoder that maps physical states directly to synthetic camera images.

Architecture: A state-to-image decoder ( $g_\theta: S \to I$ ) using fully connected layers followed by transposed convolutional layers.
Training Strategy (Dual Loss): To ensure the model is not just visually accurate but also safe for control, the model is trained with a weighted sum of two losses:
1. Pixel-level Reconstruction Loss ( $L_{rec}$ ): Ensures visual fidelity. It uses a weighted Mean Squared Error (MSE) that assigns higher weights to control-relevant regions (e.g., the pole or vehicle) and lower weights to the background.
2. Control-Consistent Loss ( $L_{ctrl}$ ): Penalizes the discrepancy between the control actions generated by the real image and the synthetic image ( $||C(\hat{I}) - C(I)||^2$ ). This ensures the synthetic image preserves the features necessary for the controller to make the correct decision.

B. Star-based Closed-Loop Verification Pipeline

The DWM is integrated into a verification loop using Star sets (a set representation for reachability analysis).

State to Image: The initial state set (represented as a Star set) is propagated through the DWM's transposed convolutional layers to generate an ImageStar (a set of possible images).
Image to Action: The ImageStar is propagated through the CNN controller to produce a set of control actions.
Action to State: The action set is propagated through the nonlinear plant dynamics (using the PyBDR toolbox) to compute the next reachable state set.
Iteration: This process repeats for $T$ time steps to form a reachable tube.

C. Conformal Prediction (CP) for Real-System Guarantees

Since the DWM is a surrogate, it may not perfectly match the real camera. To transfer the verification guarantee to the real system without making distributional assumptions:

Trajectory Deviation: The authors collect pairs of trajectories from the real system and the DWM.
Non-conformity Score: They define the maximum $L_1$ distance between the real and DWM trajectories over the horizon as the deviation score.
Inflation: Using Conformal Prediction, they compute a statistical bound ( $\Delta_{1-\alpha}$ ) on this deviation. The final reachable sets for the real system are obtained by inflating the DWM reachable sets by this bound (Minkowski sum). This ensures that with probability $1-\alpha$ , the real system's trajectory remains within the inflated set.

3. Key Contributions

Deterministic World Modeling: A novel approach to creating verifiable generative surrogates for cameras by removing stochastic latent variables, thereby keeping image sets physically grounded and tractable.
Dual-Loss Training: The introduction of a control-consistent loss that enforces behavioral consistency, ensuring that visual reconstruction errors do not lead to catastrophic control failures.
First Star-based Closed-Loop Verification for Vision Systems: Extending Star-based reachability analysis to handle the full loop: State $\to$ Image $\to$ Action $\to$ State.
Probabilistic Real-System Guarantees: Using Conformal Prediction to rigorously transfer surrogate verification results to the real physical system with high probability, without assuming specific noise distributions.

4. Experimental Results

The method was evaluated on four benchmarks: CartPole, MountainCar, Pendulum, and a Braking System, comparing against a Conditional GAN (cGAN) baseline.

Verification Accuracy: The DWM significantly outperformed the cGAN baseline.
- F1 Scores: DWM achieved F1 scores between 0.92 and 0.98 across all benchmarks, whereas the cGAN baseline struggled (e.g., 0.66 on MountainCar, 0.49 on Braking System) due to excessive overapproximation.
- Precision: The DWM maintained a perfect Precision of 1.0 across all benchmarks, meaning every trajectory certified as safe was genuinely safe.
- Tightness: The DWM produced significantly tighter reachable sets. The cGAN baseline often labeled all initial states as unsafe (Recall = 1.0, but Precision = 0.0 or very low) because the latent variables caused the reachable sets to explode.
Impact of Control-Consistent Loss: Ablation studies showed that adding $L_{ctrl}$ further reduced the Conformal Prediction bounds (trajectory deviation) by 10–20% compared to using only reconstruction loss, directly improving verification tightness.
Real-System Guarantees: After applying Conformal Prediction inflation, the method maintained high performance on stable systems (MountainCar, Braking) while correctly identifying high-risk scenarios in sensitive systems (CartPole) where small errors compound rapidly.

5. Significance

This work bridges a critical gap in the safety certification of AI-driven autonomous systems.

Solves the "Black Box" Camera Problem: It provides a mathematically rigorous way to close the verification loop for image-based controllers, which was previously impossible due to the lack of analytical camera models.
Balances Diversity and Verifiability: It resolves the tension between generative diversity (needed for training) and verifiability (needed for safety) by removing latent variables entirely.
Rigorous Safety Guarantees: By combining deterministic reachability with statistical Conformal Prediction, the framework offers distribution-free, high-probability safety guarantees for real-world systems, moving beyond theoretical surrogates to practical deployment readiness.

The paper concludes that while the current approach struggles with highly complex scenes (cluttered backgrounds), it represents a significant step forward in making end-to-end vision controllers formally verifiable and safe for deployment.