⚡ electrical engineering

Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC

The paper introduces SLS², a framework that combines action-conditioned latent world models with conformal prediction-enhanced robust model predictive control to enable safe, vision-based motion planning with probabilistic safety guarantees.

Original authors: Devesh Nath, Anutam Srinivasan, Haoran Yin, Ruitong Jiang, Jeffrey Fang, Glen Chou

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Devesh Nath, Anutam Srinivasan, Haoran Yin, Ruitong Jiang, Jeffrey Fang, Glen Chou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to perform a delicate task, like tying a knot in a rope or stacking a block, but the robot only has a camera. It can't see the "true" state of the world (like exact angles or distances); it only sees a stream of raw images.

The paper presents a new method called SLS2 (Safe Latent Space System Level Synthesis) that helps robots plan safe movements directly from these camera images, even when the robot's internal "guess" about how the world works is imperfect.

Here is how it works, broken down into simple concepts and analogies:

1. The "Dream World" (Latent World Model)

Robots usually need a perfect map of the world to plan. But in the real world, getting that map is hard.

The Analogy: Imagine the robot closes its eyes and starts "dreaming" a simplified version of the world. Instead of processing millions of pixels, it creates a compact, abstract "dream state" (a latent space) that captures the essence of the image.
What the paper does: They train the robot to learn this dream world. They teach it to predict what the next "dream" will look like after it makes a move. This allows the robot to simulate thousands of future scenarios in its head very quickly, without needing to actually move the robot.

2. The "Imperfect Dreamer" (Prediction Errors)

The problem is that the robot's dream isn't perfect. Sometimes it predicts the block will move left, but it actually moves right.

The Analogy: It's like a weather forecaster who is usually right but occasionally gets the wind direction wrong. If you plan a picnic based only on their forecast, you might get soaked.
The Paper's Solution: Instead of ignoring these mistakes, SLS2 admits them. It uses a statistical tool called Conformal Prediction to draw a "safety bubble" around the robot's predictions.
- Think of this bubble as a fog of uncertainty. The robot knows, "I think the block is here, but because I might be wrong, the block could actually be anywhere inside this foggy bubble."

3. The "Safety Bubble" (Robust Planning)

Most planning methods try to find the perfect path through the dream world. SLS2 does something smarter: it finds a path that stays safe even if the robot's dream is slightly off.

The Analogy: Imagine driving a car through a thick fog. A normal driver might hug the center line, hoping they are right. A SLS2 driver drives in the middle of the lane but keeps a wide buffer zone on both sides. Even if the car drifts slightly (due to the fog/error), it won't hit the wall.
How it works: The system calculates "reachable tubes." These are 3D tunnels that represent all the places the robot could end up given its uncertainty. The planner ensures that the entire tunnel stays within the "safe zone" and avoids "obstacle zones" (like hitting a wall or dropping a block).

4. The "Safety Inspector" (Conformal Obstacle Checker)

The robot also needs to know what an "obstacle" looks like in its dream world.

The Analogy: Imagine a bouncer at a club. In the real world, the bouncer checks IDs. In the robot's dream world, the bouncer checks the "dream state."
The Paper's Solution: They train a special "bouncer" (a neural network) that looks at the dream state and says, "Safe" or "Unsafe." Crucially, they use the same statistical "fog" method to calibrate this bouncer. This ensures the bouncer doesn't get too lenient; if there's a 99% chance a state is unsafe, the bouncer treats it as unsafe.

5. The "Super-Speed Planner" (GPU Acceleration)

Calculating all these safety bubbles and tunnels for a high-dimensional dream world is usually very slow.

The Analogy: It's like trying to calculate the trajectory of a million different cars at once. Doing it one by one would take forever.
The Paper's Solution: They use System Level Synthesis (SLS), a mathematical trick that allows them to run these calculations in parallel on a powerful computer chip (GPU). This is like having a million calculators working at the same time, making the planning fast enough to happen in real-time.

What They Tested It On

The authors tested this system on several tasks, showing it works better than previous methods:

The Reacher: A robot arm trying to touch a target without bending its joints too far (a safety constraint).
The Cube: A robot arm picking up a cube and moving it without lifting it too high (which would be unsafe).
The Rope: A complex task where two robot arms manipulate a floppy rope. This is very hard because the rope can twist in infinite ways.
Hardware: They even tested it on a real robot arm in the lab, not just in a computer simulation, successfully stretching a rope from a start position to a goal position.

The Bottom Line

SLS2 is a framework that lets robots plan safe, complex movements directly from camera images. It does this by:

Learning a simplified "dream" of the world.
Calculating exactly how wrong that dream might be (the "fog").
Planning a path that stays safe even if the dream is wrong (the "safety bubble").
Doing all this fast enough to run on real robots.

The result is a robot that is much less likely to crash or break things, even when it doesn't have perfect information about the world.

Technical Summary: SLS2 – Probabilistically-Safe Latent World Model Control

Problem Statement

The paper addresses the challenge of safe feedback motion planning for robots operating directly from raw pixel observations (e.g., RGB images) in open-world environments. Traditional safety-critical control methods rely on well-defined, low-dimensional state representations and known dynamics, which are often unavailable or difficult to estimate for deformable objects and complex visual tasks. While recent latent world models (e.g., JEPA architectures) can infer compact state representations and dynamics directly from images, they introduce significant safety risks:

Imperfect Dynamics: Learned latent dynamics are approximations and may be inaccurate over multi-step rollouts.
Constraint Translation: Physical safety constraints (e.g., obstacle avoidance, joint limits) must be translated into the latent space, where the relationship to physical safety is not guaranteed.
Lack of Calibration: Existing latent safety methods often rely on heuristic approximations or lack formal, calibrated guarantees that the true system state remains within safe bounds despite prediction errors.

The core problem is to synthesize a control policy that plans in a learned latent space while providing probabilistic safety guarantees for the true physical system, accounting for both model prediction errors and distributional shifts.

Methodology: SLS2 (Safe Latent Space System Level Synthesis)

The authors propose SLS2, a framework combining learned latent world models with System Level Synthesis (SLS) robust Model Predictive Control (MPC) and Conformal Prediction (CP). The pipeline consists of three main stages:

1. Training a Gradient-Friendly Latent World Model

The system learns a compact, Markovian latent representation from an offline dataset of image transitions $\mathcal{D} = \{(o_t, u_t, o_{t+1})\}$ .

Architecture: An action-conditioned joint-embedding predictive architecture (similar to Le-WM). A Vision Transformer (ViT) encoder maps images to latent embeddings $z_t$ .
Markov State Construction: To ensure the latent state satisfies the Markov property for multi-step prediction, the state is augmented with finite-difference features: $s^z_t = [z_t, \Delta z_t, \dots, \Delta^K z_t]$ .
Dynamics Learning: An MLP predictor $f_\phi$ learns the transition $s^z_{t+1} = f_\phi(s^z_t, u_t)$ .
Training Objectives: The model is trained using a multi-step latent prediction loss, a SIGReg loss to prevent representational collapse, and a temporal straightening loss to encourage smooth latent trajectories. This design facilitates efficient gradient-based trajectory optimization (e.g., iLQR).

2. Conformal Uncertainty Quantification

To handle the inaccuracies of the learned dynamics and the latent constraint checker, the authors use Split Conformal Prediction (CP) to calibrate uncertainty sets with statistical guarantees.

Latent Dynamics Error: A covariance model $\Sigma_\psi$ is trained to predict the shape of the error distribution. CP is used to compute a scaling factor $q$ (a quantile of non-conformity scores) such that the true prediction error $e_t$ lies within an ellipsoidal set $E(s, u)$ with probability $1-\delta$ .
Latent Constraint Checker: A latent obstacle classifier $c_\psi$ is trained to score safety. CP is used to determine a threshold $\eta$ such that the classifier correctly identifies safe/unsafe regions with high probability.
In-Domain Support: An in-domain set $\mathcal{I}$ is defined as a calibrated Gaussian sublevel set to ensure the planner does not operate in regions far outside the training distribution.

3. Robust MPC via System Level Synthesis (SLS)

The core control loop uses System Level Synthesis (SLS) to solve a robust MPC problem in the latent space.

Formulation: Instead of optimizing feedback gains directly, SLS optimizes over closed-loop system responses (matrices $\Phi_s, \Phi_u$ ) that map disturbances to state/control deviations.
Robustness: The CP-calibrated ellipsoidal error sets $E(s, u)$ are injected as disturbance sets into the SLS formulation. The optimizer propagates these uncertainties over the planning horizon to construct reachable tubes ( $R^s_k, R^u_k$ ).
Constraints: The optimization enforces that the entire reachable tube remains within the safe latent set (defined by the calibrated constraint checker) and the in-domain set. This results in "constraint tightening," where the nominal trajectory is adjusted to ensure that even under worst-case bounded errors, the true system remains safe.
Implementation: The approach leverages GPU acceleration to handle the high-dimensional latent states and the computational load of propagating uncertainty tubes efficiently.

Key Contributions

Latent World Model for Planning: The authors demonstrate that training compact, Markovian latent models with specific regularization (SIGReg, temporal straightening) enables accurate multi-step rollouts and efficient gradient-based optimization directly from pixels.
Conformalized Uncertainty Sets: The paper introduces a method to calibrate both latent dynamics errors and latent safety classifier errors using Conformal Prediction, providing statistically valid uncertainty sets for robust planning.
Scalable Robust MPC: The integration of CP-calibrated uncertainty into GPU-accelerated System Level Synthesis allows for reachability-constrained robust MPC in high-dimensional latent spaces, a task previously considered computationally prohibitive.
Empirical Validation: The method is validated on diverse vision-based tasks (navigation, pick-and-place, deformable object manipulation) in both simulation and real-world hardware, demonstrating superior safety and goal-reaching performance compared to baselines.

Experimental Results

The authors evaluated SLS2 on four tasks: Reacher, OGBench Cube, Push-T, and a Bimanual Rope manipulation task.

Nominal Planning Performance:
- SLS2's nominal planner (using iLQR on the learned model) significantly outperformed baselines (Le-WM, DINO-WM, PLDM) in success rates.
- Reacher: 83.5% success (vs. 17–40% for baselines).
- Rope: 93.75% success (vs. 7–25% for baselines).
- Ablation studies confirmed that the Markovian state representation and multi-step training losses were critical for performance.
Robust Safety Performance:
- In constrained tasks (e.g., joint angle limits, height thresholds, obstacle avoidance), SLS2 achieved 100% safety rates in simulation and hardware, whereas non-robust constrained iLQR and other safety filters (HJ-filtered, LPB) frequently violated constraints.
- Robust Success: SLS2 maintained high task completion rates (e.g., 88.57% on Rope, 94.29% on Cube) while strictly adhering to safety constraints.
- Hardware: The method was successfully deployed on real KUKA iiwa arms for rope manipulation, demonstrating convergence to the goal state directly from pixels without explicit state estimation.

Significance and Claims

The paper claims that SLS2 bridges the gap between the flexibility of learned world models and the rigor required for safety-critical control. By combining conformal prediction for statistical calibration and system level synthesis for scalable robust optimization, the method provides probabilistic guarantees that the true system state remains within safe bounds, even when operating in a high-dimensional latent space with imperfect dynamics.

The authors emphasize that their approach does not require hand-designed state representations or known dynamics, making it applicable to complex, deformable, and visually rich tasks. They note that while the method improves safety and success rates significantly, it relies on the assumption that closed-loop residual scores are exchangeable with calibration scores, and performance can be bottlenecked by the assumption of isotropic latent representations in tasks requiring complex sequential maneuvers (like Push-T).