💻 computer science

Certifying Plans under Model Mismatch: A Trilemma for Reachability from Scarce Data

This paper establishes a fundamental trilemma in certifying control plans under model mismatch with scarce data, demonstrating that any sound deterministic certifier must either sacrifice trajectory containment, accept arbitrarily large uncertainty, or restrict model-error behavior, and proposes the ForeReach method to construct set-membership envelopes and zonotopic tubes that safely decline certification when data support is insufficient.

Original authors: Yanliang Huang, Zhen Zhang, Ahmad Hafez, Wenyuan Wu, Peng Xie, Zhuoqi Zeng, Amr Alanwar

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanliang Huang, Zhen Zhang, Ahmad Hafez, Wenyuan Wu, Peng Xie, Zhuoqi Zeng, Amr Alanwar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car. You can't just let it loose on the real highway immediately; that would be dangerous. Instead, you train it in a video game simulator. The simulator is great, but it's not perfect. The physics in the game might be slightly "off" compared to the real world—maybe the tires grip a little differently, or the wind pushes the car in a way the game didn't predict. This difference between the game world and the real world is called "model error."

Now, imagine the robot has learned a cool new driving maneuver in the game. Before you let it try this move on a real street, you need a safety inspector to check: "If the robot does this, will it crash?" The inspector has a map of the simulator's rules, but they only have a few scattered notes about how the real car actually behaves. These notes are like isolated snapshots: "At this specific speed and turn, the car went here." The big question is: Can we guarantee the robot is safe using just those few snapshots, even if the robot tries a move in a place where we have no notes at all? This paper tackles that exact puzzle, exploring the limits of what we can know when we have very little data about the real world.

The researchers behind this study, working with the Technical University of Munich and Hainan Bielefeld University, discovered a fundamental "trilemma"—a three-way standoff where you can't have everything you want. They proved that if you try to certify a robot's plan using only a few scattered data points, you face three impossible choices:

Be absolutely safe (Sound): Your safety guarantee must cover every possible way the real world could behave.
Be useful (Informative): Your safety zone shouldn't be so huge that it covers the entire universe; it needs to be tight enough to tell you if the robot will actually hit a wall.
Be open-minded (Unrestricted): You shouldn't make up rules about how the real world behaves in places where you haven't looked yet.

The paper proves that you can only pick two of these three. If you refuse to make up rules about the unknown (keeping the model "unrestricted"), and you demand to be absolutely safe, your safety zone becomes infinitely wide. It's like a security guard who, seeing a few footprints in the sand, decides that to be 100% sure no one sneaked in, they must assume the intruder could be anywhere in the entire city. That's a safe assumption, but it's not very helpful for catching the thief.

To solve this, the authors propose a new method called ForeReach. Instead of guessing blindly, ForeReach asks for a "side note" from the robot's designers: a promise about how fast the real world's behavior can change. Think of it like a speed limit for the robot's confusion. The designers must declare, "The difference between our game and reality can't change faster than this amount." ForeReach then checks if the few data points we have agree with this promise. If they do, it builds a "tube" (a safe path) around the robot's plan.

Here is how it works in practice:

The Check: It looks at the scattered data points. If two points suggest the "speed limit" of confusion is broken, it immediately says, "Nope, your promise is wrong," and stops.
The Tube: If the promise holds, it draws a safe tube around the robot's path. This tube gets wider the further the robot travels from the known data points, because the uncertainty grows.
The Verdict: If the tube stays inside the "safe zone" and avoids obstacles, the robot gets a green light. But if the robot tries to drive into a region where we have no data and the tube gets too wide (or hits a wall), ForeReach politely refuses to certify the plan. It says, "I can't guarantee safety here because I'm flying blind."

The researchers tested this on two systems: a simple point-mass robot and a more complex "dynamic bicycle" (a model of a car). They compared ForeReach against other methods that try to guess the safety zone using statistics or global rules. The results were clear: other methods often gave a "green light" to dangerous plans just because they didn't know better, leading to crashes in simulation. ForeReach, however, was honest. When the robot tried to drive into an area with no data, ForeReach said, "I can't certify this," and refused to give a certificate. But when the robot stayed in areas where they had data (or where the "speed limit" promise was tight enough), ForeReach successfully certified the plan with a very narrow, useful safety tube.

In one specific test with the bicycle model, when the robot tried to drive into an unsupported area, other methods claimed the path was safe 42% of the time (at low data counts), but those paths were actually unsafe. ForeReach, on the other hand, correctly abstained from certifying those dangerous paths 100% of the time. When they added more data points along the path the robot actually took, ForeReach's "certified recall" (how often it correctly said "yes, this is safe") jumped from 49% to 98%.

The paper doesn't claim to have solved the problem of robot safety forever. Instead, it draws a hard line in the sand: you cannot have a safety guarantee that is both perfectly reliable and usefully precise without some extra information about how the world behaves. You can't just rely on a few data points to tell you everything. You need a "declaration" of limits from the designers. If that declaration is true, ForeReach works beautifully. If the declaration is wrong, the method can sometimes catch the error, but it can't always prove the declaration is right just by looking at the data.

Ultimately, this work teaches us that in the world of robotics, "more data" isn't always the magic bullet. Sometimes, the most important thing is knowing what kind of rules the world follows, even in the places we haven't looked yet. Without those rules, the best safety inspector can only say, "I don't know," and that, the authors argue, is the only honest answer.

Technical Summary: Certifying Plans under Model Mismatch: A Trilemma for Reachability from Scarce Data

Problem Statement
The paper addresses the challenge of certifying fixed control sequences (e.g., action chunks from learned policies like Diffusion Policy or $\pi_0$ ) for deployment on real-world robotic systems where the dynamics differ from the nominal simulation model. The core difficulty arises when the target system provides only a sparse set of isolated one-step transitions. In this setting, a certifier must determine if a planned trajectory will remain safe (avoiding an unsafe set $O$ ) despite the unknown model error (residual) between the nominal model $f_0$ and the true target dynamics $f^\star$ .

The authors identify a fundamental limitation: away from the observed data points, the model error can vary arbitrarily within a small neighborhood without contradicting the observations. If the model error is unrestricted, two smooth target systems consistent with the same sparse data can diverge arbitrarily once the plan enters an unobserved region. Consequently, any deterministic certifier that is uniformly sound (guaranteeing safety for all consistent systems) must either decline to certify or return a reachable tube with an arbitrarily large projected width, rendering the certificate uninformative.

Methodology: ForeReach
The authors propose ForeReach, a pre-execution certifier that navigates this limitation by incorporating side information. The method operates under the following framework:

Side Information Requirement: ForeReach requires a supplied component-wise Lipschitz bound ( $L$ ) on the model error over a prescribed certification domain ( $Z_{cert}$ ). This bound must be established independently of the sparse data (e.g., via analytic dynamics with bounded parameters or a separately certified residual model).
Pairwise Consistency Test: Before propagation, the method checks if the observed residual pairs contradict the supplied Lipschitz bound. If $|e_{r_i, k} - e_{r_j, k}| > \sum L_{k\ell}|z_{i,\ell} - z_{j,\ell}| + 2\sigma_k$ , the declaration is rejected, and the certifier abstains.
Set-Membership Envelope: Conditional on the bound holding, the method constructs a set-membership envelope for the residual at each queried state-input set. This envelope is derived by intersecting the observation-wise bounds extended via the Lipschitz constant (McShane extension) and the observation noise bounds.
Zonotopic Propagation: The uncertainty is propagated forward using the nominal model $f_0$ . The reachable set is represented as a zonotope, incorporating the nominal map enclosure (using interval Hessian bounds for nonlinear remainder terms) and the residual envelope.
Certification Logic: The certifier issues a safety certificate only if:
- The propagation remains within the certification domain ( $Z_{cert}$ ).
- Every projected slice of the reachable tube avoids the unsafe set ( $O$ ).
- The method distinguishes between three types of abstention: pairwise contradiction ( $\perp_F$ ), domain exit ( $\perp_D$ ), and safety failure ( $\perp_S$ ).

Key Contributions

The Trilemma Theorem: The authors prove a "trilemma" for deterministic certifiers using finite one-step data. A certifier cannot simultaneously be:
- Uniformly sound over an unrestricted residual class.
- $\bar{\rho}$ -informative (returning a tube with finite projected width).
- Unrestricted beyond the observations.
  They further derive a finite, plan-dependent lower bound on the projected width for bounded smooth classes of model error, showing that even with smoothness constraints, a non-zero width is forced by the data-free gap.
ForeReach Algorithm: A practical method that combines supplied Lipschitz bounds, pairwise consistency checks, and zonotopic propagation to provide trajectory containment conditional on the validity of the supplied declaration.
Empirical Validation: Experiments on two benchmark systems (an affine point-mass and a six-state dynamic bicycle) demonstrate that ForeReach correctly abstains when plans enter unsupported regions (ensuring soundness) while recovering certification when relevant target data is available.

Experimental Results

Point-Mass System: Global calibration baselines certified unsupported plans but failed to contain the target trajectories (0% coverage), resulting in high false-safe rates. ForeReach achieved 100% complete-tube coverage on unsupported plans by abstaining, and 99.8% certified recall on corridor-supported plans.
Dynamic-Bicycle System: Similar trends were observed. Global calibration and other baselines (Gaussian processes, conformal methods) either produced uninformative tubes or failed to contain trajectories in unsupported regions. ForeReach maintained 100% coverage on unsupported plans (by abstaining) and achieved 98.3% certified recall on supported corridors with $N=400$ samples.
Learned Action Chunks: When applied to plans generated by a learned policy, the abstention rate correlated strongly (Spearman $\rho=0.697$ ) with the distance of the plan's nominal rollout to the nearest observation, effectively tracking the "support" of the data.
Incorrect Declarations: The study showed that if the supplied Lipschitz bound is too small (underestimating regularity), the pairwise test may pass (if no local contradiction exists), but the resulting tube may be too narrow to contain the true trajectory, leading to false certificates. This highlights that the method's soundness is conditional on the correctness of the external declaration.

Significance and Claims
The paper claims that the primary limitation in certifying plans from scarce data is not the quantity of data, but the lack of constraints on the model error in unobserved regions. The authors argue that a usable certificate requires structural information (like Lipschitz bounds) supplied from outside the data. While the data can refute an incorrect declaration, it cannot confirm a correct one in unobserved regions.

The significance of the work lies in:

Formalizing the theoretical limits (the trilemma) of data-driven reachability with sparse, isolated transitions.
Providing a method that explicitly trades off between soundness and informativeness based on the validity of side information, rather than assuming global properties of the data distribution.
Demonstrating that "abstention" is a necessary and informative outcome when a plan leaves the data support, contrasting with baselines that may produce narrow but unsound certificates.

The authors emphasize that their guarantee is conditional: the certificate is valid only if the supplied Lipschitz bound holds throughout the certification domain. The method does not learn the bound from the sparse data but utilizes it to construct a sound envelope.

Technical Summary: Certifying Plans under Model Mismatch: A Trilemma for Reachability from Scarce Data

More like this