💻 computer science

ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models

ContactGuard is a pre-contact execution monitoring system that utilizes an action-conditioned latent world model trained on unlabelled data to predict short-horizon visual consequences of planned actions, enabling real-time failure detection and abort signals for contact-rich manipulation tasks without modifying the underlying policy.

Original authors: Gehan Zheng, Matthew Johnson-Roberson, Weiming Zhi

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gehan Zheng, Matthew Johnson-Roberson, Weiming Zhi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots are learning to do chores, but they are still a bit clumsy, like a toddler trying to stack blocks. In the field of robotics, specifically "manipulation," the goal is to get machines to grab, move, and arrange objects just like humans do. The big challenge isn't just moving the arm; it's the moment the robot's hand actually touches the object. This is called "contact." If a robot approaches a cup at the wrong angle, it might knock it over, slip off the side, or push it away before it even tries to lift it. Once that clumsy bump happens, the scene is ruined, and the robot has to start over.

To fix this, scientists use "visuomotor policies." Think of these as the robot's brain and nervous system combined. They look at the camera feed (vision) and decide what the motors should do next (motor control). Modern robots often move in "chunks," meaning they plan a whole sequence of moves at once—like "reach, close fingers, lift"—instead of deciding every tiny movement one by one. The problem is, by the time the robot realizes it's about to make a mistake, it's often too late to stop. The paper we are about to explore asks a simple but powerful question: Can we build a "safety net" that looks at the robot's planned moves and predicts if they will fail before the robot actually touches the object?

Meet ContactGuard: The Robot's Crystal Ball

The researchers behind this paper, Gehan Zheng and colleagues, have built a system called ContactGuard. You can think of it as a super-smart, invisible co-pilot that sits next to the robot's brain. Its only job is to watch the robot's upcoming "chunk" of actions and shout "STOP!" if it sees a disaster coming.

Here is how it works, using a fun analogy: Imagine you are playing a video game where you have to jump over a pit. You press a button to jump. A normal robot would just jump and hope for the best. If it misses, it falls in the pit. ContactGuard is like a friend who has a crystal ball. Before you even press the jump button, your friend looks at the crystal ball, sees the jump you planned to make, and says, "Whoa, that jump is too short! You're going to fall!" Because your friend predicted the fall before you moved, you can choose not to jump, and you stay safe.

The Magic of the "Latent World Model"
How does this crystal ball work? The paper uses something called a "Latent World Model." In the past, trying to predict the future for a robot meant trying to generate a perfect, photorealistic video of what the next second would look like. That is incredibly hard and slow, like trying to paint a masterpiece in a split second.

Instead, ContactGuard uses a shortcut. It doesn't try to paint the picture; it tries to understand the story of the picture. It translates the camera images into a compact "summary" (called a latent embedding). It learns how these summaries change when the robot moves. So, instead of predicting "a blurry cup will fall," it predicts "the summary of the scene will change in a way that usually means failure." This is much faster and smarter.

The "Pre-Contact" Superpower
The coolest part of ContactGuard is its timing. It doesn't wait until the robot is already touching the cup. It looks at the plan just before the gripper closes.

The Plan: The robot's main brain says, "I'm going to close my gripper in 0.5 seconds."
The Check: ContactGuard takes that plan and runs a quick simulation in its head (a "rollout"). It asks, "If the robot does exactly this, what will the world look like 0.5 seconds later?"
The Verdict: It checks the result. If the simulation shows the cup slipping or the gripper missing, it sends a signal to abort the move. The robot stops, opens its hand, and tries again, all without ever actually bumping the cup.

What the Experiments Showed
The team tested this on a real robot with a 14-jointed arm and three cameras. They tried four different tasks: picking up a cup, a box, a pencil, and folding a towel. These are tricky because the objects are small, slippery, or soft.

The results were impressive. ContactGuard was much better at spotting failures than other methods:

Better than "Current Sight": If you just look at the robot's current view without imagining the future, you miss a lot of dangers. ContactGuard's "imagination" gave it a huge advantage.
Better than "Bad Guesses": They tested what happens if they gave the system the wrong action plan (like shuffling the order of moves). The system's prediction dropped to near-random guessing, proving that it really was using the specific action plan to make its decision, not just looking at the background.
Speed: It was fast enough to run in real-time. On a powerful computer, it took less than 20 milliseconds to check the plan and make a decision. That's faster than a human blink, so it doesn't slow the robot down.

What It Can't Do (Yet)
The paper is very honest about its limits. ContactGuard is a "veto" system, not a "fix-it" system. If it says "Stop," the robot stops. But ContactGuard doesn't know how to fix the problem or try a different way to grab the object. It just says, "Don't do that specific move." The robot would need a separate system to figure out what to do next. Also, it only works for short-term actions (like a single grab), not for long, complex tasks that take minutes to plan.

Why This Matters
This research suggests that we don't need to rebuild the entire robot brain to make it safer. We can just add a "guardian" layer on top of existing robots. This guardian uses a clever, fast way to imagine the future and stop the robot before it makes a mess. It suggests that by teaching robots to "think ahead" in a compact, summary way, we can make them much more reliable in the real world, preventing those frustrating moments where a robot knocks over a glass of water just because it approached the wrong way.

=== SUMMARY ===

Technical Summary: ContactGuard

Problem Statement

In contact-rich robotic manipulation, failures often occur due to minor errors near the moment of contact (e.g., off-center approaches, nudging objects, or premature gripper closure). These errors are difficult for visuomotor policies to avoid under occlusion, clutter, or distribution shifts. Crucially, by the time conventional detectors identify a failure, the robot has often already committed to the contact, making correction difficult or impossible. This is particularly limiting in wrist-camera setups where close views aid observation but poor approach trajectories may already disturb the scene before detection.

Current execution monitoring methods often rely on detecting failures after they occur or require joint training with the policy. There is a need for a pre-contact execution monitor that can evaluate the likely consequence of an imminent action chunk before physical contact occurs, allowing the system to abort a poor interaction without modifying the underlying policy.

Methodology: ContactGuard

ContactGuard is a pre-contact execution monitor designed for chunked visuomotor policies. It operates as a policy-decoupled predictive verifier that runs alongside an existing "black-box" policy. The system consists of three main components:

1. Action-Conditioned Latent World Model

Instead of generating photorealistic future frames (which is computationally expensive and ambiguous), ContactGuard learns a compact representation of future outcomes.

Architecture: Based on the Joint-Embedding Predictive Architecture (JEPA) principle, specifically adapting the LeWorldModel (LeWM) framework. It uses a shared multi-view visual encoder (ViT-Tiny) to map synchronized camera observations into a latent space.
Prediction: A causal Transformer predictor, conditioned on the robot's planned action chunk, forecasts the evolution of these latent embeddings. It predicts how the visual state embedding will change under the specific actions proposed by the policy.
Training: The world model is trained on unlabelled robot trajectories using next-latent regression with a SIGReg anti-collapse regularizer. This allows the model to learn task-agnostic visual dynamics without requiring success/failure labels.

2. Lightweight Failure Probe

Once the world model is trained and frozen, a lightweight classifier is trained to interpret the predicted future latents.

Input: The probe takes the predicted latent embedding ( $\hat{z}_{t+K}$ ) resulting from rolling out the model $K$ steps under the planned action chunk.
Training: A simple logistic regression probe is trained on a small, labelled set of pre-contact clips (binary success/failure). It maps the predicted future latent to a failure probability $P(\text{fail})$ .
Decoupling: This separates the learning of visual dynamics (unlabelled, large data) from outcome supervision (labelled, small data).

3. Online Pre-Contact Monitoring Loop

During deployment, ContactGuard operates in real-time alongside the visuomotor policy (e.g., ACT):

Trigger: The system scans the upcoming action chunk for an imminent contact event (e.g., gripper closure).
Anchor: When the contact event is within a pre-defined horizon ( $k_{pre}$ ), the monitor activates.
Rollout: The frozen world model is rolled forward $K$ steps using the policy's planned actions to generate a predicted post-contact latent.
Verification: The frozen probe scores this latent. If $P(\text{fail}) > \tau$ (a task-specific threshold), the system aborts execution before the gripper closes. Otherwise, the policy continues unchanged.

Key Contributions

Formulation of Pre-Contact Monitoring: The paper formulates the problem of evaluating the consequence of an imminent contact action before commitment, specifically for chunked visuomotor policies.
Action-Conditioned Latent Monitor: Introduction of ContactGuard, which uses a latent world model to roll out compact multi-view visual embeddings under planned actions, avoiding pixel-level video prediction.
Demonstration of Future Latent Utility: Through ablation studies, the authors show that predicted future latents contain failure information not available in current observations or raw planned actions alone.
Policy-Decoupled Deployment: The system enables live real-robot pre-contact aborts without modifying the underlying policy, searching for candidate actions, or generating pixel-level videos.

Experimental Results

The authors evaluated ContactGuard on four real-world contact-rich manipulation tasks (Pick-and-place with cups/boxes, Pencil-and-notebook, Towel-fold) using a 14-DoF dual-arm robot.

Performance vs. Baselines: ContactGuard significantly outperformed:
- Direct-linear: A baseline predicting failure directly from current state and actions without imagination.
- LeWM: A single-view world model baseline.
- Current Latent: An ablation where the probe uses the current latent instead of the predicted future latent.
- External Detectors: Established runtime failure detectors like FAIL-Detect, RND, and SAFE.
Key Metrics: ContactGuard achieved the highest Balanced Accuracy and ROC AUC across all tasks. For example, on the "Cup" task, it achieved 0.940 Balanced Accuracy compared to 0.780 for LeWM and 0.580 for Direct-linear.
Information Source: Ablation studies confirmed that the signal arises from the imagined consequence of the specific proposed action. Shuffling action chunks or using corrupted actions caused performance to drop to near-chance levels, proving the monitor is sensitive to the specific action plan, not just the visual state.
Runtime: The system runs efficiently on an NVIDIA RTX 5090, with full inference (encode + rollout + probe) taking approximately 19ms for a 1-second horizon, well within the pre-contact time budget.
Proprioception: Interestingly, adding proprioceptive state (joint positions/efforts) to the single-view model decreased performance in their small-data regime, suggesting that straightforward state fusion can act as a domain-specific shortcut that displaces grasp-relevant visual dynamics.

Significance and Claims

The paper claims that action-conditioned latent imagination serves as a practical substrate for preventing likely failures before physical contact occurs.

Pre-emptive vs. Reactive: Unlike methods that detect failure after contact has disturbed the scene, ContactGuard intervenes before the scene changes.
Modularity: It treats the visuomotor policy as a black box, requiring no joint training or access to policy internals. This makes it an attachable safety layer for existing systems.
Efficiency: By operating in latent space rather than pixel space, it achieves high accuracy with low computational overhead suitable for real-time control.
Limitations: The authors modestly note that ContactGuard prevents failures by abstaining (aborting) but does not inherently recover from them or complete the task after an abort. Future work would require coupling the monitor with recovery modules or multi-sample policies to select alternative actions.

In summary, ContactGuard demonstrates that predicting the short-horizon consequences of planned actions in a compact latent space provides a robust, policy-agnostic mechanism for ensuring safety in contact-rich robotic manipulation.