💻 computer science

SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

The paper introduces SafeManip, a property-driven benchmark that utilizes Linear Temporal Logic over finite traces (LTLf) to evaluate temporal safety violations in robotic manipulation, revealing that even high-performing vision-language-action policies frequently exhibit unsafe behaviors despite achieving task success.

Original authors: Chengyue Huang, Khang Vo Huynh, Sebastian Elbaum, Zsolt Kira, Lu Feng

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chengyue Huang, Khang Vo Huynh, Sebastian Elbaum, Zsolt Kira, Lu Feng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to make a sandwich. In the past, if the robot successfully put the bread, cheese, and ham together and placed the sandwich on a plate, we would say, "Great job! The task is complete!"

But the authors of this paper, SAFEMANIP, argue that "completing the task" isn't enough. A robot could make a perfect sandwich but do so in a terrifyingly unsafe way: it might drag a dirty knife across a clean counter, drop the bread on the floor before picking it up again, or slam the fridge door while holding a glass of milk.

SAFEMANIP is a new "report card" for robots that doesn't just ask, "Did you finish?" It asks, "Did you finish safely?"

Here is how the paper breaks it down, using simple analogies:

1. The Problem: The "Finish Line" Trap

Think of a robot's job like a race. Traditionally, we only care if the robot crosses the finish line. But what if the robot tripped over a baby, ran through a wall, and then crossed the line? It finished, but it was a disaster.

The paper says many robots are like this. They might succeed at the goal (the task) but fail at the journey (the safety). These failures often happen over time. It's not just about being in a bad spot right now; it's about doing the wrong thing at the wrong time.

Example: A robot touches a clean spoon after it has touched raw chicken. The spoon is clean at the start and the end, but the sequence of events was dangerous.

2. The Solution: The "Safety Script" (SAFEMANIP)

To fix this, the researchers created SAFEMANIP. Think of this as a set of safety scripts written in a special language called LTLf (which is like a strict set of rules for time).

Instead of just checking if the robot hit a wall, these scripts check the story of the robot's actions:

The "Grasp" Rule: Once you pick up a slippery bottle, you must hold it tight until you are ready to put it down. You can't let it wobble and fall halfway.
The "Cleanliness" Rule: If you touch raw meat, you cannot touch a clean bowl until you have washed your hands (or the robot's gripper).
The "Door" Rule: You can't reach inside a microwave unless the door is fully open. You can't put a second plate in if the first one is still inside.

These scripts are reusable templates. Just like you can use the same "recipe" for making different types of sandwiches, the researchers can use the same safety rules for different tasks, whether it's cleaning a kitchen, cooking, or organizing a pantry.

3. The Test: The "Kitchen Simulator"

The researchers tested this system on 50 different household tasks (like making coffee, washing dishes, or reheating food) inside a computer simulation called RoboCasa. They used six different AI robots (including some very advanced ones like $\pi_0$ and GR00T) to try these tasks.

They didn't just watch if the robot finished; they ran the robot's actions through their "safety scripts" to see if it broke any rules along the way.

4. The Shocking Results

The findings were surprising and a bit scary:

Success $\neq$ Safety: The smarter the robot got at finishing tasks, it didn't necessarily get safer. In fact, some robots that finished more tasks actually broke more safety rules.
The "Success-but-Unsafe" Trap: Many robots finished the task, but they did it in a way that would be dangerous in real life. For example, a robot might successfully put a cup in the dishwasher, but it did so by dropping the cup, letting it roll across the floor, and then scooping it up. The task was "done," but the safety script screamed "FAIL."
Longer Tasks = More Danger: The more complex the task (like cooking a full meal vs. just picking up a cup), the more likely the robot was to make a safety mistake. The longer the story, the more chances there are for a plot hole.

5. The Takeaway

The paper concludes that we need a new way to judge robots. We can't just look at the final score. We need to watch the whole movie.

SAFEMANIP provides a tool to watch the movie frame-by-frame, catching the moments where the robot is being reckless, even if it eventually gets the job done. It helps us understand where and when robots fail, so we can fix them before we let them into our actual kitchens.

In short: Just because a robot can do the job doesn't mean it can do it without breaking things, hurting people, or making a mess. SAFEMANIP is the tool that finally lets us check the "how" alongside the "what."

Technical Summary: SAFEMANIP

Problem Statement

Robotic manipulation has traditionally been evaluated primarily by task success metrics (e.g., success rate). However, successful task completion does not guarantee safe execution. Many safety failures in robotic manipulation are temporal in nature, arising from the sequence and timing of events rather than instantaneous state violations. Examples include a robot touching a clean surface after handling contaminated food, releasing an object before it is fully inside an enclosure, or dropping an item during transport. Existing benchmarks often rely on task-specific hazard labels, instantaneous collision checks, or cumulative trajectory costs. While useful, these metrics often obscure which safety rule was violated, when it occurred, and whether a task was completed safely or merely completed. There is a lack of reusable, property-driven frameworks that explicitly evaluate temporal safety properties across diverse manipulation tasks.

Methodology

The authors introduce SAFEMANIP, a property-driven benchmark designed to explicitly evaluate temporal safety in robotic manipulation. The framework consists of three core components:

1. Formal Specification Language (LTLf)

SAFEMANIP utilizes Linear Temporal Logic over finite traces (LTLf) to define safety properties. Unlike standard LTL used for infinite traces, LTLf is suited for finite robotic rollouts that terminate upon task success, failure, or horizon limits.

Atomic Propositions: The system grounds continuous observations (object poses, contact events, gripper states, fixture states) into Boolean predicates (e.g., Contaminated, StableGrasp, EnclosureCleared).
Temporal Operators: The framework employs standard operators: Next ( $\bigcirc$ ), Until ( $U$ ), Always ( $\square$ ), and Eventually ( $\diamond$ ) to specify ordering, invariants, and recovery requirements.
Monitoring: Each LTLf formula is compiled into a Deterministic Finite Automaton (DFA). As a rollout generates a symbolic trace, the DFA updates its state to determine if the property is satisfied or violated.

2. Reusable Safety Property Templates

The benchmark defines a suite of eight safety categories with reusable LTLf templates. These templates can be instantiated with task-specific objects, fixtures, regions, or skills, allowing generalization across environments:

Collision and Contact Safety: Avoiding unsafe contact throughout manipulation.
Grasp Stability: Ensuring an object remains stably held until release.
Release Stability: Ensuring a released object settles safely (no rolling, falling, or spilling).
Cross-Contamination: Enforcing sanitization before clean contact after handling contaminants.
Action-Onset Safety: Verifying safe preconditions before initiating a skill.
Mechanism Safety: Requiring retraction and recovery after fixture impacts (e.g., doors, drawers).
Containment Safety: Ensuring transferred liquids or objects reach their intended receiver.
Enclosure and Access Safety: Enforcing sequencing constraints (e.g., clearing an enclosure before inserting new items, waiting for full opening before reaching in).

3. Evaluation Protocol

Task Suite: The benchmark is instantiated on 50 RoboCasa365 tasks covering diverse household manipulation skills (cleaning, cooking, storage, etc.), grouped into seven manipulation suites.
Policies Evaluated: Six Vision-Language-Action (VLA) policies and training variants were tested, including $\pi_0$ , $\pi_{0.5}$ , GR00T N1.5, and three GR00T variants (pretraining-only, target-only, and combined).
Metrics: The framework decouples task success from safety. It reports:
- Task Success Rate: Fraction of rollouts completing the task.
- Safety Violation Rate: Overall and per-property violation frequencies.
- Outcome Decomposition: Categorizing rollouts into success-and-safe, success-but-unsafe, fail-but-safe, and fail-and-unsafe.
- Unsafe-State Exposure Rate: The percentage of timesteps spent in a violating state.

Key Results

The evaluation of six VLA policies across 50 tasks yielded the following findings:

Task Success Does Not Imply Safety: Improvements in task success rates do not reliably translate into safer execution. For instance, $\pi_{0.5}$ showed a slight increase in task success over $\pi_0$ but a significant increase in safety violations (from 69.7% to 82.8%). Many "successful" rollouts were classified as success-but-unsafe.
Category-Specific Failures: Safety failures are not uniformly distributed. Collision/Contact and Release Stability violations were the most frequent. Release failures were particularly persistent, often leaving objects unsettled for multiple timesteps.
Impact of Task Horizon and Structure:
- Horizon: Safety violation rates generally increased as task horizons extended from atomic to long-horizon tasks. Longer executions introduce more safety-critical transitions (contacts, placements, fixture motions), accumulating violations even when the final goal is achieved.
- Suite Dependence: Complex suites involving cooking, cleaning, and serving exposed significantly more temporal violations than simple fixture tasks.
Prompt-Based Guidance: An exploratory analysis using safety prompts on the GR00T-tpt model showed a tradeoff: prompts encouraged more conservative behavior (lowering violation rates slightly) but significantly reduced task success rates.

Significance and Claims

The paper claims that SAFEMANIP provides a reusable evaluation layer for diagnosing temporal safety failures that go beyond simple task completion or binary safety labels.

Beyond Task Success: The authors demonstrate that task success can mask significant temporal safety failures. By separating task completion from safe execution, SAFEMANIP reveals that models often complete tasks unsafely.
Granular Diagnosis: The property-driven approach allows researchers to identify specific failure modes (e.g., unstable release vs. collision) and their prevalence across different task structures, which aggregate metrics often obscure.
Generalizability: The use of reusable LTLf templates allows the same safety specifications to be applied across different objects, fixtures, and environments, facilitating comparative safety analysis.
Modest Scope: The authors acknowledge that SAFEMANIP evaluates specified properties rather than all possible hazards. The current instantiation relies on simulator state for grounding predicates, and the results are descriptive of the RoboCasa365 benchmark protocol rather than a universal claim for all robotic domains. The work serves as a foundation for future integration of property-level safety signals into policy training and real-world deployment.