💻 computer science

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

RAFAIL is a novel framework that detects robotic manipulation failures by identifying anomalies in task-relevant relationships between entities using point-cloud representations and relationship-specific OOD detectors, achieving high accuracy without requiring failure data or runtime VLM inference.

Original authors: Loris Schneider, Edgar Welte, Rania Rayyes

Published 2026-09-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Loris Schneider, Edgar Welte, Rania Rayyes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have become remarkably skilled at moving through the world, picking up objects, and assembling parts with a dexterity that once seemed impossible for machines. Yet, despite this progress, they remain fragile. When a robot encounters a situation it has not seen before—a slightly different lighting condition, an object placed in an unexpected spot, or a subtle slip of the hand—it often fails silently. It might continue moving as if nothing is wrong, eventually causing damage or ruining a task. For robots to be truly useful in homes or factories, they need a way to recognize when they are making a mistake and stop before the error becomes permanent. The challenge is not just noticing that something is different from the past, but understanding whether that difference actually matters. A harmless shift in the background might look strange to a computer, while a critical error in how a gripper holds a cup might go unnoticed.

Researchers at the Karlsruhe Institute of Technology have developed a new method to solve this problem, creating a system that watches a robot's actions not by looking at the whole scene, but by focusing on the specific relationships between the objects the robot is touching. They call this system RAFAIL. Instead of trying to understand the entire environment at once, which can be overwhelming and prone to false alarms, the system breaks the task down into pairs of interacting things: the gripper and the object, or the object and its target. By monitoring how these specific pairs relate to one another, the system can spot when a task is going wrong with much greater precision than previous methods.

The core idea behind this work is that most robotic failures happen because the relationship between two objects goes wrong. If a robot is trying to pour water from a cup into a bowl, the failure is not that the room looks different, but that the cup is tilted at the wrong angle relative to the bowl. To teach a robot to recognize these critical moments, the researchers first used a powerful type of artificial intelligence known as a vision-language model. This model is like a very smart observer that can look at a video of a successful task and describe what is happening, identifying which objects are important and how the task is progressing. However, running this smart observer every time a robot moves would be too slow and computationally expensive.

To get around this, the researchers used the smart observer only once, offline, to study a collection of successful demonstrations. They asked it to label which relationships between objects were most important at each step of the task and to track how far along the task was. These labels were then used to train a much simpler, faster system that runs alongside the robot in real time. This new system learns to create a compact mathematical description of each important relationship, such as the space between a gripper and a cup. It then checks these descriptions against what it has learned from successful runs. If a relationship starts to look strange—meaning the objects are in a configuration that was never seen during the successful training—it triggers an alert. Crucially, the system only pays attention to these alerts if the relationship is currently important for the task. If the gripper is holding a cup that is not yet relevant to the next step, a slight wobble is ignored. But if that same cup is being positioned for pouring, the system becomes highly sensitive to any deviation.

The team tested this approach on three different real-world tasks using a robotic arm equipped with a camera and a gripper. In one task, the robot had to pick up a cylinder and place it in a bowl. In another, it had to lift a fallen cup and set it upright. In the third, it had to pour a ball from a cup into a bowl. They ran the robot through hundreds of attempts, some of which were deliberately set up to fail by moving objects to unusual positions or adding distractions to the background. The new system successfully detected failures in 73.4% of the attempts while only raising false alarms in about 11.6% of the successful runs. This performance was significantly better than other leading methods that rely on measuring general uncertainty or looking at the robot's entire view of the world. Those other methods often struggled to tell the difference between a harmless change in the scene and a genuine mistake, either missing failures or stopping the robot unnecessarily.

What makes this result particularly promising is that the system does not need to see any examples of failure to learn how to detect them. It learns entirely from watching successful tasks, which are easier and safer to collect. Furthermore, when the system did detect a failure, it was usually able to pinpoint exactly which relationship had gone wrong. In the pouring task, for instance, it correctly identified that the cup and the bowl were misaligned in nearly 70% of the cases where it raised an alarm. This ability to localize the error suggests that the system is not just guessing that something is wrong, but is actually understanding the specific interaction that has broken down.

The researchers acknowledge that the system is not perfect. It relies on the ability to clearly see and track the objects involved in the task. If an object is hidden, if the camera loses track of it, or if the failure involves a force that cannot be seen, the system might miss it. Additionally, very subtle errors that do not change the visual relationship between objects might slip through. However, the study demonstrates that focusing on the specific connections between objects, rather than the whole picture, offers a robust and efficient way to keep robots safe. By teaching machines to watch the right things at the right time, this approach brings us closer to robots that can work alongside humans without constant supervision, knowing when to stop and ask for help before a small mistake becomes a big problem.

Technical Summary: RAFAIL – Relationship-Aware Failure Detection for Robotic Manipulation

Problem Statement

Reliable robotic manipulation requires the ability to detect execution failures in real-time. While Vision-Language Models (VLMs) can semantically assess task outcomes, their continuous runtime inference introduces prohibitive computational overhead. Conversely, traditional Out-of-Distribution (OOD) and uncertainty-based detectors are computationally efficient but often lack semantic specificity; they may trigger alerts on harmless scene variations while missing subtle, task-critical interaction errors. Furthermore, many existing detectors rely on policy-internal features that are not universally accessible across different policy architectures. The core challenge is to detect deviations that specifically affect task success without incurring the runtime cost of VLMs or relying on inaccessible internal policy states.

Methodology: RAFAIL Framework

RAFAIL (Relationship-Aware Failure Detection) addresses these challenges by decomposing global observations into task-relevant relationships between entities (e.g., gripper-object, object-target) rather than analyzing the scene globally. The framework operates through an offline training phase and a lightweight online deployment phase.

1. Offline Training and Semantic Distillation

The system leverages a VLM exclusively during the offline phase to generate semantic supervision from successful demonstrations:

Task Description & Object Extraction: A VLM analyzes demonstration frames to generate a task description, identify start/goal states, and extract task-relevant objects.
Relationship Importance & Progress Labeling: A smaller VLM is prompted to rank the importance of object relationships and the progress of the task across pairs of frame sequences. These rankings create a preference dataset.
Distillation: Two neural networks are trained to distill this semantic understanding:
- An Importance Labeler (ResNet18 + MLP) predicts the importance score ( $r_t$ ) for each relationship at a given time step.
- A Task Progress Labeler (ResNet18 + BiLSTM) predicts a continuous task progress value ( $p_t$ ) ranging from 0 to 1.
Relationship Representation Learning: A DGCNN encoder is trained using a contrastive loss to map point clouds of specific object pairs into a low-dimensional latent space ( $z$ ). The loss is weighted by the predicted relationship importance, encouraging the encoder to distinguish between different relationship states (positive/negative samples) while prioritizing semantically important interactions.

2. Online Deployment

During runtime, the system operates without VLM inference:

Policy Execution: A manipulation policy (ManiFlow) predicts actions based on point cloud and proprioceptive inputs. It simultaneously outputs the auxiliary relationship importance ( $\hat{r}_t$ ) and task progress ( $\hat{p}_t$ ) estimates.
Relationship-Specific OOD Detection: For each relationship $k$ , an independent OOD detector (specifically Random Network Distillation, RND) evaluates the latent representation $z^k_t$ . This detector is conditioned on the current task progress $p_t$ using FiLM layers.
Failure Gating: An OOD score ( $s^k_t$ ) is generated for each relationship. A failure is triggered only if a relationship is both important (above a threshold $\theta_{imp}$ ) and anomalous (OOD score above a calibrated threshold $\theta^k_{ood}$ ).
$f^k_t = \mathbb{I}[\hat{r}^k_t > \theta_{imp}] \cdot \mathbb{I}[s^k_t > \theta^k_{ood}]$
The final failure signal is the maximum of these gated scores across all relationships.

Key Contributions

Relationship-Aware Framework: A novel approach that decomposes global observations into dedicated representations for individual task-relevant relationships, allowing for targeted failure detection.
Semantic Distillation without Runtime VLM: A method to distill relationship importance and task progress from a VLM into lightweight neural labelers, enabling semantic awareness during deployment without online VLM inference.
Task-Progress-Conditioned OOD Detection: An OOD detector that integrates task progress and relationship importance to localize anomalous interactions, remaining independent of policy-internal latent representations.

Experimental Results

RAFAIL was evaluated on three real-world robotic manipulation tasks: PickPlace, Uprighting, and Pouring. The system was tested against strong baselines including Action Chunk Entropy (ACE), FIPER, and various OOD detectors (logpZO, RND, PCA-kmeans).

Performance: RAFAIL achieved a balanced accuracy of 73.4% across all tasks, outperforming the strongest baseline (ACE) which achieved 65.2%.
Trade-off: RAFAIL maintained a favorable balance between True Positive Rate (TPR: 58.5%) and False Positive Rate (FPR: 11.6%), whereas other methods often struggled to maintain this trade-off consistently across different tasks.
Temporal Alignment: Analysis of detection timing showed that RAFAIL's signals were more closely aligned with the physical manifestation of failure compared to baselines, which often triggered significantly earlier or later.
Localization: In true-positive detections, the relationship triggering the alert matched the manually annotated failure relationship in 93.3% of PickPlace cases, 69.3% of Pouring cases, and 68.0% of Uprighting cases.
Ablation Studies: Removing the relationship-wise encoding (encoding the whole scene instead) or removing the importance gating mechanism significantly degraded performance, confirming that focusing on specific relationships and suppressing irrelevant ones is critical for robustness.

Significance and Limitations

The paper posits that RAFAIL offers a practical solution for reliable autonomous operation by detecting execution failures with high accuracy and low computational overhead, without requiring failure data for training. By focusing on task-relevant relationships, the system avoids the noise of global scene variations.

However, the authors modestly acknowledge limitations:

Subtle Failures: Failures caused by small pose deviations that remain within the distribution of successful relationship states may go undetected.
Unmodeled Entities: The framework assumes task-critical failures can be represented by pairwise relations among segmentable entities. Failures involving unmodeled entities, global scene properties, non-geometric contact forces, or segmentation/tracking breakdowns may remain undetected.

Future work is suggested to investigate finer-grained relational representations to capture these subtle errors.