RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation

1. Problem Statement

Vision-Language-Action (VLA) models have enabled robots to perform complex manipulation tasks via natural language prompts. However, these models, typically trained via imitation learning, struggle in dynamic real-world environments due to Out-of-Distribution (OOD) conditions.

• The Challenge: When a robot encounters unexpected scenarios (e.g., object slippage, gripper failure, or spatial misalignment), VLA models often fail silently or degrade in performance because they lack a runtime monitoring system to detect deviations from the learned distribution.
• Limitations of Existing Solutions:
  • Rule-based/Classification methods: Require exhaustive enumeration of failure modes or manual preconditions, failing to handle the combinatorial variability of real-world tasks.
  • VLM-based monitors (e.g., Sentinel): Rely on high-level reasoning and chain-of-thought, resulting in multi-second latency, which is too slow for real-time intervention in closed-loop control.
  • Existing Flow-based methods (e.g., FailDetect): Often concatenate image and robot features, leading to entangled representations and suboptimal feature processing.

2. Methodology: RC-NF

The authors propose Robot-Conditioned Normalizing Flow (RC-NF), a real-time, unsupervised anomaly detection model. It monitors the joint distribution of robot states and object motion trajectories to ensure consistency with the intended task.

Core Architecture

RC-NF is built upon Normalizing Flows (NF), which map complex data distributions to a simple Gaussian latent space via invertible transformations.

• Conditional Modeling: Unlike standard NFs, RC-NF conditions the flow on both the robot state ($s$) and task embedding ($\tau$). This allows the model to learn the probability density of successful execution given a specific task and robot configuration.
• Anomaly Scoring: During inference, the model calculates the log-likelihood of the current observation. A low likelihood (high negative log-likelihood) indicates an anomaly. If the score exceeds a calibrated threshold, an OOD event is triggered.

Key Innovation: Robot-Conditioned Point Query Network (RCPQNet)

The core of RC-NF is a novel affine coupling layer called RCPQNet, which replaces standard coupling layers in the flow. It decouples robot and object features while preserving their interaction:

1. Input Processing:
   • Object Representation: Video streams are processed by SAM2 to extract object masks, which are grid-sampled into point sets ($\mathcal{X}$).
   • Task Embedding: Natural language prompts are encoded into surface vectors on a hypersphere to ensure maximal separation in latent space.
   • Robot State: Includes joint angles, gripper state, and Cartesian pose.
2. Dual-Branch Feature Encoding:
   • Dynamic Shape Branch: Normalizes point sets to capture relative shape changes (robust to translation/scale).
   • Positional Residual Branch: Captures absolute spatial information lost during normalization.
   • These branches use GRUs and Transformers to model temporal dependencies.
3. Task-Aware Robot-Conditioned Query:
   • The robot state is projected and modulated by the task embedding (via FiLM) to create query tokens.
   • The object point features serve as memory tokens.
   • A Cross-Attention mechanism within a Transformer allows the model to generate context-dependent affine parameters (scale $\gamma$ and shift $\beta$) for the flow transformation. This ensures the flow adapts dynamically to the specific task and robot state.

Anomaly Handling Mechanism

RC-NF operates as a plug-and-play module alongside the VLA policy:

• State-Level OOD: If the robot's physical trajectory drifts (e.g., object falls), RC-NF triggers a homing procedure to rollback the trajectory and re-align with the task.
• Task-Level OOD: If the environment changes such that the task is impossible (e.g., drawer closes), RC-NF signals the high-level planner to replan the task sequence.

3. Key Contributions

1. RC-NF Model: An unsupervised, real-time anomaly detector using a conditional normalizing flow with a novel RCPQNet layer that effectively fuses task, robot, and object point-set data.
2. LIBERO-Anomaly-10 Benchmark: A new evaluation benchmark extending LIBERO-10, introducing three specific anomaly types:
   • Gripper Open: Robot fails to grasp (object remains static).
   • Gripper Slippage: Object slips due to zero friction.
   • Spatial Misalignment: Robot moves to the wrong location.
3. Real-World Integration: Demonstration of RC-NF as a sub-100ms monitoring module that enhances the robustness of VLA models (specifically $\pi_0$) without modifying their underlying architecture.

4. Experimental Results

• Simulation Performance (LIBERO-Anomaly-10):
  • RC-NF achieved State-of-the-Art (SOTA) performance across all anomaly types.
  • Metrics: Outperformed the best baseline (FailDetect) by ~8% in AUC and ~10% in Average Precision (AP).
  • Comparison with VLMs: While VLMs (GPT-5, Gemini, Claude) performed moderately on gripper issues, they degraded to near-random performance ($\approx$0.5 AUC) on Spatial Misalignment, highlighting their inability to reason about fine-grained spatial trajectories compared to RC-NF's geometric approach.
• Ablation Studies:
  • Removing Task Embeddings caused a massive drop in performance for Spatial Misalignment, proving the necessity of task-aware conditioning.
  • Removing Robot State features significantly hurt detection of "Gripper Open" anomalies, confirming the need for joint robot-object reasoning.
  • The Dual-Branch Encoding was critical, with the Dynamic Shape branch being the most important for detecting trajectory deviations.
• Real-World Deployment:
  • Latency: Detection occurs within 100 ms on consumer-grade GPUs (RTX 3090).
  • Effectiveness: In experiments with a Franka arm, RC-NF successfully prevented failures by triggering trajectory rollbacks (homing) when objects slipped or moved unexpectedly, and triggered replanning when environmental conditions changed (e.g., a drawer closing).

5. Significance

This work addresses a critical bottleneck in deploying VLA models: reliability in dynamic environments.

• Safety & Adaptability: By providing a fast, unsupervised OOD signal, RC-NF allows robots to self-correct or request replanning before catastrophic failures occur.
• Efficiency: It bridges the gap between high-level semantic understanding (VLA) and low-level control, offering a solution that is significantly faster than VLM-based monitors and more robust than rule-based systems.
• Generalizability: The "plug-and-play" nature of RC-NF means it can be integrated into existing imitation learning pipelines without retraining the core policy, making it a practical solution for immediate deployment in unstructured real-world settings.