⚡ electrical engineering

Towards General Language-Conditioned Latent Safety Filters

This paper proposes a language-conditioned safety filtering framework using Hamilton-Jacobi actors and critics to dynamically enforce diverse safety constraints across various robotic tasks, demonstrating reduced violations and partial transfer to unseen constraint instances.

Original authors: Ihab Tabbara, Yuxuan Yang, Hussein Sibai

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ihab Tabbara, Yuxuan Yang, Hussein Sibai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots aren't just clumsy machines programmed to do one specific job, like a toaster that only toasts bread. Instead, picture a robot that can understand you like a helpful friend. You could say, "Please clean up the table," or "Build a tower with these blocks," and it would figure out how to do it. This is the exciting frontier of "generalist" robot learning, where artificial intelligence combines what it sees (vision) with what you say (language) to decide what to do (action). But here's the catch: just because a robot is smart enough to understand your request doesn't mean it's smart enough to know what not to do. If you ask it to clean a table, it might accidentally knock over a precious vase or spill a glass of milk.

To keep robots safe, scientists use something called a "safety filter." Think of this filter as a strict, invisible guardian angel standing right next to the robot's brain. Before the robot moves its arm, the guardian checks the plan. If the plan looks like it might hit a vase, the guardian steps in, grabs the controls, and steers the robot away from danger. Traditionally, these guardians were very rigid; you had to teach them specifically about vases, then separately about wine glasses, and then separately about hot coffee. If you wanted the robot to avoid something new, you had to retrain the guardian from scratch. The big question researchers are asking is: Can we teach a robot's guardian to understand any safety rule just by listening to us speak, so it can adapt instantly to new situations without needing a total reboot?

This paper, titled "Towards General Language-Conditioned Latent Safety Filters," explores exactly that idea. The authors, Ihab Tabbara, Yuxuan Yang, and Hussein Sibai, propose a new kind of safety filter that doesn't need to be retrained for every new rule. Instead, they built a system where the guardian can listen to a natural language instruction—like "don't touch the milk carton"—and instantly apply that rule to the robot's actions. They tested this on robots performing tasks like picking up blocks, wiping tables, and stacking objects.

The researchers found that this "language-conditioned" filter works surprisingly well. In their experiments, the new filter successfully prevented the robot from crashing into specific objects when told to avoid them, even when the robot's original plan was to crash right into them. For example, in a task where a robot had to stack colored blocks in a specific order, the filter helped the robot follow the instructions correctly about 80% of the time, even when the instructions changed. The filter was also able to generalize to some extent; when they tested it on objects or colors it had never seen before (like a yellow book instead of a red one), it still managed to follow the rules better than random chance, though it wasn't perfect.

However, the paper is careful not to call this a perfect solution. The authors show that while the filter is a huge step forward, it still makes mistakes, especially when the robot is looking at the world through a camera (vision) rather than knowing the exact position of every object (which is like having a superpower the robot doesn't actually have). They also tested using advanced AI models to act as the "eyes" that tell the filter what is dangerous, but found that even the smartest current models aren't reliable enough to be the sole judge of safety on their own.

The study suggests that we are moving toward a future where we can have one general safety filter that learns to understand many different rules just by reading them, rather than needing a unique filter for every single object in a room. While the current version isn't flawless and still struggles with completely new scenarios, the results suggest that this approach is a promising path forward. It hints at a future where robots can be given complex, changing safety instructions on the fly, making them safer and more useful in our homes and workplaces without needing a team of engineers to reprogram them every time we move a piece of furniture.

Technical Summary: Towards General Language-Conditioned Latent Safety Filters

Problem Statement

As robot policies evolve from task-specific controllers to generalist Vision-Language-Action (VLA) models capable of executing diverse natural language instructions, ensuring safe deployment remains a critical challenge. Safety requirements vary significantly across users, environments, and applications, even for identical tasks. Existing safety filters are largely constraint-specific; they rely on fixed notions of safety (e.g., forward invariance against a pre-defined failure set) and require redesign or retraining whenever safety specifications change. Furthermore, current VLA policies often fail to implicitly learn safety from training data or reliably follow explicit language-based safety prompts. This paper investigates language-conditioned safety filtering, aiming to create a single, learned safety filter that can adapt to deployment-time safety requirements expressed through natural language, thereby supporting the safe deployment of generalist robot policies.

Methodology

The authors propose a framework that extends Hamilton–Jacobi (HJ) reachability analysis to condition on language-specified constraints. The core approach involves learning a constraint-conditioned safety actor and critic that operate as a runtime filter for a nominal policy.

1. Formulation

The system operates at each timestep $t$ with an observation $o_t$ , a task instruction $\tau$ , and a natural-language safety constraint $c$ (e.g., "avoid the milk box"). A nominal policy $\pi_{nom}$ proposes an action $a^{nom}_t$ . The safety filter evaluates this action using a learned HJ critic $Q_\theta(z_t, a_t)$ , where the state representation $z_t$ encodes both the visual observation and the language constraint.

Critic: Estimates the discounted worst-case safety value of executing $a^{nom}_t$ followed by the optimal safety policy.
Actor: A learned policy trained to maximize safety.
Filtering Logic: If $Q_\theta(z_t, a^{nom}_t) \geq \epsilon$ (where $\epsilon$ is a safety margin), the nominal action is executed. Otherwise, the filter intervenes, sampling candidate actions from the HJ actor and perturbations of the nominal action, selecting the closest safe action that satisfies the constraint.

2. Representations and Encoding

The paper evaluates two input representations for the HJ actor/critic:

Privileged Oracle (Ground Truth): Uses the full system state $s_t$ (object poses, robot state) and a multi-hot encoding of the constraint. This serves as an upper bound to test the efficacy of the HJ formulation without perception errors.
Vision-Language (VL): Uses raw RGB observations (third-person and wrist-mounted views), robot proprioception, and the language constraint. To encode these, the authors utilize the $\pi0.5$ VLA backbone, which processes multi-view observations and language through a shared transformer. The resulting token sequence is aggregated via mean pooling to form the latent representation $z_t$ .

3. Failure Functions

A critical component is the failure function $h(z_t)$ , which defines the boundary of the unsafe set.

Geometric Constraints: For tasks like collision avoidance, $h$ is defined by distance metrics (e.g., distance to an obstacle minus a threshold).
Semantic Constraints: For complex semantic failures (e.g., toppling an object), the authors explore using Vision-Language Models (VLMs) as failure oracles. They query 11 different VLMs to determine if a state violates a constraint, mapping the output to a positive or negative constant. However, they find current VLMs are not reliable enough for direct runtime use as failure oracles due to high false-negative rates on safety violations.

4. Training

The HJ critic and actor are trained using Soft Actor-Critic (SAC) to minimize the Bellman error loss, approximating the optimal value function for avoiding the constraint-defined failure set. The training data includes 5,000 manipulation trajectories (derived from modified LIBERO tasks), deliberately enriched with safety violations (approx. 90%).

Key Contributions

Formulation: The paper formulates language-conditioned safety filtering, extending HJ reachability to condition on natural-language constraints rather than fixed failure sets.
Evaluation: Comprehensive evaluation across three manipulation tasks (Safe Grab, Safe Wipe, Stack Blocks) examining constraint satisfaction, out-of-distribution (OOD) generalization, and the trade-off between safety, task success, and intervention rates.
Comparative Analysis: A comparison between the proposed general language-conditioned filter and specialized, non-language-conditioned filters trained per constraint.
VLM Benchmarking: An evaluation of 11 VLMs as potential failure functions for robotic manipulation, highlighting their current limitations in reliability.
Dataset Release: The authors commit to releasing a dataset of 5,000 LIBERO-style manipulation trajectories with safety violations.

Experimental Results

The experiments were conducted in RoboSuite environments using both Ground Truth (GT) and Vision-Language (VL) observation modalities.

Safety vs. Task Success: Both General (language-conditioned) and Single (constraint-specific) filters significantly reduced collision rates compared to the unfiltered nominal policy. In GT settings, filters preserved task performance well. In VL settings, collision rates remained higher, attributed to the loss of spatial features during token aggregation (mean pooling).
General vs. Specialized Filters:
- In Stack Blocks (sequential constraints), the General filter outperformed Single filters in Order-Correctness (OC), suggesting that sharing a filter allows for knowledge transfer across related constraints (e.g., "grasp red" vs. "grasp blue").
- In Safe Grab and Safe Wipe (collision avoidance), results were mixed. The General filter achieved lower collision rates than Single filters in the VL setting but sometimes at the cost of lower task success rates. In GT settings, the General filter required significantly fewer interventions (lower IR) than Single filters while maintaining comparable safety.
Constraint Specificity: The General filter demonstrated the ability to distinguish between specified obstacles and benign objects. In scenes with two objects, the filter intervened to avoid the named obstacle while allowing interaction with the unnamed object, proving it does not indiscriminately avoid all objects.
Out-of-Distribution (OOD) Generalization: The General filter exhibited partial transfer to unseen constraints:
- Safe Grab/Wipe: Performance degraded when tested on unseen object types (e.g., moka pot, yellow book), with collision rates increasing.
- Stack Blocks: The filter generalized well to unseen colors (Color-Only shift) but performance dropped when both shape and color were unseen. Analysis suggested the filter relies on both color and shape cues; when the prompt specified a shape absent from the scene (e.g., "cube" when only cylinders were present), the filter could still identify the target based on color, though with reduced accuracy.
VLM Reliability: The evaluation of 11 VLMs as failure oracles revealed that while some models (e.g., Gemini Robotics-ER 1.6) achieved high non-failure accuracy, they often failed to detect true safety violations (low failure accuracy). This indicates that current VLMs are not yet reliable enough to serve as direct runtime failure oracles but may serve as weak annotators for training learned failure functions.

Significance and Claims

The paper claims to demonstrate a path toward general safety filters that can adapt to diverse, language-specified safety requirements without requiring a separate model for every specification. The authors argue that a single learned safety critic can represent multiple constraints, compose active constraints from a single instruction, and generalize to unseen constraint instances within a family.

The work suggests that safety filters can undergo a similar transition to generalist VLA policies: moving from fixed, specification-specific monitors to general, specification-conditioned enforcement mechanisms. This could enable safety filters to be trained at scale and adapted for downstream tasks, analogous to the development of generalist policies.

However, the authors remain modest about the current state of the technology. They acknowledge that:

The General filter does not guarantee constraint satisfaction, particularly under OOD conditions.
Performance degrades when constraints are not seen during training.
The current approach is limited to specific types of constraints (avoid/approach) and does not yet handle complex temporal logic or fully arbitrary safety specifications.
Current VLMs are insufficient as direct runtime oracles for defining failure sets.

The paper concludes that while the proposed framework shows promise for partial transfer and generalization, further research is needed to improve token aggregation strategies, jointly fine-tune VLA backbones with safety critics, and develop more robust methods for handling ambiguous or complex language constraints.