🤖 AI

Contextual Safety Reasoning and Grounding for Open-World Robots

The paper introduces CORE, a novel safety framework that leverages vision-language models for online contextual reasoning and spatial grounding to enforce adaptive, probabilistically safe robot behaviors in open-world environments without prior environmental knowledge.

Original authors: Zachary Ravichandran, David Snyder, Alexander Robey, Hamed Hassani, Vijay Kumar, George J. Pappas

Published 2026-02-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zachary Ravichandran, David Snyder, Alexander Robey, Hamed Hassani, Vijay Kumar, George J. Pappas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to walk through a busy, unpredictable city.

In the past, robot safety was like giving a robot a strict, pre-printed map with red "Do Not Enter" zones drawn on it. If the robot saw a "Wet Floor" sign, it wouldn't know what to do unless someone had specifically programmed that exact sign into its map beforehand. If the robot walked into a new office or a park, it would be blind to the rules of that specific place.

This paper introduces CORE (Contextual Safety Reasoning and Enforcement), a new way to teach robots to be safe by giving them common sense instead of just a map.

Here is how CORE works, explained through a simple story:

1. The "Smart Eyes" (Vision-Language Model)

Imagine the robot has a pair of super-smart glasses connected to a brain that knows how to speak and understand the world (this is the Vision-Language Model or VLM).

Old Way: The robot sees a red cone. It thinks, "That is an object. I must stop."
CORE Way: The robot sees a line of red cones. Its "brain" looks at the picture and thinks, "Ah, these cones are arranged in a line. That's not just a pile of plastic; it's a barrier. It means the area behind them is a construction zone. I shouldn't just stop; I need to go around."

The robot doesn't need a pre-written rule for "cones." It looks at the scene, reasons about the context (why are these cones here?), and figures out the safety rule on the spot.

2. The "Translator" (Semantic Grounding)

Once the robot's brain understands the rule ("Don't go behind the cones"), it needs to tell its legs (the motors) exactly where to stop.

The Analogy: Imagine the robot is looking at a photo of a room. It sees a "No Entry" zone. The Grounding module is like a translator that takes the sentence "Don't go behind the cones" and draws a virtual invisible wall on the floor in the robot's physical world.
It turns the idea of a safe zone into a physical boundary the robot can measure and respect.

3. The "Reflex" (Control Barrier Functions)

Now the robot has its invisible wall. But what if the robot is moving fast and its "eyes" are a little blurry? What if it misjudges the distance?

The Analogy: Think of this as the robot's reflexes. Even if the robot's brain is thinking, "I think I can squeeze through," the reflex module acts like a seatbelt and airbag. It mathematically guarantees that no matter how fast the robot is going or how fuzzy the image is, it will physically stop before hitting the invisible wall.
It accounts for uncertainty. If the robot isn't 100% sure where the wall is, the reflexes make the wall slightly bigger to be safe, ensuring the robot never crashes.

Why is this a big deal?

The "Open World" Problem: Real life is messy. You can't program a robot with every possible "Wet Floor" sign, "Crowded Hallway," or "Fragile Vase" scenario it might ever see.
The CORE Solution: CORE allows the robot to walk into a brand new environment (like a stranger's house or a busy hospital) and immediately understand the safety rules just by looking at the pictures. It learns the rules while it walks.

The "Safety Net"

The researchers also proved mathematically that even if the robot's "brain" makes a mistake or the camera is blurry, the "reflex" system is strong enough to keep the robot safe with a very high probability (like 90%+).

In summary:
Before, robots were like tourists with a rigid guidebook who got lost if the scenery changed.
With CORE, robots are like experienced locals who can look at a street, see a "Wet Floor" sign, understand that the floor is slippery, and know to walk carefully, all without needing a map. They use eyes to see, a brain to reason, and reflexes to stay safe.

1. Problem Statement

Robots operating in open-world environments (homes, offices, warehouses) face a critical safety challenge: contextual variability. Traditional safety methods rely on fixed, pre-defined geometric constraints (e.g., "avoid obstacles within 1 meter") or static safety specifications provided by experts. However, real-world safety is often semantic and context-dependent.

Examples: A "wet floor" sign is not a physical obstacle but indicates a hazard zone; a line of traffic cones creates a prohibited area even if individual cones are small; a robot must respect social norms by keeping a buffer around people.
The Gap: Existing systems cannot infer these rules online without prior maps or explicit user instructions for every possible scenario. They lack the ability to reason about why a situation is dangerous and translate that reasoning into immediate control constraints.

2. Methodology: The CORE Framework

The authors propose CORE (Contextual Safety Reasoning and Enforcement), a three-stage framework that enables robots to operate safely in unknown environments without prior knowledge of safety specifications.

Stage 1: Contextual Safety Reasoning (VLM-Based)

Mechanism: Uses a Vision-Language Model (VLM) to analyze real-time visual observations (RGB-D camera).
Process: The VLM is prompted with a structured reasoning process to identify semantic classes (e.g., wet_floor_sign, person, traffic_cone) and apply specific spatial operators to define safety regions:
- ON: Traversable surfaces (Safe).
- NEAR: Immediate collision buffers for solid objects (Unsafe).
- AROUND: Semantic danger zones or social buffers (Unsafe).
- BETWEEN: Prohibited areas formed by the arrangement of objects (e.g., a line of cones).
Output: The VLM generates a JSON object containing safety-relevant semantics, a chain-of-thought justification, and a list of safe/unsafe predicates.

Stage 2: Semantic Grounding

Mechanism: Maps the abstract VLM predicates into the robot's physical coordinate system.
Process:
1. Segmentation: An open-vocabulary segmentation model (SAM3) identifies the pixels corresponding to the semantic classes in the image.
2. Spatial Logic Application: The operators are applied to the pixel masks (e.g., AROUND applies morphological dilation; BETWEEN computes the convex hull).
3. 3D Projection: The resulting 2D image-space safe/unsafe sets are projected into a 3D point cloud using depth data and accumulated into a 2D grid map.
4. Barrier Construction: A barrier function $h(x)$ is constructed as a signed distance field over the safe set, accounting for perception uncertainty (probability of detection).

Stage 3: Probabilistic Safety Enforcement

Mechanism: Uses Control Barrier Functions (CBFs) to synthesize safe control inputs.
Process: The controller solves a Quadratic Program (QP) to minimally modify a nominal control input ( $u_{nom}$ ) such that the robot remains within the safe set defined by $h(x) \geq 0$ .
Uncertainty Handling: The framework provides probabilistic safety guarantees ( $1-\delta$ ) despite perceptual uncertainty. It models the probability of detecting an unsafe region and uses a "safe traversal" theorem to ensure that even with imperfect perception, the robot is unlikely to enter an undetected hazard.

3. Key Contributions

VLM-Based Reasoning Module: A novel approach to infer context-dependent safety constraints directly from visual observations without a priori environmental knowledge.
Semantic Grounding Method: A pipeline that translates natural language safety predicates into spatially-defined safe sets using segmentation and geometric projection.
Probabilistic CBF Enforcement: A control synthesis mechanism that guarantees safety with high probability under perception uncertainty, bridging the gap between high-level semantic reasoning and low-level continuous control.
Theoretical Guarantees: A formal proof (Theorem 1) establishing bounds on safe traversal rates based on the robot's perception latency and detection probability.

4. Experimental Results

The authors evaluated CORE in both simulation (NVIDIA Isaac Sim) and real-world hardware (Boston Dynamics Spot robot) across diverse environments (warehouses, hospitals, residential areas).

Performance vs. Baselines:
- CORE achieved a 96.6% success rate on safe tasks and 93.3% on unsafe tasks (preventing violations).
- Oracle (Ground Truth): 100% / 96.6% (CORE performs nearly as well as a system with perfect prior knowledge).
- No Context (Offline LLM): 93.3% safe / 16.6% unsafe (Failed to prevent violations because it lacked online reasoning).
- Geometric Only: 100% safe / 0% unsafe (Failed to stop robots from entering contextually dangerous areas like "wet floors" or "restricted zones").
Ablation Studies:
- Removing the Chain-of-Thought (CoT) reasoning from the VLM significantly degraded performance (safe region detection dropped to 61%), proving that explicit reasoning is crucial.
- Removing Safety Logic instructions reduced performance, highlighting the need for structured prompts.
- Gemma 3 27B was identified as the optimal VLM, balancing accuracy and latency (4.1s) compared to larger or smaller models.
Hardware Deployment: CORE successfully operated on a Spot robot, preventing it from entering restricted zones marked by cones and avoiding social buffers around people, though it faced some challenges with depth noise in real-world grounding.

5. Significance

This paper represents a significant step toward truly autonomous open-world robots.

Shift from Static to Dynamic Safety: It moves away from the paradigm of "hard-coded safety rules" toward "dynamic safety reasoning," allowing robots to adapt to novel situations they have never seen before.
Bridging Semantics and Control: It successfully integrates high-level semantic understanding (VLMs) with rigorous low-level control theory (CBFs), solving the "grounding" problem that often plagues AI-driven robotics.
Scalability: By not requiring pre-built maps or manual specification of every hazard, the framework is scalable to complex, unstructured environments where human supervision is limited.
Safety Guarantees: The inclusion of probabilistic safety guarantees under uncertainty addresses a major criticism of using large AI models in safety-critical applications, providing a mathematical framework for risk management.

In summary, CORE demonstrates that robots can learn to "understand" safety contexts in real-time, reason about them using VLMs, and enforce them with mathematically rigorous control, making them viable for deployment in unpredictable human-centric environments.