HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task

1. Problem Statement

Vision-Language Models (VLMs) are increasingly used as decision-makers for embodied agents (robots) to execute complex household instructions. However, they face critical safety vulnerabilities in real-world deployment:

• Contextual Risks: While VLMs can identify explicit malicious commands, they struggle with implicit contextual risks. These occur when a benign instruction becomes hazardous due to subtle environmental states (e.g., "heat food in the microwave" is safe with a ceramic bowl but dangerous with a metal one).
• Limitations of Existing Safeguards:
  • Rule-based methods: Lack scalability in object-dense scenes due to the combinatorial explosion of potential interactions and the difficulty of manual predicate engineering.
  • Model-based methods (Prompt Engineering): Suffer from unfocused perception. In cluttered environments, VLMs often overlook subtle risk cues (e.g., metal cutlery on a plate) or hallucinate dangers based on coarse object presence, leading to either missed hazards or excessive "oversafety" (false positives).

2. Methodology

The authors propose HomeGuard, an architecture-agnostic, plug-and-play safeguard designed to bridge high-level semantic safety with low-level visual grounding.

A. Core Mechanism: Context-Guided Chain-of-Thought (CG-CoT)

HomeGuard introduces a structured reasoning protocol that forces the model to perform active perception before semantic judgment. Instead of holistic reasoning, the process is decomposed into four sequential steps:

1. Instruction Intent Screening: Detects inherently malicious motives.
2. Interaction Targets Inspection: Identifies and localizes the specific objects the agent must manipulate (e.g., the food container).
3. Environmental Constraints Analysis: Identifies background objects or spatial relationships that impose safety constraints (e.g., a power cord near water, or a flammable object near a heat source).
4. Integrated Risk Assessment: Synthesizes the visual evidence from steps 2 and 3 with the instruction to determine the final safety status.

Key Innovation: The model is required to output Visual Anchors (bounding boxes) for both interaction targets and constraint areas before making a safety judgment. This ensures the reasoning is traceable to specific perceptual evidence.

B. Training Strategy

A two-stage training paradigm is employed to instill this reasoning structure and refine grounding capabilities:

1. Supervised Fine-Tuning (SFT): Uses a curated dataset to teach the model the 4-step CG-CoT structure and safety principles. Low-Rank Adaptation (LoRA) is used to preserve general capabilities.
2. Reinforcement Fine-Tuning (RFT): Utilizes Group Relative Policy Optimization (GRPO) with a composite reward function. Crucially, this stage introduces Process Rewards that explicitly supervise the intermediate grounding steps:
   • Outcome Rewards: Format compliance, safety accuracy, semantic consistency, and principle alignment.
   • Process Rewards (Grounding Precision): Uses Intersection over Union (IoU) metrics to reward the model for accurately localizing the target and constraint bounding boxes in the reasoning chain. This enforces precise visual attention.

C. Dataset: HomeSafe

To support this training, the authors created HomeSafe, the first dataset designed for visually grounding hazardous contexts.

• Scale: ~10,257 unsafe scenarios and ~5,710 safe scenarios.
• Generation: Uses an edit-based synthesis pipeline on real-world images (CA-1M, SUNRGBD) to create counterfactual pairs (safe vs. unsafe versions of the same scene) while preserving environmental fidelity.
• Annotations: Includes fine-grained 2D bounding boxes for targets and constraints, object states, and step-by-step Chain-of-Thought reasoning traces.

3. Key Contributions

1. Context-Guided Chain-of-Thought (CG-CoT): A novel reasoning framework that decomposes risk assessment into active visual grounding followed by semantic judgment, effectively mitigating hallucinations and missed risks.
2. Process-Reward Reinforcement Learning: A training strategy that uses spatial grounding (IoU) as a reward signal to enforce precise intermediate perception, a significant departure from standard outcome-only RL.
3. HomeSafe Dataset: A large-scale, high-quality dataset with counterfactual safe/unsafe pairs and dense visual-textual annotations for embodied safety.
4. Actionable Safety Outputs: The model generates not just a safety label, but spatial waypoints (bounding boxes) that can be directly used by downstream planners for collision avoidance and trajectory generation.

4. Experimental Results

Experiments were conducted on HomeSafe-Bench (a held-out test set) and four public benchmarks (EARBench, MSSBench, PaSBench, SafeAgentBench).

• Risk Identification: HomeGuard-8B achieved a Risk Identification Rate (RIR) of 90.98%, outperforming the best proprietary model (Gemini-3-Pro, 87.45%) and open-source baselines.
• Grounding Accuracy: It significantly improved Target IoU (0.72) and Constraint IoU (0.56), demonstrating superior ability to locate specific risk-critical regions compared to base VLMs.
• Reduced Oversafety: HomeGuard reduced the Oversafety Rate (OR) to 13.14%, a significant improvement over baselines (e.g., 25.23% for Gemini-3-Pro), meaning it avoids false alarms in safe scenarios.
• Efficiency: Despite the complex reasoning, HomeGuard achieved a 4.38x efficiency gain over standard long-thinking models by focusing attention only on relevant regions.
• Downstream Utility: When integrated into a VLM planner (IS-Bench), HomeGuard improved the Safe Success Rate from 29.54% to 45.65%, proving its ability to generate actionable safety constraints for trajectory planning.

5. Significance

HomeGuard addresses a critical bottleneck in the deployment of embodied AI: the gap between abstract safety knowledge and physical reality. By enforcing visual grounding as a prerequisite for reasoning, it solves the "unfocused perception" problem that plagues current VLMs.

• Generalizability: The architecture-agnostic design allows it to be applied to various robotic platforms (mobile manipulators, arms) without retraining the core robot controller.
• Safety & Trust: It reduces both physical harm (by catching subtle risks) and operational friction (by reducing false refusals).
• Future Impact: The generated visual anchors serve as a bridge between high-level planning and low-level control, enabling robots to perform explicit collision avoidance and safe trajectory generation, paving the way for safer real-world household robots.