💬 NLP

HomeSafeBench: A Benchmark for Embodied Vision-Language Models in Free-Exploration Home Safety Inspection

This paper introduces HomeSafeBench, the first benchmark for embodied agents to perform free-exploration home safety inspection via egocentric views, and proposes CueBack, a data-construction method that significantly improves Vision-Language Model performance by leveraging the temporal structure of hazard detection to close the gap between AI and human inspectors.

Original authors: Jiashu Yao, Haoyu Wen, Siyuan Gao, Yuhang Guo, Zeming Liu, Heyan Huang

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiashu Yao, Haoyu Wen, Siyuan Gao, Yuhang Guo, Zeming Liu, Heyan Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can see the world through its eyes and talk to you about what it sees. This is the world of Embodied AI, where computers aren't just sitting on a screen; they are "bodied" agents that can move around, look at things, and make decisions in a 3D world. Think of them like a digital explorer with a backpack, ready to solve puzzles or find lost items. But there's a catch: these robots only see what's directly in front of their "face" (called an egocentric view). They can't see behind the sofa or under the rug unless they physically walk over there and look. This makes them great at following instructions, but terrible at being proactive detectives.

Now, imagine you want this robot to be a safety inspector for your house. Its job is to wander through your home, spot dangerous things like a knife left on a counter or a rug that could make someone trip, and tell you about them before anyone gets hurt. This is a huge deal because most home accidents happen because we forget to look at the obvious dangers. The big question researchers are asking is: Can these AI explorers actually do this job, or are they too clumsy and blind to the risks?

The Paper: HomeSafeBench and the "CueBack" Trick

The researchers behind this paper, HomeSafeBench, decided to build a giant, digital obstacle course to test exactly this. They created a benchmark (a standard test) where an AI agent has to freely explore a fully interactive 3D home, looking around and moving from room to room, just like a human would. The goal? To find five specific types of dangers: fire hazards (like paper near a stove), electric shocks (like a toaster in a sink), falling objects (like a heavy box on a high shelf), trip hazards (like a banana peel on the floor), and child safety risks (like sharp knives within a kid's reach).

They built 1,000 different scenarios, each with one to five hidden dangers, and asked the best AI models in the world to find them. The results were a bit of a reality check. Even the smartest commercial AI models, which usually ace these kinds of tests, struggled mightily. The best model managed to find only about 34.7% of the dangers (measured by a score called F1), while a human inspector would have found 98.0%.

Here is the weird part: the AI wasn't just "missing" things randomly. It was being overly cautious. It had a high "precision" (when it said something was dangerous, it was usually right) but a terrible "recall" (it missed almost everything else). It was like a security guard who only screams "Fire!" when the building is already burning, but ignores the smoking candle on the table. The AI was only spotting the most obvious, loud dangers and ignoring the quiet, sneaky ones, especially things that could fall or hurt a child.

The Solution: "CueBack"

The researchers realized that training these robots by making them walk around in the simulator over and over again was too slow and expensive. Instead, they came up with a clever, low-cost trick called CueBack.

Think of it like this: Imagine you are watching a video of a detective solving a crime. In the video, the detective walks into a room, sees a muddy footprint (the clue), and then walks over to the window to confirm the suspect was there (the confirmation). If you just showed the detective the final answer ("The suspect was at the window!"), they wouldn't learn how to find the clue in the first place.

The CueBack method looks at a "perfect" path where a human (or a super-smart teacher) already knows where the danger is. It then rewinds the video to the very first moment the danger became visible from a distance. It tells the AI: "Hey, before you knew for sure, you were just exploring. But right here, you saw a hint. From this point on, you should be looking closer to confirm it."

By teaching the AI to recognize these "clues" before the "confirmation," they created a new set of training data without needing to run expensive simulations. They took a small, open-source AI model (called Qwen3-VL-4B) and taught it using this new method.

The Results

The results were surprisingly good. After this "CueBack" training, the small AI model's performance jumped from a weak 18.7% to a strong 45.3%. This is a big deal because it means this small, free model actually became better at finding home dangers than the most expensive, closed-source commercial models (which only reached 34.7%).

The study suggests that the main problem wasn't that the AI couldn't move or see; it was that it didn't know how to look for subtle risks. By teaching it to spot the early hints and then investigate, the researchers showed that you don't need a super-computer to build a good home safety inspector; you just need the right way to teach it how to pay attention. The team has released their test, the training data, and the code so anyone can try to build even better safety robots in the future.

Technical Summary: HomeSafeBench and CueBack

Problem Definition

Home safety hazards, such as flammable objects near heat sources or sharp tools within a child's reach, are a leading cause of preventable domestic injuries. While recent Vision-Language Models (VLMs) have enabled embodied agents to perform tasks like navigation and visual exploration, automating home safety inspection remains a significant challenge. Existing benchmarks for household safety either abstract scenes into text (discarding visual cues) or restrict agents to fixed viewpoints, failing to capture the inherently active nature of real-world inspection. In reality, an inspector must navigate a partially observable 3D environment, adjust viewpoints, and identify unknown hazards based solely on egocentric visual feedback.

Current state-of-the-art VLMs struggle with this task, exhibiting a systematic deficiency in hazard recognition. Evaluations show that even the best models achieve only ~34.7% F1 score compared to 98.0% for human inspectors, with a specific tendency to under-report hazards (low recall) while maintaining high precision. This suggests models overlook subtle risks, particularly falling-object and child-safety hazards.

Methodology

1. HomeSafeBench Benchmark

The authors introduce HomeSafeBench, the first benchmark for free-exploration home safety inspection with egocentric visual feedback.

Environment: Built on the VirtualHome simulator, it places an embodied agent in fully interactive 3D homes.
Task: The agent must navigate, adjust its viewpoint, and report hazards purely from rendered first-person views.
Scope: Covers five hazard categories: Fire, Electric Shock, Falling Object, Trip, and Child Safety.
Scale: Comprises 1,000 human-validated inspection tasks containing 3,000 hazard instances (1–5 hazards per task).
Construction: A four-stage pipeline involving human annotation of hazard locations and object attributes, rule-based single-hazard generation, VLM-based filtering for visual clarity and semantic validity, and data-level task composition.
Evaluation: Uses a VLM judge for hazard-level matching (requiring category and object instance alignment) to compute Precision, Recall, and F1.

2. CueBack: Offline Data Construction

To address the performance gap without the prohibitive cost of online reinforcement learning in a heavy Unity-based simulator, the authors propose CueBack, a lightweight, simulator-free offline data-construction method.

Core Insight: Inspection follows a "clue-precedes-confirmation" structure. A hazard is often hinted at from a distance (a visual clue) before being confirmed up close.
Process:
1. Trajectory Generation: The authors generate privileged action trajectories (where the agent knows hazard locations) and synthesize executable navigation paths using a room-level directed observation graph.
2. Split-Point Identification: For each hazard, the method backtracks along the trajectory to identify the earliest frame ( $k_h$ ) where the hazard clue becomes visible to the agent.
3. Reasoning Rewriting: The supervision signal is rewritten to respect this temporal structure:
  - Before $k_h$ (Exploration): Reasoning justifies navigation as broad coverage or searching for high-risk regions without naming the specific unseen hazard.
  - After $k_h$ (Confirmation): Reasoning acknowledges the visible clue, hypothesizes the risk, and justifies approaching for confirmation.
  - Reporting: Justified only when visual evidence is sufficient.
Goal: This prevents "privileged information leakage," ensuring the model learns to recognize hazards from observations rather than memorizing trajectories.

Key Contributions

HomeSafeBench: The first benchmark for free-exploration home safety inspection, featuring 1,000 tasks across five hazard categories with human validation.
Comprehensive Evaluation: A systematic analysis of state-of-the-art commercial and open-source VLMs, revealing a large performance gap with humans and a systematic tendency to under-report hazards (low recall).
CueBack Method: A novel offline data-construction technique that exploits the clue-precedes-confirmation structure of inspection. It enables a small 4B-parameter open-source model to surpass leading closed-source models on out-of-distribution tasks.
Open Release: The benchmark, training dataset, and code are publicly available.

Experimental Results

The authors evaluated a broad range of models on HomeSafeBench:

Baseline Performance: The best closed-source model (Doubao-Seed-2.0-Pro) achieved an F1 of 34.7%, while the best open-source model (Qwen3-VL-4B) scored 18.7%. Human performance was 98.0%.
Systematic Failure: Across all models, Precision significantly exceeded Recall (e.g., 58.9 vs. 17.8 for GLM-5V-Turbo), indicating models flag only obvious cases (like trip hazards) but miss subtle ones (falling objects, child safety).
CueBack Performance: Fine-tuning a Qwen3-VL-4B model on CueBack-constructed data raised the average F1 from 18.7% to 45.3% on an out-of-distribution test set.
- This result surpasses the strongest closed-source model (34.7%) by a significant margin (+10.6%).
- It also outperforms other baselines like SFT-Action (37.8%) and SFT-Both (39.3%), demonstrating the efficacy of the clue-precedes-confirmation reasoning structure.
- The method showed substantial gains on the most difficult categories, such as Child Safety (F1 from 16.6% to 31.4%) and Electric Shock (27.6% to 39.7%).

Significance

The paper claims that HomeSafeBench reveals a critical bottleneck in current embodied AI: a lack of specialized risk recognition rather than a lack of navigation capability. The significance of the work lies in demonstrating that simulator-free, offline data construction can effectively close the gap between small open-source models and large commercial systems for safety-critical tasks. By leveraging the natural temporal structure of inspection (clue before confirmation), CueBack enables agents to learn hazard awareness without the computational cost of online reinforcement learning, making high-performance home safety inspection more accessible and deployable.