🤖 AI

Autonomous VR-Based Risk Detection for Situational Awareness in Dangerous Settings

This paper presents a VR-based human-robot interaction framework that leverages Vision Language Models to identify and annotate hazards in simulated dangerous environments, demonstrating that this approach significantly enhances operator situational awareness, clarity, and comfort compared to unannotated baselines.

Original authors: Mohammad Eskandari, Murali Krishna Varma Indukuri, Stephanie M. Lukin, Cynthia Matuszek

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad Eskandari, Murali Krishna Varma Indukuri, Stephanie M. Lukin, Cynthia Matuszek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship, but instead of steering through the stars, you are navigating a chaotic, dangerous disaster zone. You can't be everywhere at once, and your eyes can only look in one direction at a time. This is the reality for first responders, factory safety inspectors, and anyone working in high-risk environments. They need "situational awareness"—a superpower that lets them know exactly what's dangerous, where it is, and what to do about it, all without getting overwhelmed. To build this superpower, scientists are mixing two exciting technologies: robots that can see and think, and virtual reality (VR) headsets that let humans step inside a digital world. Think of robots as brave, tireless scouts that can walk into a crumbling building or a toxic spill zone. Think of VR as a magic window that lets a human operator stand safely in their office while seeing exactly what the robot sees. The big question researchers are asking is: Can we teach these robot scouts to not just see the danger, but to explain it to us in a way that feels natural, clear, and helpful?

This paper, titled "Autonomous VR-Based Risk Detection for Situational Awareness in Dangerous Settings," dives into exactly that question. The researchers built a system where a robot (simulated in a virtual world) wanders around a hazardous scene, like a messy workshop after an earthquake. As the robot moves, it takes pictures and sends them to a "Vision-Language Model" (VLM). You can think of a VLM as a super-smart digital brain that has read millions of books and seen millions of images; it can look at a photo and say, "Hey, that gas tank looks unsecured," or "That chemical bottle is in the wrong place." The robot then drops virtual "flags" or pop-up notes in the VR world to mark these dangers. A human wearing a VR headset can then walk through this digital scene, and as they get close to a danger, a little note pops up explaining the risk, and they can even hear the robot "speak" the warning out loud.

The team tested this idea with 27 people in a user study. They asked the participants to explore a virtual makerspace twice: once with no notes (just a messy room) and once with the robot's helpful annotations. The results were promising. The participants overwhelmingly preferred the annotated version, reporting that they felt much more aware of their surroundings and found the system very clear and useful. In fact, the robot's "brain" successfully identified 18 out of 19 hidden hazards in the simulation, missing only one and occasionally making a few mistakes where it thought something was dangerous when it wasn't (like seeing a ghost gas tank that wasn't there). The study suggests that combining these smart robot brains with immersive VR headsets is a powerful way to keep people safe and informed in scary situations. However, the authors are careful to note that this was a simulation; while the results look great, the system still needs to be tested in the real, messy world before it can be fully trusted to save lives.

The Story of the Digital Scout

The Setup: A Robot, a Brain, and a Magic Window
The researchers created a digital playground using a simulator called RIVR. In this world, they set up a "makerspace" (a workshop) that had been shaken up by a fake earthquake. They scattered 19 specific dangers around the room, like blocked fire extinguishers, leaking chemicals, and unsecured gas cylinders. They knew exactly where these were, acting as the "ground truth" or the answer key.

Then, they sent a virtual robot (a Husky UGV, which is like a four-wheeled robot dog) to patrol the room. As the robot drove around, it snapped photos and sent them to a VLM (specifically, a model called GPT-4o). The robot asked the VLM a simple question: "What is dangerous in this picture?" The VLM, acting like a seasoned safety inspector, analyzed the image and replied with a list of hazards and a short explanation.

The Magic Interface
Once the robot finished its patrol, the system took those answers and turned them into "Points of Interest" (POIs) inside the VR world. Imagine walking through a video game where, as you approach a dangerous object, a glowing red pin bounces up and down. When you get close, a panel pops up telling you exactly what the problem is. If the text was too hard to read in the headset, the user could press a button to have the computer read the warning out loud. This was the "annotated" condition.

The participants then put on VR headsets and explored the same room. First, they walked through the "unannotated" version, where the room was just messy with no help. Then, they walked through the "annotated" version with the robot's helpful notes.

What They Found
The results were quite clear. When asked to rate the experience, the participants gave the annotated system very high marks.

Clarity: The average score was 4.6 out of 5. Most people felt the danger markers were very easy to spot.
Usefulness: The system scored 4.58 out of 5 for being useful in improving safety awareness.
Future Use: Participants gave it a 4.5 out of 5, saying they would likely use this kind of system in real life.

Statistically, these scores were significantly higher than a "neutral" rating of 3, meaning the positive feedback wasn't just a fluke; it was a strong trend. In fact, only one person out of 27 preferred the messy room without the notes.

The Robot's Scorecard
The researchers also checked how well the robot's "brain" actually did its job. Out of the 19 hazards planted in the scene:

The system correctly identified 18 of them.
It missed 1 hazard (a cylinder hidden behind another object).
It made 4 "false alarms" (saying something was dangerous when it wasn't). Two of these were "hallucinations" (inventing a danger that wasn't there), and two were mislabeling real dangers (calling a gas cylinder "safe" when it was actually "unsecured").

This means the system caught about 94.7% of the real dangers (recall) but was right about its warnings 81.8% of the time (precision). The authors suggest that the system works well, but it gets confused when objects are crowded together or hidden from view.

Why This Matters (and What It Doesn't Mean Yet)
This paper suggests that using AI to help robots "talk" to humans in VR is a winning combination for safety. It shows that people feel more comfortable and aware when a robot helps them spot dangers. However, the authors are very careful not to say this is a finished product. They point out that this was all done in a computer simulation. The "robot" didn't actually walk through a real earthquake site, and the "dangerous chemicals" were just digital images.

They also note some limitations. The study was small (27 people), and they only tested one specific AI model. They also found that the AI sometimes "hallucinated" dangers, which is a problem if you are relying on it to keep you safe. The researchers propose that future work should try to make the AI explain why it thinks something is dangerous (so humans can trust it better) and test this system in real-world scenarios with actual first responders.

In short, this paper is a hopeful step forward. It shows that if we give robots a smart brain and a VR headset, they can become excellent guides for humans in dangerous places. But before we hand over the keys to the rescue mission, we need to make sure the robot doesn't get confused, doesn't see ghosts, and can handle the messy reality of the real world.

Technical Summary: Autonomous VR-Based Risk Detection for Situational Awareness in Dangerous Settings

Problem Statement
In high-risk environments such as disaster response, industrial safety, and search-and-rescue operations, maintaining situational awareness is critical for preventing accidents and supporting human decision-making. However, human operators often struggle to identify hazards due to fatigue, information overload, or the complexity of the environment. While autonomous robots can assist in these scenarios, their utility is limited by the effectiveness of Human-Robot Interaction (HRI). Specifically, there is a need for systems that not only detect hazards autonomously but also communicate them clearly to human operators without causing cognitive overload. Although Vision-Language Models (VLMs) have demonstrated strong potential for scene understanding, their integration into human-facing robotic systems for safety-critical annotation remains underexplored.

Methodology
The authors present a VR-based HRI framework designed to evaluate how VLM-assisted robots can support situational awareness in simulated hazardous environments. The system pipeline operates as follows:

Environment Simulation: The system utilizes the Robot Interactions in Virtual Reality (RIVR) simulator, integrated with ROS and Unity. The virtual environment is a reconstructed makerspace containing 19 ground-truth hazards established against OSHA General Industry standards. While the study primarily used manually modeled scenes, the framework supports 3D Gaussian Splatting for photorealistic reconstruction from image collections (demonstrated via drone footage of the East Palestine train derailment).
Autonomous Perception: A simulated Husky UGV explores the environment, capturing images at a fixed rate. These images are transmitted to a Vision-Language Model (specifically OpenAI's GPT-4o, model gpt-4o-2024-11-20) via an engineered prompt. The prompt instructs the model to act as a safety inspector, identifying hazards, describing them, and suggesting safety measures, returning the output in a structured JSON format.
Annotation and Interface: Based on the VLM's response, the system generates "Points of Interest" (POIs) in the virtual environment. These POIs are visual markers (transparent red pins) that activate when a user approaches within 1 meter. Upon activation, an information panel displays the hazard description. To address readability issues in VR, the system includes an auditory feature where the hazard description is read aloud using audio generated by the VLM.
User Study: The system was evaluated through an IRB-approved user study with 27 participants. Participants explored the same makerspace environment under two conditions: an unannotated baseline and the fully annotated VLM-assisted version. They rated the system on five dimensions: Clarity, Effectiveness, Trust, Usefulness, and Future Use, using a 5-point Likert scale.

Key Contributions
The paper makes three primary contributions:

Open-Source Annotation System: The development of an open-source system designed to increase situational awareness in virtual high-risk environments using VLM-based robots within the RIVR simulator.
Immersive Interface Evaluation: The presentation and evaluation of an immersive interface featuring virtual markers with auditory message playback, designed to convey safety-critical information intuitively.
Empirical Validation: A user study demonstrating that participants reported greater comfort and improved situational awareness when interacting with the annotated system compared to a baseline.

Results
The study yielded statistically significant positive results:

User Preference: 26 out of 27 participants preferred the annotated environment over the unannotated baseline for safety assessment.
Quantitative Ratings: All five evaluation dimensions were rated significantly above the neutral midpoint (3.0) with $p < .001$ . "Clarity" received the highest mean rating (4.6), followed by "UI Usefulness" (4.58) and "Future Use" (4.50). "Trust" scored 4.16.
Correlation Analysis: A strong correlation (0.70) was found between the clarity of hazard indicators and the likelihood of future use, suggesting that ease of spotting hazards drives adoption.
Hazard Detection Performance: The system successfully identified 18 out of 19 ground-truth hazards (94.7% recall), supporting the hypothesis that VLMs can detect the majority of hazards. The system produced 4 false positives (21.1%) and 1 false negative (5.3%). Failures were attributed to object occlusion and close proximity of objects, which increased error rates.
Model Comparison: In a validation step, GPT-4o significantly outperformed the open-weight Qwen2.5-VL-72B-Instruct model (F1 score of 0.878 vs. 0.39) on the same keyframes.

Significance and Claims
The authors claim that combining VLM-based robotic perception with immersive visualization is a promising approach for supporting situational awareness in hazardous settings. The study suggests that VLMs can effectively serve as a component in HRI systems to autonomously annotate hazards, thereby reducing operator cognitive load and enhancing proactive risk management.

The paper modestly frames these findings as a step toward practical application, noting that while the system performed well in simulation, real-world deployment requires further validation through Sim2Real processes. The authors emphasize that the tool is intended to improve hazard awareness and risk management, particularly in scenarios where rapid identification and prioritization of risks are necessary. They acknowledge limitations, including the small sample size of the user study and the reliance on a single VLM, but posit that the framework provides a foundation for future work involving dynamic environments, active hazard response, and the integration of first-responder personnel in evaluations.

The Story of the Digital Scout

More like this