💻 computer science

See Something, Say Something: Context-Criticality-Aware Mobile Robot Communication for Hazard Mitigations

This paper presents a context-criticality-aware framework for autonomous mobile robots that leverages VLM/LLM-based perception to dynamically adapt hazard communication urgency based on situational factors, thereby reducing response times and increasing user trust compared to fixed-priority baselines.

Original authors: Bhavya Oza, Devam Shah, Ghanashyama Prabhu, Devika Kodi, Aliasghar Arab

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bhavya Oza, Devam Shah, Ghanashyama Prabhu, Devika Kodi, Aliasghar Arab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking down a busy street with a robot companion. Suddenly, the robot spots a sharp knife on the ground.

In the old days, a robot might have screamed, "DANGER! KILLER KNIFE! CALL THE POLICE!" immediately. This would cause everyone to panic, even if the knife was just sitting in a kitchen where a chef was chopping vegetables. The robot would be like a smoke alarm that goes off every time you toast a piece of bread—eventually, people stop listening to it, or they get so scared they freeze.

This paper introduces a smarter robot that acts more like a wise, observant security guard than a frantic alarm clock. It follows the rule: "See something, say something," but with a crucial twist: It figures out what to say and how to say it based on the situation.

Here is how it works, broken down into simple concepts:

1. The "Context is King" Rule

The robot uses advanced "eyes and brains" (AI models) to look at a scene and ask three questions before it speaks:

How bad is this? (Criticality)
How fast do we need to act? (Time Sensitivity)
Who can fix this? (Feasibility)

The Analogy: Think of a fire.

Scenario A: You see a small flame in a fireplace.
- Old Robot: "FIRE! EVACUATE THE BUILDING!" (Panic!)
- New Robot: "I see a fire in the fireplace. It's under control. No action needed." (Calm.)
Scenario B: You see a small flame in a pile of dry leaves in a hallway.
- Old Robot: "FIRE! EVACUATE THE BUILDING!" (Panic, but maybe too late or too loud.)
- New Robot: "Fire detected in the hallway. It's spreading fast. Call the fire department immediately!" (Urgent, specific, and helpful.)

The object (the fire/knife) is the same, but the story around it changes everything.

2. The Robot's "Brain" (The AI)

The robot doesn't just recognize objects; it understands the story of the room.

It uses a Vision-Language Model (VLM) to "see" the picture and describe it (e.g., "A knife on a cutting board next to a chef").
It uses a Large Language Model (LLM) to think about that description (e.g., "Is this dangerous? No, it's cooking. Okay, I'll just say 'Hello' or stay quiet.").

If the robot sees a knife in a kitchen, it thinks, "Ah, dinner time. Low risk." It might just say, "I see you are cooking," or stay silent.
If it sees the same knife in a school hallway, it thinks, "Oh no! Someone could get hurt. High risk!" It immediately triggers a loud alarm and texts the security team.

3. The "Volume Knob"

The robot has a built-in "volume knob" for its voice.

Low Risk: It speaks softly, like a polite inquiry. "Is everything okay?" or "I noticed some trash."
Medium Risk: It speaks with a clear, alert tone. "Please be careful, there is a wet floor."
High Risk: It sounds urgent and authoritative. "EMERGENCY! MOVE AWAY! HELP IS COMING!"

This prevents "Alarm Fatigue." If a robot screams "DANGER!" every time it sees a toy gun or a piece of trash, people will ignore it when a real danger appears. By saving the screaming for real emergencies, people actually listen when it matters.

4. The Results: Trust and Speed

The researchers tested this robot over 60 times in real buildings (hallways, kitchens, etc.).

The Result: People trusted the robot 82% of the time.
The Comparison: When they used a "dumb" robot that treated every object the same, people trusted it much less because it was either too scary or too slow.
The Speed: Because the robot knew exactly who to call (the janitor for trash, the police for a weapon, the nurse for a fall), it got help faster.

The Big Picture

This paper is about teaching robots emotional intelligence and situational awareness.

Instead of being a robot that blindly follows a rulebook ("If Knife = Alarm"), it is a robot that understands the nuance of life. It knows that a knife in a kitchen is a tool, but a knife in a hallway is a threat. By adjusting its message to fit the situation, it keeps people safe without scaring them unnecessarily.

In short: It's the difference between a robot that is a loud, annoying siren and a robot that is a calm, reliable partner who knows exactly when to whisper and when to shout.

1. Problem Statement

Autonomous Mobile Robots (AMRs) are increasingly deployed in public and industrial spaces where they must act as proactive safety agents. However, current communication systems often rely on fixed-priority, context-free alerts. This leads to two critical failure modes:

Over-escalation: Triggering high-intensity alarms for low-risk situations (e.g., a knife in a kitchen), causing "alarm fatigue" and eroding user trust.
Under-escalation: Failing to recognize the urgency of a hazard due to a lack of contextual understanding (e.g., a knife in a public corridor vs. a kitchen).

The core problem is the lack of a systematic framework that allows robots to assess criticality, time sensitivity, and mitigation feasibility to generate adaptive, context-aware communication. Without this, robots cannot effectively coordinate with humans or emergency responders, increasing the time to action and the risk of damage.

2. Methodology

The authors propose a four-step sequential decision framework grounded in crisis management theory, implemented using Vision-Language Models (VLMs) and Large Language Models (LLMs).

A. Formal Problem Formulation

The system maps an observation $O_t$ to a communication tuple $c_t$ through four steps:

Hazard Detection: Identifying the hazard category ( $a_t$ ) from multimodal observations (RGB image, attention maps).
Contextual Factor Extraction: Extracting a triple $\theta_t = (d_t, \tau_t, \phi_t)$ $θ_{t} = (d_{t}, τ_{t}, ϕ_{t})$ :
- Criticality Level ( $d_t$ ): Severity of the hazard in its specific environment (Low, Medium, High).
- Time Sensitivity ( $\tau_t$ ): Urgency of the response (Immediate, Soon, Near Future).
- Feasibility of Mitigation ( $\phi_t$ ): Who can act (Robot, Person of Contact, External Help).
Risk Assessment: Computing an overall criticality score ( $k_t$ ) based on the factors and environmental context (e.g., crowd density).
Criticality-Aware Communication: Generating a message tuple $M_t = (m_t, \gamma_t, \chi_t)$ $M_{t} = (m_{t}, γ_{t}, χ_{t})$ where:
- $m_t$ : The text content.
- $\gamma_t$ : Tone intensity (0–10 scale).
- $\chi_t$ : Character type (Inquiry, Alert, Urgent).
- Constraint: Audible alarms are only triggered for Medium/High criticality to prevent fatigue.

B. System Architecture & Implementation

Hardware: A patrolling mobile robot equipped with an NVIDIA Jetson Nano, Intel RealSense RGB-D camera, LiDAR, microphone, and speaker.
Software Stack: Built on ROS 1 Melodic with an edge-to-cloud pipeline.
- Perception: Uses Grad-CAM to generate visual attention heatmaps and BLIP (Bootstrapped Language-Image Pre-training) to generate scene captions.
- Reasoning: A ChatGPT-4.0 API (specifically gpt-4-0613) fuses the visual heatmap and scene caption to infer the hazard, extract contextual factors, and compute the risk score.
- Output: The LLM generates the message text, tone, and recipient routing (Nearby, Remote, or Coordination).
Latency Management: The system averages ~12 seconds per cycle (mostly cloud API latency). A fallback rule-based policy activates if cloud latency exceeds 20 seconds to ensure safety.

3. Key Contributions

Context-Criticality Framework: A novel formalization of robot communication that moves beyond object detection to contextual risk assessment. It explicitly distinguishes between identical objects in different contexts (e.g., a knife in a kitchen vs. a corridor).
Adaptive Communication Policy: The system dynamically adjusts message tone, urgency, and recipient routing based on a computed criticality score, rather than using static rules.
VLM/LLM Integration: Demonstrates the effective use of foundational models (BLIP + GPT-4) for visual-grounded reasoning, allowing the robot to disambiguate ambiguous scenarios (e.g., distinguishing a toy gun from a real one) based on environmental cues.
Alarm Fatigue Mitigation: Introduces a strict constraint where audible alarms are suppressed for low-criticality events, preserving human trust and preventing panic.

4. Experimental Results

The system was validated through 60+ runs in building corridors and a kitchen setting, comparing the proposed context-aware system against a fixed-priority baseline (which triggers high alerts for specific objects regardless of context).

Detection Accuracy: The context-aware pipeline achieved 80% accuracy, a 10% improvement over the object-level baseline (70%). This was largely due to better disambiguation in ambiguous scenarios.
User Trust: The system achieved 82% user trust, significantly higher than the fixed-priority baseline. Users preferred the context-aware system (78% preference) because it avoided unnecessary panic.
Command Understanding: Users demonstrated 85% understanding of the robot's instructions.
Safety Compliance: The system strictly adhered to the alarm constraint (Eq. 13), activating alarms only for Medium/High criticality ( $\rho_t \geq 5$ ).
Overall Effectiveness ( $\epsilon$ ): The system achieved an aggregate effectiveness score of 0.81, balancing detection accuracy, message appropriateness, coordination success, and latency.

5. Significance

This paper addresses a critical gap in autonomous safety: how a robot communicates is as important as what it detects.

Trust & Adoption: By preventing "cry wolf" scenarios (false alarms), the framework fosters greater human trust in AMRs, which is essential for their widespread deployment in public spaces.
Efficiency: It optimizes the "time to action" by ensuring the right message reaches the right person (e.g., a maintenance worker for trash vs. emergency services for a fallen person) with the appropriate urgency.
Scalability: The use of foundational models (VLM/LLM) suggests a path toward generalizable safety agents that can handle novel hazards without retraining specific rule sets for every scenario.

Future Work: The authors plan to explore factor ablation studies, formal verification of the criticality function, migration to ROS 2, and the use of Partially Observable Markov Decision Processes (POMDP) for more robust policy optimization.