💻 computer science

Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Models

This paper presents a multimodal explainability module that integrates vision-language models and heat maps to enable autonomous mobile robots to articulate their navigation decisions in natural language, thereby enhancing user trust and understanding in dynamic social environments.

Original authors: Oluwadamilola Sotomi, Devika Kodi, Aliasghar Arab

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oluwadamilola Sotomi, Devika Kodi, Aliasghar Arab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking down a busy hallway, and a robot is trying to get past you. In the old days, the robot would just stop abruptly or swerve around you without saying a word. You'd be left thinking, "Why did it stop? Is it broken? Is it going to hit me?" That silence creates confusion and a lack of trust.

This paper is about teaching robots to talk back (in a friendly, helpful way) so you know exactly what they are thinking.

Here is the breakdown of their idea, using some everyday analogies:

1. The Problem: The "Black Box" Robot

Think of a traditional robot like a magic 8-ball. You shake it, it gives you an answer (it moves), but you have no idea how it decided that answer. If the robot suddenly stops in front of you, you don't know if it's because it saw a person, a dog, or a glitch. This uncertainty makes people nervous and hesitant to work with robots.

2. The Solution: The "Chatty Navigator"

The authors built a special "brain add-on" for their robot that acts like a tour guide with a camera. Instead of just moving silently, the robot now:

Sees what's happening (using a camera).
Thinks about what it sees (using AI to spot people or obstacles).
Explains its thoughts out loud (using natural language).

The Analogy: Imagine a self-driving car that doesn't just brake, but says, "I'm slowing down because there's a dog playing near the curb, and I want to make sure it's safe." That's what this robot does.

3. How It Works (The Three-Step Dance)

The robot uses three tools working together, like a team of experts:

The Eyes (Camera & Heatmaps): The robot looks at the world and highlights the "hot spots" (like where a person is standing) using a red heat map, just like a thermal camera in a spy movie.
The Translator (BLIP): This part looks at the picture and writes a simple caption, like "A person is walking toward me."
The Storyteller (LLM): This is the robot's voice. It takes the picture and the caption and turns them into a friendly sentence.
- Input: "Person detected at 2 meters."
- Output: "I see a person walking toward me, so I'm taking a small detour to give them space."

4. The Experiment: Testing Trust

The researchers put this robot to the test with 30 real people in a hallway. They ran two scenarios:

The Silent Robot: It moved around, but said nothing.
The Chatty Robot: It moved around and explained its decisions in real-time.

The Results:

Trust Skyrocketed: When the robot explained itself, people trusted it much more. It felt less like a scary machine and more like a helpful colleague.
Less Confusion: People felt more in control and understood why the robot was doing what it was doing.
Better Behavior: The robot actually made fewer sudden, jerky stops when it was "thinking out loud," because the system was designed to be smoother and more socially aware.

5. The Catch: The "Thinking Time"

There is one small hurdle. Because the robot has to take a picture, analyze it, and write a sentence, it takes a little bit of time (about 20 seconds on average in their test).

The Analogy: It's like asking a friend for directions. If they take 20 seconds to think, you might get impatient. The researchers admit they need to make the robot "think" faster so the explanation arrives while the robot is still moving, not after it has already stopped.

The Big Takeaway

This paper proves that robots don't just need to be smart; they need to be transparent.

By giving robots a voice to explain their decisions, we turn them from mysterious, unpredictable machines into trustworthy partners. It's the difference between a stranger who suddenly grabs your arm and a friend who says, "Hey, let's move this way so we don't bump into that table."

In the future, for robots to live and work alongside us in hospitals, offices, and homes, they won't just need to be safe; they'll need to be honest about why they are safe.

1. Problem Statement

As Autonomous Mobile Robots (AMRs) are increasingly deployed in dynamic social environments (e.g., hospitals, offices, homes), a critical gap exists in Human-Robot Interaction (HRI). Traditional AMRs often operate as "black boxes," making decisions without communicating their reasoning. This lack of transparency leads to:

Reduced Trust: Users cannot predict robot behavior, leading to hesitation or avoidance.
Social Friction: Robots may violate social norms (e.g., interrupting conversations, invading personal space) because they lack high-level social reasoning.
Limited Explainability: Existing Explainable AI (XAI) approaches in robotics often focus on internal logic or static post-hoc analysis, failing to provide real-time, human-readable, multimodal explanations that combine visual context with natural language.

The core challenge is to develop a system that allows an AMR to not only navigate safely but also articulate its decision-making process in real-time to align with human expectations and social norms.

2. Methodology

The authors propose a multimodal explainability module integrated into a ROS2-based navigation stack. The system is designed to perceive the environment, analyze social conflicts, and generate natural language justifications accompanied by visual heatmaps.

A. System Architecture

The module consists of four interconnected ROS2 nodes:

Camera Node: Captures real-time visual input.
BLIP Node (Bootstrapping Language-Image Pretraining): Generates image captions to provide semantic context to the visual data.
Heatmap Node: Uses Grad-CAM to generate visual saliency maps, highlighting the specific image regions influencing the robot's decision (e.g., a group of people).
LLM Node (Large Language Model): Synthesizes the image caption and heatmap summary into a coherent, natural language explanation.

B. Mathematical Formulation

The navigation task is defined as a tuple $T_{nav} = (S, G, P, E, \varepsilon)$ , where:

$S$ : State (robot pose, velocity, human positions).
$G$ : Goal configuration.
$P$ : Planned trajectory.
$E$ : Set of multimodal explanations generated at time $t$ .
$\varepsilon$ : Explainability Score ( $0 \le \varepsilon \le 1$ ), quantifying how well the robot's behavior is understood by humans.

The system enforces Social Constraints ( $\Omega_{norm}$ ) categorized into:

Human Safety: Maintaining safe distances ( $d_{human} \ge \max\{d_{social}, d_{safe}\}$ ).
Socially Acceptable Motion: Limiting abrupt stops or speed variations ( $|\dot{v}| \le \alpha_{social}$ ) unless necessary for collision avoidance.
Social Navigation: Respecting human groups and ongoing interactions ( $h(P, q_{human}) \ge 0$ ).

C. The Explainability Pipeline

When a potential social conflict is detected:

The system captures an image ( $X$ ).
Grad-CAM generates a heatmap ( $H$ ) highlighting salient regions.
BLIP generates a text caption ( $L$ ) of the scene.
A Prompt Engineering strategy guides the LLM (e.g., GPT-3.5 Turbo) to synthesize these inputs into a one-sentence explanation starting with "I see..." and ending with a rerouting phrase.
The output is displayed as a heatmap overlay and spoken aloud via a speaker.

D. Latency Optimization

The total latency ( $T_{total}$ ) is the sum of camera, BLIP, heatmap, and LLM processing times. The paper acknowledges that LLM inference (often cloud-based) introduces significant latency ( $T_{LLM}$ ), which is inversely proportional to compute power. The authors formulate an optimization problem to minimize $T_{total}$ within a maximum allowable threshold ( $T_{max}$ ) to ensure real-time relevance.

3. Key Contributions

Multimodal Explainability Framework: A novel integration of Vision-Language Models (VLMs), Grad-CAM heatmaps, and LLMs to generate real-time, context-aware explanations for AMRs.
Socially Aware Navigation: Formalization of social norms (safety, motion smoothness, group respect) as mathematical constraints within the navigation stack.
Real-Time Implementation: Deployment of the system on a Raspberry Pi 4B with a robot, demonstrating the feasibility of running complex VLM pipelines in dynamic environments.
Quantitative Validation: Introduction of a Confusion Matrix approach to measure the alignment between the robot's perceived "reasons" and human expectations, alongside a user study to measure trust.

4. Experimental Results

The system was tested on a MYAGV robot in a 14-meter delivery task across two modes: Manual and Autonomous, with and without the explainability module.

A. Navigation Performance (Table I)

Social Conflicts: The system successfully detected social conflicts (2 in manual, 3 in autonomous) only when the module was active, triggering re-planning.
Sudden Stops: The "With Explainability" (WE) condition reduced sudden stops significantly (e.g., from 21 to 18 in autonomous mode), indicating smoother, more socially compliant trajectories.
Efficiency: Total trajectory length remained nearly identical, but total time was slightly reduced with explanations, suggesting more efficient decision-making.

B. User Study (n=30)

Participants (students and faculty) rated the robot on trust, clarity, and control.

Preference Score: 76.7% of users preferred the robot with explanations.
Trust Increase: Trust levels increased by 16.7% when explanations were provided.
Understanding: User understanding of robot decisions increased by 23.3%.
Control: 70% of users felt more in control when explanations were given.

C. Technical Metrics

Accuracy: The confusion matrix analysis of the explanation module yielded an accuracy of 82.14% (TP=82, TN=79, FP=20, FN=15) in aligning with human expectations.
Latency: Average latency was ~20 seconds (ranging 5.9s to 50.6s) due to hardware limitations (Raspberry Pi) and network transmission for LLM inference. The authors note that sub-5s latency is achievable with edge computing upgrades.

5. Significance and Conclusion

This paper demonstrates that transparency is a catalyst for trust in human-robot collaboration. By moving beyond internal logic to provide multimodal, natural language explanations, the proposed system bridges the gap between robotic decision-making and human social intuition.

Impact on HRI: The study validates that robots which "speak their mind" (via text and visual heatmaps) are perceived as more predictable, trustworthy, and socially acceptable.
Future Directions: The authors highlight the need for latency optimization (e.g., edge computing, model distillation) to make explanations truly real-time. Future work will focus on refining the social constraint models and deploying the system in more complex, unstructured environments.

In summary, the research establishes a foundational framework for Explainable Social Navigation, proving that integrating VLMs and XAI into AMRs significantly enhances their acceptance and safety in human-centric spaces.