💻 computer science

Navigation beyond Wayfinding: Robots Collaborating with Visually Impaired Users for Environmental Interactions

This paper presents a collaborative human-robot guidance system that integrates a robot's precise localization with a visually impaired user's physical manipulation capabilities to facilitate complex environmental interactions, demonstrating superior safety and efficiency compared to traditional white canes and non-adaptive guiding systems.

Original authors: Shaojun Cai, Nuwan Janaka, Ashwin Ram, Janidu Shehan, Yingjia Wan, Kotaro Hara, David Hsu

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shaojun Cai, Nuwan Janaka, Ashwin Ram, Janidu Shehan, Yingjia Wan, Kotaro Hara, David Hsu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a busy, unfamiliar city with a friend who is blind. Your friend has a white cane, which is great for tapping the ground to find cracks or bumps. But what happens when you need to find a specific elevator button on a wall, pull a heavy door open, or locate a specific chair in a crowded room? The white cane can tell your friend that something is there, but it can't tell them exactly where to reach or how to interact with it.

This paper introduces a new kind of "robotic guide dog" designed to solve that exact problem. It's not just about leading someone from Point A to Point B; it's about helping them do things once they get there.

Here is the breakdown of how this system works, using some everyday analogies:

1. The Problem: The "Last 10 Feet" Struggle

Current robot guides are like GPS navigation apps. They are excellent at telling you, "Turn left at the next intersection" or "You are 50 feet from the museum." But once you arrive at the museum, the GPS doesn't help you find the specific door handle or the ticket booth button.

For a visually impaired person, the journey to the destination is easy, but the final interaction is a nightmare. They might have to wave their hands in the air, feeling around blindly for a tiny button or a door handle, which is stressful, slow, and sometimes embarrassing.

2. The Solution: A "Smart Partner" with Two Modes

The researchers built a robot (based on a four-legged dog) that acts less like a GPS and more like a thoughtful human partner. It switches between two distinct "modes" depending on what the user is doing:

Mode A: The "Tour Guide" (Lead Mode)

What it does: The robot acts like a tour guide leading a group. It uses its 360-degree cameras and laser sensors to find the destination (like an elevator or a door).
The Magic: Instead of just stopping near the object, it calculates the perfect spot to stand. It positions the user so they are facing the right side, at the right height, and within arm's reach of the target.
Analogy: Imagine a tour guide who doesn't just say, "The museum is over there." Instead, they walk you right up to the door, stop exactly where the handle is, and say, "Reach out to your left, the handle is right here."

Mode B: The "Shadow" (Adaptation Mode)

What it does: Once the user starts interacting (like pulling a door open or pressing a button), the robot switches to "Shadow Mode."
The Magic: It stops giving orders and starts moving with the user. If the user pulls a door open, the robot gently steps aside so it doesn't get in the way, but stays close enough to catch them if they stumble. It waits patiently until the user is done and ready to move again.
Analogy: Think of a dance partner. When you are leading the dance (opening the door), the partner follows your lead, moving out of your way but staying connected so you don't lose your balance.

3. How They Talk to Each Other

The system uses a mix of voice and a special leash.

The Leash: It's not just a rope; it has sensors. If the user pushes the leash forward, the robot knows they want to move. If they hold it steady, the robot knows they are working on something.
The Voice: The robot gives clear, simple instructions like, "Stand on the left," or "The button is 10 inches up." It's like a co-pilot giving you specific flight instructions rather than vague directions.

4. The Results: Faster, Safer, and Less Stressful

The researchers tested this robot against three things:

A traditional white cane.
A "dumb" robot that just stops near the target and waits (no adaptation).
A real guide dog (tested with one participant).

The Findings:

vs. White Cane: The robot was much faster at finding specific targets (like elevator buttons) and caused fewer bumps and collisions. Users felt less stressed because they didn't have to "search" blindly.
vs. The "Dumb" Robot: The smart robot was significantly better. Because it moved with the user while they opened doors, the whole process was smoother. The "dumb" robot often stopped too far away, forcing the user to stretch awkwardly or let go of the leash.
vs. The Real Guide Dog: Interestingly, the robot was actually faster at finding specific targets like elevator buttons. Real dogs are amazing at avoiding obstacles and walking in a straight line, but they can get distracted or don't always understand that "elevator" means "go to the button." The robot never gets distracted and knows exactly where the button is.

The Big Picture

This paper isn't just about building a robot; it's about changing how we think about assistive technology.

Old Way: "Here is a path. Follow it."
New Way: "Here is a path, and here is how to open the door, press the button, and sit down. I will help you do it."

By combining the robot's "eyes" (sensors) with the human's "hands" (ability to manipulate objects), this system creates a team where the robot handles the navigation and the human handles the interaction. It turns a stressful, confusing journey into a smooth, confident walk.

1. Problem Statement

Existing robotic guidance systems for Blind and Visually Impaired (BVI) individuals primarily focus on wayfinding (path following and obstacle avoidance) and assume a clear path to the destination. However, they fail to address environmental interactions—tasks requiring the user to locate, approach, and manipulate specific, often subtle, objects to enable movement (e.g., pressing elevator buttons, opening doors, pulling out chairs).

Current systems suffer from two main limitations:

Lack of Precision: They often stop near a destination but leave the user to find the specific interaction point (e.g., a small button) through trial and error, causing cognitive load and physical strain.
Rigid Coordination: They lack dynamic adaptation during the interaction phase. Once the robot stops, it often remains static, potentially blocking the user's movement or failing to adjust as the user manipulates the environment (e.g., a door swinging open).

2. Methodology

A. Design Requirements (Formative Study)

Through interviews and prototype trials with six BVI participants, the authors derived three core design requirements:

D1 (Accurate Positioning): The robot must position the user precisely to minimize search effort for small targets.
D2 (Adaptive Interaction): The robot must adjust its motion to avoid obstructing the user during manipulation (e.g., moving out of the way of a swinging door).
D3 (Effective Communication): Clear, low-burden two-way communication to coordinate actions and allow user control.

B. System Architecture

The proposed system is a dual-mode human-robot collaboration framework implemented on a Unitree Go2 quadruped robot.

Hardware:

Sensors: Ricoh Theta Z1 (360° camera) for visual perception and LiDAR (Mid360) for depth and localization.
Interface: A leash equipped with an IMU to detect user intent (pushing forward) and handle rotation; a 5-way directional button for user input.
Compute: Jetson Orin NX (planning) and Jetson Orin AGX (computer vision).

Perception Pipeline:

Object Detection: Uses NanoOWL (open-vocabulary detection) and Segment Anything 2 (SAM2) to identify "Main Objects" (e.g., door, elevator) and "Interaction Targets" (e.g., handle, button).
3D Geometry: Combines 2D masks with LiDAR data to estimate object dimensions, target height, and normal vectors.
Visual Servoing: Uses a hybrid approach (Image-Based and Position-Based) to align the robot's head precisely with the interaction target.

Dual-Mode Control Strategy:
The system switches between two modes based on the task state:

Lead Mode (Navigation & Positioning):
- Goal: Guide the user to the interaction target.
- Task-Aware Placement: Calculates a "stop goal" ( $x^*$ ) that minimizes collision risk, ensures the user can reach the target without releasing the leash, and avoids the swept area of the object's motion (e.g., door swing).
- Side Positioning: Instructs the user to stand on the specific side (left/right) of the robot to optimize reachability (e.g., standing further from the elevator for button access).
- Execution: Uses A* path planning and visual servoing to align the robot head with the target.
Adaptation Mode (Interaction):
- Trigger: Activated when the robot detects the user's hand on the interaction target.
- Control: Employs a modified Artificial Potential Field (APF) controller.
  - Attractive Force: Pulls the robot toward the user's position and handle input.
  - Repulsive Force: Pushes the robot away from obstacles (detected via LiDAR) to prevent collisions.
- Orientation: Blends the user's handle rotation with the goal direction to allow the robot to yield to the user's movement while maintaining proximity.
- Feedback: Monitors path clearance (e.g., is the door open wide enough?) and prompts the user to confirm before resuming Lead Mode.

3. Key Contributions

Novel System Scope: A robotic guide system that extends beyond simple wayfinding to support complex environmental interactions (manipulating objects).
Dual-Mode Collaboration: A novel control architecture that seamlessly switches between precise, task-aware guidance (Lead) and dynamic, non-intrusive adaptation (Adaptation), bridging the gap between robotic sensing and human manipulation.
Real-World Validation: Comprehensive evaluation in real-world scenarios (elevators, doors, chairs) demonstrating that the system outperforms both traditional aids (white canes) and non-adaptive robotic baselines.

4. Results

The study involved 9 BVI participants (8 white cane users, 1 guide dog user) performing four tasks (Elevator, Pull-Door, Push-Door, Chair) and a full route-level navigation test.

A. Comparison: Full System vs. White Cane

Efficiency: The Full System was significantly faster in the Elevator task (mean difference: 11s, $p=.005$ ) and generally faster in other tasks. The primary gain was in the "Locate" phase (finding the button/handle).
Independence: Required fewer experimenter interventions.
Smoothness: Significantly fewer collisions (hits) with the environment compared to the white cane, especially in cluttered areas.
User Feedback: Users rated the system higher for "target localization" and overall experience.

B. Comparison: Full System vs. Non-Adaptive Baseline

Efficiency: The Full System was significantly faster across all tasks, particularly in the Push-Door task (mean difference: 16.6s, $p<.001$ ).
Mechanism of Improvement: The non-adaptive system stopped at a safe distance, forcing users to overreach. The Full System's Adaptation Mode allowed it to move closer and follow the user's manipulation, reducing search time.
User Preference: Users preferred the Full System for its ability to announce the correct standing side and confirm actions before moving, reducing anxiety about balance and obstacles.

C. Route-Level Performance

Time: Full System completed the route faster (307.6s) than the white cane (399.8s), though the difference was not statistically significant ( $p=.102$ ).
Workload: The Full System resulted in a significantly lower NASA-TLX workload ( $M=26.1$ vs. $62.5$, $p=.020$ ).
Safety: Zero collisions with the Full System vs. 0.67 collisions with the white cane.

D. Comparison with Guide Dog

In a single case study, the robot outperformed the guide dog in efficiency for the elevator task (24s vs. 56s) due to the dog's lack of semantic understanding and susceptibility to distraction.
However, the guide dog was faster in straight-path navigation due to higher agility, highlighting a trade-off between semantic precision and raw speed.

5. Significance

This work represents a paradigm shift in assistive robotics for the visually impaired. By moving beyond simple path-following to task-aware environmental interaction, the system addresses a critical gap in autonomous mobility.

Human-Robot Collaboration: It demonstrates that a hybrid approach—combining the robot's precise sensing with the user's dexterity—is superior to fully autonomous manipulation (which is complex and costly) or passive guidance.
Real-World Applicability: The system proves effective in unstructured, real-world environments (crowded canteens, elevators), reducing both physical effort and cognitive load for BVI users.
Future Direction: The study validates the importance of adaptive role-switching and user-centered design, paving the way for more generalizable assistive technologies that support complex daily activities beyond simple navigation.