GuideTWSI: A Diverse Tactile Walking Surface Indicator Dataset from Synthetic and Real-World Images for Blind and Low-Vision Navigation

This paper introduces GuideTWSI, a diverse dataset combining synthetic and real-world images to address the scarcity of Tactile Walking Surface Indicator (TWSI) data, specifically bridging the gap between East Asian directional bars and North American/European truncated domes to improve navigation safety for blind and low-vision individuals.

The Gist

Imagine you are walking down a busy street with your eyes closed. You can't see the curb, the crosswalk, or the danger of a subway platform edge. But, you can feel the ground beneath your feet.

For blind and low-vision pedestrians, the ground isn't just concrete; it's a language. Special textured tiles called Tactile Walking Surface Indicators (TWSIs) act like "stop signs" and "directional arrows" made of bumps and bars.

• Directional Bars: Long, parallel ridges that say, "Keep walking this way."
• Truncated Domes: Rows of round, bumpy dots that scream, "STOP! Danger ahead! You are about to step off a curb or into traffic."

The Problem: The Robot's Blind Spot

For decades, scientists have been building robots to guide blind people. But these robots have a major problem: they are bad at reading the ground.

Most existing data sets used to train these robots are like a cookbook that only has recipes for one specific type of cake.

1. Wrong Flavor: Most data comes from Asia, where the "bumpy dots" (domes) are rare. The data is full of "stripes" (bars).
2. Wrong Angle: The photos are taken from a human's eye level (standing up). But guide robots are often on the ground (like dogs or four-legged bots), looking down at a steep angle.
3. The Result: A robot trained on this limited data might walk right past a "Stop" sign (the domes) because it only knows how to look for "Go" signs (the stripes). This is a safety disaster.

The Solution: GuideTWSI (The "Universal Translator")

The researchers in this paper built a massive new library of data called GuideTWSI. Think of it as creating a "Universal Translator" for the ground. They didn't just take photos; they built a three-part system to teach robots how to see the bumps:

1. The "Virtual Reality" Gym (Synthetic Data)

Since taking thousands of photos of every possible sidewalk in the world is impossible, the team built a video game world using Unreal Engine (the same tech used for high-end movies and games).

• The Analogy: Imagine a flight simulator for pilots. You can crash a plane 1,000 times in the simulator without anyone getting hurt.
• What they did: They created 15,000+ virtual scenes with different weather (rain, fog, bright sun), different sidewalk colors, and different camera angles. They programmed the game to automatically label every single bump. This gave the robots a "gym" to practice on before hitting the real streets.

2. The "Real World" Collection (Curated Data)

They gathered existing photos from the internet and cleaned them up, like organizing a messy attic. They fixed bad labels and made sure all the pictures spoke the same "language" so the robot could learn from them.

3. The "Field Test" (Real Robot Data)

They strapped a camera to a quadruped robot (a robot dog) and sent it out to walk around campuses and neighborhoods. They took 2,400+ photos of real bumps from the robot's low, downward-looking perspective. This is the "final exam" data.

The Magic Result

When they trained the robot's "brain" (AI model) using only the old, limited data, it was clumsy. It missed the bumps or stopped too late.

But when they added the Virtual Reality Gym data to the mix? The robot became a pro.

• Accuracy Jump: The robot's ability to spot the bumps improved by up to 29% (a huge leap in AI terms).
• Real-World Success: They put the robot on a leash (metaphorically) and sent it out to real streets. It stopped at the bumps 96% of the time, even in places it had never seen before. It stopped at the perfect distance—close enough to be safe, but far enough to let the human step forward comfortably.

Why This Matters

Think of this like teaching a dog to guide a blind person.

• Before: The dog was trained only on smooth grass. When it hit a gravel path, it got confused and kept walking into traffic.
• Now: We gave the dog a "simulation" of every type of ground imaginable (mud, gravel, ice, rain) and then tested it on real streets. Now, it knows exactly when to sit down and say, "Stop, boss, we're at the edge."

In short: This paper created the first massive, diverse "textbook" for robots to learn how to read the ground. By mixing real photos with a video game world, they taught robots to safely guide blind people across the most dangerous parts of the city.

Technical Summary

Here is a detailed technical summary of the paper "GuideTWSI: A Diverse Tactile Walking Surface Indicator Dataset from Synthetic and Real-World Images for Blind and Low-Vision Navigation."

1. Problem Statement

Tactile Walking Surface Indicators (TWSIs), specifically truncated domes (used as detectable warnings at curbs and crossings) and directional bars (used for continuous guidance), are critical safety landmarks for Blind and Low-Vision (BLV) pedestrians. While autonomous mobility assistive robots (e.g., guide dog robots) aim to replicate the stopping behaviors of trained guide dogs, they face significant challenges:

• Data Scarcity & Bias: Existing urban perception datasets are heavily biased toward East Asian directional bars and first-person viewpoints. They lack truncated domes, which are the standard warning system in North America and Europe.
• Viewpoint Mismatch: Existing datasets often feature human-eye-level perspectives, failing to capture the egocentric (top-down) or low-angle views required for ground robots (e.g., quadrupeds) to accurately detect hazards.
• Generalization Failure: Models trained on bar-centric data fail to generalize to dome-based warnings, leading to missed detections and unsafe navigation (e.g., failing to stop before a curb drop-off).

2. Methodology

The authors propose GuideTWSI, a comprehensive solution comprising a new dataset and a synthetic data generation pipeline to address the data gap.

A. The GuideTWSI Dataset

The dataset is a fusion of three distinct components totaling over 39,000 images:

1. SDome-15K (Synthetic): A photorealistic synthetic dataset generated using Unreal Engine 4 (UE4) and AirSim.
   • Generation: Uses 10 diverse environments (e.g., City Park, Downtown) with randomized lighting, weather (rain, fog), and surface materials.
   • Objects: Custom truncated dome modules modeled to ADA (Americans with Disabilities Act) standards with varied colors (yellow, red, white, gray).
   • Viewpoints: Simulates robot-mounted cameras via circular orbits and top-down sweeps to capture diverse angles.
   • Annotations: Automatically generated dense ground truth including semantic segmentation masks, instance masks, 2D bounding boxes, and depth maps.
2. RBar-22K (Curated Real-World): A compilation of existing open-source datasets (SideGuide, Tenji10K, TP) and community repositories. The authors performed rigorous cleaning, deduplication, and format unification (converting to RLE and polygon formats) to ensure consistency.
3. RDome-2K (Real Robot Data): A new dataset collected by the authors using a Unitree Go2 quadruped robot equipped with a downward-facing Intel RealSense D435 camera (70° tilt). It captures 2,466 RGB frames of truncated domes in real-world US environments (campus, suburban, rural) with human-verified annotations.

B. Synthetic Data Pipeline

The pipeline allows for scalable data creation without expensive human collection. It randomizes:

• Environmental factors: Sun position, brightness, fog, rain.
• Object properties: Texture and color of tactile blocks.
• Camera trajectories: Low-angle, top-down, and orbital paths to mimic robot navigation.

C. Experimental Setup

• Models: The authors benchmarked state-of-the-art segmentation models, including YOLOv11-seg (N and X variants), Mask2Former, SAM2.1+UNet, and DINOv3-based models (RegCls and EoMT).
• Training Strategy: Models were trained on combinations of real data only vs. real data augmented with synthetic data (RBar-22K + SDome-15K).
• Evaluation: Tested on the held-out RDome-2K real-world robot dataset.
• Robot Deployment: The best-performing model (YOLOv11-seg-N) was deployed on a Unitree Go2 robot with an NVIDIA Jetson AGX Orin, running at 43 FPS using TensorRT. The system uses a segmentation-based closest-point detection strategy to trigger stops.

3. Key Contributions

1. GuideTWSI Dataset: The largest and most diverse TWSI dataset to date, specifically addressing the lack of truncated dome data and robot-relevant viewpoints.
2. Synthetic Data Pipeline: A novel UE4/AirSim pipeline that generates high-fidelity, annotated synthetic data for accessibility perception, bridging the sim-to-real gap.
3. Real-World Validation: The first demonstration of a guide robot reliably stopping at truncated domes in unseen real-world environments, achieving a 96.15% success rate.

4. Results

• Segmentation Performance:
  • Training on real data alone yielded moderate results (e.g., Mask2Former mIoU of 0.5777), often missing dome regions (low recall).
  • Synthetic Augmentation: Adding SDome-15K significantly improved performance across all models.
    • DINOv3+EoMT achieved the largest gain: mIoU increased from 0.5804 to 0.8756 (+29.52 points).
    • Mask2Former mIoU rose from 0.5777 to 0.8375.
    • Recall improved drastically (e.g., Mask2Former recall jumped from 0.5975 to 0.8669), indicating the model learned to detect more complete dome structures.
• Robot Deployment:
  • In field trials across 5 different environments (urban, suburban, residential), the robot achieved a 96.15% stop success rate (100/104 trials).
  • Precision: No false positives were observed; the robot never stopped erroneously when no domes were present.
  • Safety: Successful stops occurred at a safe average distance of 39 cm from the dome start, providing sufficient space for a human user to react.
  • Failures: The 4 failures were due to extreme lighting conditions (lens flare) causing late detection, not total misses.

5. Significance

• Safety-Critical Navigation: This work directly addresses a life-safety gap in robotic assistance for the BLV community. Reliable detection of truncated domes is essential for preventing falls at curbs and platform edges.
• Overcoming Data Bias: By shifting focus from East Asian directional bars to North American/European truncated domes and incorporating robot-centric viewpoints, the paper corrects a major geographic and perspective bias in existing literature.
• Synthetic Data Efficacy: The study provides strong empirical evidence that high-quality synthetic data is not just a supplement but a necessary component for training robust perception models in safety-critical domains where real-world data is scarce or difficult to collect.
• Open Source: The release of the dataset, model weights, and code fosters further research in accessible robotics and urban perception.

In conclusion, GuideTWSI establishes a new benchmark for tactile indicator recognition, demonstrating that combining curated real-world data with scalable synthetic generation enables autonomous robots to perform safety-critical stopping behaviors with high reliability.