A Dataset for Crucial Object Recognition in Blind and Low-Vision Individuals' Navigation

This paper introduces a publicly available dataset comprising 21 videos of blind and low-vision individuals navigating outdoor spaces, along with a refined taxonomy of 90 crucial objects and corresponding annotations, to address the limitations of existing computer vision models in supporting safe navigation for this demographic.

The Gist

The Big Picture: A "Safety Net" for the Eyes

Imagine you are walking down a busy street. You can see a puddle, a low-hanging tree branch, a parked car, and a "Stop" sign. Your brain instantly processes all this to keep you safe.

Now, imagine you are blind or have low vision. You rely on a white cane to tap the ground. The cane is great at finding things on the ground (like a curb or a trash can), but it's like a flashlight that only shines on your feet. It can't tell you about a low-hanging branch that might hit you in the face, or a slippery patch of ice, or a maintenance truck blocking the sidewalk.

This paper is about building a new "digital safety net" to help fill in those gaps.

The researchers realized that the AI apps blind people use today (like "Seeing AI") are like students who only studied for a very specific, easy test. They know what a "dog" or a "car" looks like, but they don't know about the specific, tricky things that actually cause accidents for people navigating the world without sight.

The Problem: The "Generic" Dictionary

Think of the current AI models as having a dictionary. But it's a dictionary written by people who can see, for people who can see.

• The Old Dictionary: Contains words like "Dog," "Car," "Tree," and "Person."
• The Missing Words: It's missing words like "Low-hanging branch," "Wet pavement," "Barrier post," "Ice on the sidewalk," or "A hose lying on the ground."

The researchers found that if you ask a standard AI, "Is there a low-hanging branch here?" it often says, "I don't know what that is," or it misses it entirely. This is dangerous because those are the exact things that can trip someone up or cause a collision.

The Solution: A New "Survival Guide"

To fix this, the team created a brand-new dataset (a massive collection of training data) specifically for blind navigation. Here is how they built it:

1. The Field Trip (Data Collection)
Instead of just taking photos of perfect, sunny streets, they went to YouTube and Vimeo. They watched 21 real-life videos of blind people navigating the world. They looked for the "scary moments"—the times the person had to stop, dodge, or get confused.

2. The Focus Group (The Expert Panel)
They didn't just guess what was important. They held a meeting (a focus group) with 6 experts:

• People who are blind or have low vision.
• Orientation and Mobility (O&M) trainers (teachers who teach people how to navigate).
• Sighted people who work closely with the blind community.

They asked them: "If you had a magic AI assistant, what would you want it to yell out to warn you?"

• The Answer: They didn't just want to know about "cars." They wanted to know about "overhanging branches," "closed sidewalks," "wet surfaces," and "moving walks" (like those in airports).

3. The New Taxonomy (The 90-Item Checklist)
From these discussions, they created a list of 90 specific objects that matter for safety. They grouped them into fun categories, like:

• "The Sneaky Ones": Things that are hard to see or feel (like ice or a hose).
• "The Head-Bangers": Things that are too high for a cane to touch (like tree branches).
• "The Road Blockers": Things that shouldn't be on the sidewalk (like a parked maintenance truck).

The Test: Putting the AI to the Test

The researchers took the best, most famous AI models in the world (the "champions" of computer vision) and asked them to look at their new videos.

The Result? The AI models failed miserably.

• They could spot a "Person" or a "Bus" easily.
• But when it came to the 90 crucial safety items, they were often clueless. They missed the low-hanging branches, the barrier posts, and the wet pavement.

The Analogy: It's like giving a Formula 1 race car driver a map of a dirt track they've never seen, but only teaching them how to drive on a highway. They are fast and smart, but they don't know the rules of this specific, dangerous terrain.

Why This Matters

This paper isn't just about making a list; it's about saving lives and increasing independence.

• Current Tech: "I see a person." (Good, but not enough).
• Future Tech (with this dataset): "Warning: There is a low-hanging branch 3 feet ahead, and the sidewalk is icy on the left. Please step right."

By making this dataset public, the researchers are handing the keys to the AI developers. They are saying, "Here is the textbook you need to study. If you train your AI on this, you can build a true navigation assistant that keeps blind people safe."

The Bottom Line

We have amazing technology, but it's currently "blind" to the things that matter most to blind people. This paper provides the missing vocabulary and the training manual needed to teach AI how to see the world through the eyes of safety, not just sight. It's a step toward a future where technology doesn't just describe the world, but actively protects us while we move through it.

Technical Summary

1. Problem Statement

Current computer vision (CV) models and large-scale datasets (e.g., ImageNet, MS COCO, Mapillary Vistas) are insufficient for aiding Blind and Low-Vision (BLV) individuals in real-time navigation.

• Accessibility Gap: Mainstream datasets lack "accessibility-aware" annotations. They often miss fine-grained, action-oriented objects crucial for BLV safety, such as specific sidewalk obstructions, overhanging branches, or specific paving types.
• Reliability Issues: Existing models are trained on generic data and fail to detect objects that are critical for BLV users but irrelevant to sighted users (e.g., a fire hydrant vs. a generic "pole").
• Contextual Limitations: Current AI tools often provide reactive information rather than proactive guidance, and they lack the ability to distinguish between objects that are merely present versus those that pose an immediate physical threat (e.g., objects that "pick you before you pick them").

2. Methodology

The authors developed a specialized dataset and taxonomy through a multi-stage process involving data collection, community-driven refinement, and ground-truth labeling.

A. Data Collection

• Source: 21 videos were extracted from public platforms (YouTube and Vimeo) featuring BLV individuals navigating outdoor spaces.
• Selection Criteria: Videos were selected based on keywords like "blind," "white cane," and "orientation and mobility." Duration ranged from 2 to 20 minutes.
• Segmentation: The 21 videos were split into 31 video segments (clips of 5–95 seconds) focusing on specific navigational events.
• Keyframe Extraction: Using the tool Katna, keyframes were extracted from these segments (3 to 93 keyframes per segment) to represent scene transitions and lighting changes.

B. Taxonomy Development (Focus Group Study)

To ensure the object list was relevant to the BLV community, the authors conducted a focus group study:

• Participants: 6 participants (2 blind, 2 low-vision, 2 sighted experts in Orientation & Mobility).
• Process: Researchers presented an initial list of ~80 objects derived from video analysis. Participants validated, prioritized, and added new objects based on real-world navigational needs.
• Refinement: The list was refined to 90 distinct object categories, organized into 15 groups. Key categories include:
  • Sidewalk attributes (e.g., tactile paving, sloped driveways).
  • Obstructions (e.g., barrier posts, overhanging branches, closed sidewalks).
  • Moving objects (e.g., guide dogs, bicycles, wheelchairs).
  • Indoor objects (e.g., elevators, uneven stairs).
  • Traffic elements (e.g., traffic signals, crosswalks).

C. Ground Truth Labeling

• Annotation Protocol: All authors manually annotated the 31 video segments.
• Method: For each keyframe, annotators determined the presence (1) or absence (0) of all 90 object categories.
• Quality Control: To mitigate "change blindness," annotators viewed consecutive keyframes side-by-side. Each segment was annotated by at least two researchers independently, with conflicts resolved through collaboration.

3. Key Contributions

1. BLV Navigation Dataset: A publicly available dataset containing 21 videos and 31 segmented clips featuring BLV navigation in outdoor environments.
2. 90-Object Accessibility Taxonomy: A fine-grained, action-oriented taxonomy of 90 objects specifically curated for BLV navigation, refined through direct community feedback. This taxonomy includes categories largely absent from mainstream datasets (e.g., "overhanging tree branches," "barrier stumps," "wet surfaces").
3. Comprehensive Annotations: Ground-truth binary labels for all 90 objects across the extracted keyframes.
4. Benchmarking: A preliminary evaluation of seven state-of-the-art (SOTA) CV models on this specific dataset to highlight current limitations.

4. Results and Evaluation

The authors evaluated seven SOTA models across four CV genres: Object Recognition, Detection, Semantic Segmentation, and Visual Question Answering (VQA).

• Models Tested: RAM, Faster R-CNN, YOLOv7, HRNet V2, Mask R-CNN, GPV-1, and BLIP.
• Performance Metrics: Precision, Recall, and F1-score (Micro Average).
• Key Findings:
  • Low Coverage: Traditional detection and segmentation models (e.g., YOLOv7, Faster R-CNN) could only predict 12–15 of the 90 objects because these objects were missing from their training vocabularies.
  • VQA/Open-Vocabulary Advantage: Models with language encoders (BLIP, GPV-1, RAM) performed better, with BLIP achieving the highest F1 score (0.4783) across 90 categories. However, even these models struggled significantly.
  • Critical Failure: None of the models performed well on the most critical groups (Groups 3, 5, and 7: Obstructions less likely to be detected by a cane, Objects that pick you before you pick them, and Objects not supposed to be on the sidewalk).
  • Heatmap Analysis: Visualizations showed that established datasets (Mapillary Vistas, COCO, etc.) lack annotations for the most safety-critical objects identified in this study.

5. Significance and Implications

• Bridging the Gap: This work highlights that "general-purpose" AI is not yet "inclusive AI." Current models fail to detect the specific nuances of the BLV environment.
• Resource for Retraining: The dataset provides the necessary ground truth to retrain or fine-tune models using few-shot learning, enabling them to recognize accessibility-critical objects.
• Design Guidelines: The focus group study provided design implications for future assistive apps, such as:
  • Prioritization: AI should prioritize high-risk objects (e.g., overhanging branches) over low-risk ones.
  • Proactive vs. Reactive: Systems should warn users of hazards before they are encountered, rather than just describing the scene.
  • Configurability: Users need control over which objects trigger alerts to reduce cognitive load.
• Ethical Considerations: The authors emphasize that AI should augment, not replace, physical tools like white canes, and that reliability and legal disclaimers are crucial for user trust.

In conclusion, the paper argues that for AI to be truly effective for BLV navigation, it must move beyond generic object detection to specialized, context-aware recognition of accessibility-critical hazards, supported by datasets like the one proposed here.