GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance

1. Problem Statement

Safe and independent navigation is a critical challenge for the estimated 2.2 billion people worldwide affected by blindness and low vision (BLV). While Multimodal Large Language Models (MLLMs) offer potential for assistive navigation, their development is hindered by a severe scarcity of accessibility-aware datasets.

• Limitations of Existing Data: Current datasets (e.g., VizWiz, VIALM) are often small-scale, lack geographic diversity, or rely on sighted annotators who struggle to anticipate specific BLV needs (e.g., spatial orientation, obstacle proximity).
• Annotation Bottleneck: Creating high-quality guidance data traditionally requires labor-intensive expert annotation, limiting scalability.
• Technical Gap: Existing models often fail to adhere to established BLV guidance standards, particularly in providing accurate depth perception and structured spatial descriptions (e.g., clock-face directions, step counts).

2. Methodology

The authors introduce GUIDEDOG, a large-scale dataset and a novel construction pipeline designed to overcome annotation bottlenecks while maintaining high fidelity to BLV needs.

A. The GUIDEDOG Standards

To ensure the data is actionable for BLV users, the authors distilled guidelines from expert organizations (e.g., Vision Australia, Be My Eyes) into three structured standards:

1. S1 (Surroundings): Describe the current location and key environmental elements to establish spatial awareness.
2. S2 (Obstacles): Identify hazards with specific location (clock-face: 10–2 o'clock), type, and proximity (in steps).
3. S3 (Summary & Direction): Provide a concise, intuitive navigation instruction (e.g., "proceed forward slightly left") rather than precise metric measurements.

B. Human-AI Pipeline (Verification-Centric)

Instead of generating descriptions from scratch (which is error-prone), the authors employ a generation-to-verification pipeline:

1. Data Collection: Sourced 269 "walking videos" from YouTube (Creative Commons) across 46 countries and 183 cities. Filtered for pedestrian perspectives and removed non-street content.
2. Scene Information Extraction:
   • Global: Used GPT-4o to extract scene descriptions and validate street suitability.
   • Local: Used open-world object detectors and depth estimation models to identify objects and calculate relative distances (converted to "steps").
3. Silver Label Generation: An MLLM combined the extracted data with the GUIDEDOG standards to generate initial "silver labels."
4. Gold Label Verification: Human annotators (sighted experts) verified and refined the silver labels, filtering out unsuitable images and correcting inaccuracies to produce high-fidelity "gold labels."
5. Privacy: Applied EgoBlur to automatically mask faces and license plates.

C. Benchmark: GUIDEDOGQA

To evaluate fine-grained visual perception, the authors constructed GUIDEDOGQA, an 818-sample benchmark featuring:

• Object Recognition: Multiple-choice questions to verify if specific objects are present.
• Relative Depth Comparison: Questions asking which of two objects is closer/farther, testing spatial reasoning essential for navigation.

3. Key Contributions

1. GUIDEDOG Dataset: A scalable dataset of 22,084 image-description pairs (including 2,106 human-verified gold labels) covering 46 countries. It is the first dataset to combine geographic diversity with strict adherence to BLV guidance standards.
2. Scalable Pipeline: A novel human-AI workflow that shifts the annotation burden from generation to verification, enabling large-scale data collection without sacrificing quality.
3. GUIDEDOGQA Benchmark: A specialized evaluation suite for testing object recognition and depth perception in real-world egocentric scenes.
4. Comprehensive Evaluation: Extensive experiments demonstrating that current MLLMs struggle with spatial understanding and standard adherence, highlighting the need for specialized training.

4. Experimental Results

The authors evaluated various open-source (LLaVA, Qwen, Molmo), proprietary (GPT-4o, Gemini), and Socratic models.

• Guidance Generation:

  • Proprietary Models: GPT-4o demonstrated the strongest zero-shot capabilities, outperforming open-source models.
  • Fine-Tuning: Fine-tuning open-source models (Qwen-2.5-VL) on the silver labels resulted in dramatic performance improvements, surpassing many zero-shot proprietary models.
  • In-Context Learning: Few-shot examples significantly boosted performance for most models, helping them adapt to the specific GUIDEDOG standards.
  • Socratic Models: Models relying on text-only reasoning (after converting images to text) performed worse than direct MLLMs, confirming that direct visual perception is crucial for effective guidance.

• Visual Perception (GUIDEDOGQA):

  • Object Recognition: Open-source models performed surprisingly well, often outperforming proprietary models.
  • Depth Perception: This remained a significant challenge. Most open-source models performed near random chance. While fine-tuning improved depth perception, it remains a key bottleneck.
  • User Study: Human evaluators rated GPT-4o highest for adherence to standards. The low scores for "Obstacle Information" (S2) across models correlated with the poor depth perception results.

5. Significance and Impact

• Bridging the Gap: GUIDEDOG addresses the critical lack of large-scale, standardized data for BLV assistive technologies, moving beyond small, niche datasets.
• Standardization: By codifying BLV guidance into S1-S3 standards, the paper provides a clear framework for evaluating and training future assistive AI.
• Real-World Applicability: The dataset's geographic diversity ensures that models trained on it are more likely to generalize to real-world scenarios across different cultures and infrastructures.
• Future Directions: The results indicate that while MLLMs are improving, spatial understanding and depth perception are the primary hurdles. The paper calls for research focused on these specific capabilities to create truly safe and independent navigation aids for the visually impaired.

Limitations Noted: The dataset uses static images (isolating guidance quality from temporal dynamics) and relies on sighted walking-tour footage, which may differ slightly from the head-mounted camera perspective of actual BLV users (though motion analysis suggests the gap is manageable). Additionally, cross-cultural variations in spatial language are not fully modeled.