Technical Summary: VIABench

Problem Statement

Visually impaired individuals (VIIs) face significant daily challenges due to limited access to visual information. While Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on general vision-language tasks, their practical utility in real-world blind assistance remains largely underexplored. Existing benchmarks and datasets for assistive vision often suffer from critical limitations:

• Domain Gap: Datasets like WAD rely on curated travel vlogs intended for sighted audiences, failing to capture the raw, unedited nature of real-world VIIs' experiences.
• Task Limitations: Datasets like EgoBlind focus primarily on passive, user-initiated Visual Question Answering (VQA), neglecting the need for proactive, autonomous assistance.
• Temporal Constraints: Many existing video benchmarks utilize short clips (e.g., 3 seconds), which lack the temporal richness required for long-horizon reasoning and navigation.
• Evaluation Mismatch: Current online evaluation pipelines are often computationally heavy and rely on sparse annotations, making them ill-suited for assessing "reminder-style" tasks that require continuous monitoring and timely alerting.

Consequently, there is a lack of a comprehensive, versatile video benchmark collected by blind individuals that evaluates MLLMs across the full spectrum of assistive needs, including proactive anticipation and interactive guidance.

Methodology

1. VIABench Dataset Construction

The authors constructed VIABench, a large-scale, time-aligned, egocentric video benchmark specifically tailored for visual impairment assistance. The data collection followed a five-stage pipeline:

• Sourcing: Videos were gathered from two streams: (1) Real-world videos from VIIs on platforms like YouTube, Bilibili, and DouYin, and (2) Informed simulated videos recorded by trained annotators to cover specific "Vision-Guided Interaction" scenarios scarce in public data.
• Annotation: The dataset includes 761 videos totaling 46.9 hours, with 14,526 manually curated annotations. The average video duration is 222 seconds (max 1959s), significantly longer than prior datasets.
• Quality Control: A rigorous process involving pilot annotation, professional vendor collaboration, and multi-stage QA ensured high-quality, fine-grained labeling.

2. Task Definitions

VIABench defines three core tasks reflecting increasing levels of assistive complexity:

• Proactive Reminder: The model must autonomously monitor the video stream and proactively describe upcoming navigation-critical events (e.g., obstacles, stairs) without explicit user prompts. This requires accurate anticipation and real-time responsiveness.
• Visual Question Answering (VQA): The model answers user-posed questions about the environment based on current and past visual context, simulating spontaneous queries during navigation.
• Vision-Guided Interaction: A multi-turn, closed-loop task where the model provides iterative, context-aware guidance to help the user adjust their viewpoint or physical movements to achieve a specific goal.

3. Evaluation Framework: Token-Level Prompt Activation Decoding (TPAD)

To address the challenge of evaluating proactive alerting in offline models (which typically require explicit prompts), the authors propose TPAD, a novel two-stage evaluation framework:

• Mechanism: TPAD transforms a pretrained MLLM into a frame-wise proactive detector without fine-tuning. It concatenates a fixed forced-choice prompt (e.g., "Should the user be warned? (A) Yes; (B) No") with the entire video sequence.
• Scoring: For each frame, the hidden state of its final token is projected through the language modeling head to compute the probability of issuing an alert.
• Advantage: This allows for a single forward pass to generate frame-wise alert probabilities across the entire video, enabling efficient and context-aware evaluation of offline models under streaming conditions.

Key Contributions

1. VIABench Dataset: The introduction of a large-scale, real-world video benchmark for blind assistance, featuring long-form egocentric videos and diverse content. It is the first to unify Proactive Reminder, VQA, and Vision-Guided Interaction tasks within a single benchmark.
2. TPAD Framework: The proposal of a lightweight, two-stage evaluation mechanism that enables efficient assessment of proactive alert generation in long, continuous videos, bridging the gap between offline MLLMs and real-time assistive requirements.
3. Comprehensive Evaluation: A systematic evaluation of leading open-source, proprietary, and online streaming MLLMs on VIABench, providing insights into their current limitations in real-world assistive applications.

Experimental Results

The authors evaluated a wide range of models, including open-source foundation models (e.g., InternVL, Qwen, LLaVA), proprietary models (GPT-5, GPT-4o, Gemini), and online streaming models.

• Overall Performance: Current MLLMs exhibit only nascent capabilities in real-world visual assistance. Even the strongest model, GPT-5, achieved an average score of only 28.8 on the benchmark.
• Task Disparities: Performance on Proactive Reminder and Vision-Guided Interaction tasks is notably lower than on standard VQA tasks.
  • Proactive Reminder: Models frequently fail to anticipate navigation-critical events or generate guidance aligned with ground-truth intervals. They struggle with the dynamic requirement of issuing timely alerts while remaining silent when no relevant information is present.
  • Vision-Guided Interaction: Models often provide generic instructions (e.g., "pick it up") rather than spatially specific guidance, failing to account for the unique perspective and limitations of VIIs.
• Streaming Models: Existing online streaming models perform extremely poorly on the Proactive Reminder task. The authors attribute this to limited exposure to complex, real-world guidance instructions in their training data (often limited to dense captions or narrations) and a lack of robustness to low-quality, uncontrolled visual inputs.

Significance and Future Directions

The paper claims that VIABench serves as a challenging and realistic testbed to drive future research toward developing customized MLLMs for real-world assistance. The authors emphasize that current models are not yet ready for deployment in safety-critical blind assistance scenarios.

The paper outlines three specific avenues for future work:

1. Architecture Improvement: Developing model architectures capable of reliable, context-aware, real-time proactive responses.
2. Data Enrichment: Incorporating richer video-text datasets that encompass diverse instruction types, complex interactions, and real-world scenarios to improve proactive reasoning.
3. Blind-Centric Optimization: Fine-tuning models specifically for the unique perspective and intentions of blind users, addressing issues like low-resolution inputs, camera inversion, and the need for spatially specific guidance.

Ultimately, the work aims to improve the autonomy and safety of visually impaired individuals by fostering the development of more capable and dependable assistive technologies.