💻 computer science

Do VLMs See What Sensors Feel? A Scalable Expert-Guided Design for Wheelchair Accessibility Assessment from Street View

This paper proposes an expert-guided retrieval-augmented framework using vision-language models and Google Street View imagery to assess wheelchair accessibility at scale, demonstrating that while model ratings show partial alignment with real-world mobility friction derived from GPS dwell times, they effectively capture structural features like curb ramps but struggle with subtle or transient barriers.

Original authors: Dongdong Wang, Alina Hagen, Isabelle Gatmaitan, Hao Zhou, Yiwen Dong, Shabboo Valipoor, Vivian W. H. Wong, Lingyao Li

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongdong Wang, Alina Hagen, Isabelle Gatmaitan, Hao Zhou, Yiwen Dong, Shabboo Valipoor, Vivian W. H. Wong, Lingyao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a city is easy for a person in a wheelchair to navigate. Traditionally, you'd have to send a team of people to walk (or roll) every single street, checking for broken sidewalks, steep curbs, or blocked paths. It's slow, expensive, and hard to do everywhere.

This paper asks a simpler question: Can a super-smart computer program, looking at photos of the street, figure out the same things a person in a wheelchair would feel?

Here is the story of how they tried to answer that, explained simply:

The Problem: The "Invisible" Bumps

For a wheelchair user, a city isn't just a picture; it's a physical experience. A tiny crack in the pavement, a curb that's too high, or a parked car blocking the path can stop them dead in their tracks. These "friction points" are hard to map because they are everywhere, they change, and they are often too small for a standard map to notice.

The Experiment: The Computer vs. The Wheelchair

The researchers set up a test at the University of Florida. They did two things at the same time:

The "Feel" Test: They attached a GPS tracker to a manual wheelchair and had it drive around campus. They looked for places where the wheelchair stopped or slowed down for a long time (called "dwell time"). They figured that if the wheelchair stopped, it was probably because the path was hard to navigate.
The "See" Test: They took Google Street View photos of those exact same spots and fed them into a Vision-Language Model (VLM). Think of a VLM as a robot that has read every book about city planning and can "see" images.

The Secret Sauce: The Expert Guide

The researchers realized that just asking the robot, "Is this accessible?" wasn't enough. The robot might guess wrong or miss details. So, they gave the robot a cheat sheet.

They built a system where the robot could look up rules (like the Americans with Disabilities Act) and advice from real accessibility experts while it was looking at the photo. It's like giving a student a textbook and a study guide right before a test, rather than just letting them guess.

What They Found

The results were a mix of "Great job!" and "Not quite yet."

The Good News: The robot got better as it got smarter. The biggest, most powerful robot models (the "78B" and "32B" versions) started to agree with the wheelchair's GPS data. When the wheelchair struggled (stopped for a long time), the robot gave the spot a low score. When the path was smooth, the robot gave it a high score.
- Analogy: It's like a weather app that isn't perfect, but if you ask the smartest version, it can usually tell you when it's going to rain just by looking at the clouds.
The Bad News: The robot still missed some things. It was good at spotting big, obvious problems like missing ramps or blocked paths. But it struggled with subtle issues, like a slightly bumpy surface or a temporary obstacle that wasn't clearly visible in the photo.
- Analogy: The robot is like a person looking at a map; they can see a bridge is out, but they can't feel the pothole in the road unless it's huge.

The "Aha!" Moments

The researchers discovered a few tricks to make the robot work better:

Don't use the 360-degree view: Surprisingly, the robot worked better when shown two specific, straight-ahead photos (like looking forward and to the side) rather than a full 360-degree panorama. The 360 view was too confusing for the robot to focus on the details.
Ask more questions: Instead of asking one big question ("Is this accessible?"), they asked the robot ten specific questions (e.g., "Is the curb ramp there?", "Is the path wide enough?", "Are there cracks?"). This "checklist" approach made the robot much more accurate.

The Bottom Line

The paper concludes that these smart computer programs are not ready to replace human inspectors. You can't just let a robot decide if a building is legal or safe.

However, they are excellent scouts. They can scan thousands of streets quickly and say, "Hey, these 50 spots look really bad. Send a human expert to check those first." It's a way to find the trouble spots in a city without having to send a human to check every single block.

In short: The robot can see what the sensor feels, but only if you give it a good guidebook and ask it the right questions. It's a powerful tool for finding problems, but it still needs a human to confirm the details.

Technical Summary: Do VLMs See What Sensors Feel? A Scalable Expert-Guided Design for Wheelchair Accessibility Assessment from Street View

Problem Statement

Assessing wheelchair accessibility in the built environment is inherently difficult because real-world mobility is constrained by distributed, context-dependent, and often temporary barriers (e.g., uneven pavement, obstructed pathways, transient construction) that are hard to capture at scale. Current assessment methods rely heavily on labor-intensive manual audits or static GIS inventories, which fail to capture the dynamic, embodied experience of wheelchair users. While Vision-Language Models (VLMs) have shown promise in interpreting Street View imagery for urban perception, it remains unclear whether these models can identify accessibility barriers in a way that aligns with the physical reality of wheelchair navigation, specifically regarding behavioral signals like prolonged dwelling time.

Methodology

The study proposes an expert-guided retrieval-augmented generation (RAG) framework to evaluate wheelchair accessibility using Google Street View (GSV) imagery and compare it against real-world sensor data.

Data Collection & Ground Truth:
- Sensor Data: Researchers collected GPS traces from manual wheelchairs traversing high-traffic routes on the University of Florida (UF) campus. "Dwell time" (locations where the wheelchair paused for an extended period) was used as a behavioral proxy for mobility friction. The dataset includes 407 unique GSV locations linked to 1,253 dwell points and 720 non-dwell points.
- Visual Data: For each location, GSV imagery was retrieved from eight panoramic angles.
VLM Pipeline Design:
- Retrieval-Augmented Reasoning: The system retrieves relevant accessibility guidance from a knowledge base distilled from ADA standards, UF campus design guidelines, PROWAG, and expert-derived heuristics.
- Prompting Strategy: The pipeline employs a structured, multi-question prompting approach (10 sub-questions covering surface conditions, path width, obstacles, curb ramps, etc.) rather than single-question prompts.
- Model Variants: The study evaluated multiple VLMs (InternVL3 and Qwen3 families) across different sizes (2B to 78B parameters) and configurations (Base vs. RAG-augmented; Perspective Projections vs. Full Panoramas).
- Output: Models generate structured accessibility scores (10–50) with justifications based on observable visual evidence.
Evaluation Metrics:
- Correlation: Pearson correlation coefficient ( $r$ ) between dwell time (friction) and VLM accessibility scores. A strong negative correlation indicates alignment (high friction = low score).
- Distributional Agreement: Earth Mover's Distance (EMD) to measure the similarity between the distributions of dwell times and VLM scores.
- Visual Cue Analysis: Rank-difference analyses (AMI and AII) to identify which environmental objects correlate with model agreement or specific accessibility ratings.

Key Contributions

Structured VLM Pipeline: A scalable pipeline for assessing accessibility barriers from Street View imagery that integrates visual cues with domain-specific criteria.
Expert-Guided Augmentation: A strategy that grounds VLM reasoning in ADA handbooks and expert rubrics, demonstrating that retrieval of regulatory knowledge improves alignment with physical mobility patterns.
Campus-Scale Benchmark: A novel dataset linking GSV imagery with wheelchair sensor traces, enabling the evaluation of where VLM-based assessments align with or diverge from sensor-derived mobility patterns.

Results

Correlation with Mobility Friction: VLM ratings show a consistent negative correlation with wheelchair dwell time, indicating partial but meaningful alignment with behavioral proxies for mobility friction.
- Model Scale: Larger models demonstrated stronger agreement. The InternVL3-78B model achieved the highest correlation ( $r = -0.54$ ), followed by Qwen3-32B ( $r = -0.47$ ). Smaller models (e.g., 2B variants) showed weak correlations ( $r \approx -0.10$ to $-0.13$).
Impact of RAG: Retrieval-Augmented Generation significantly improved performance. For InternVL3-78B, RAG improved the correlation from $-0.40$ (non-RAG) to $-0.54$ and reduced the Earth Mover's Distance (EMD), indicating better distributional alignment.
Input Format: Perspective projections (specifically $90^\circ$ and $270^\circ$ views) yielded stronger alignment with sensor data than full panoramic inputs, likely due to reduced geometric distortion and better alignment with a forward-facing user perspective.
Prompting Strategy: Multi-question prompting substantially outperformed single-question prompting, improving correlation from $-0.21$ to $-0.54$ for InternVL3-78B.
Visual Cues: Models successfully associated infrastructural supports (crosswalks, curb ramps, tactile paving) with higher accessibility scores and alignment. Conversely, subtle surface conditions (cracks, loose surfaces) and transient obstructions (parked vehicles) were associated with lower scores and greater prediction divergence.

Significance and Claims

The paper claims that expert-guided VLMs offer a scalable screening layer for accessibility assessment. The findings suggest that while VLMs cannot replace on-the-ground professional audits or lived-experience data, they can effectively flag locations where human review is most needed.

The study positions this pipeline as a tool to support urban planners and accessibility offices by:

Providing a cost-effective method to identify potential barriers across large geographic areas.
Aligning visual inference with sensor-derived indicators of real-world navigation difficulties.
Highlighting that grounding models in regulatory standards (ADA) and expert heuristics is critical for bridging the gap between visual appearance and embodied accessibility.

The authors explicitly note limitations, including the pilot nature of the study (single campus), the temporal staleness of Street View imagery regarding transient barriers, and the fact that dwell time is a proxy rather than a direct accessibility label. Consequently, the framework is presented as a prioritization tool for human review rather than a definitive compliance audit.