GeoVisA11y: An AI-based Geovisualization Question-Answering System for Screen-Reader Users

The paper introduces GeoVisA11y, an open-source, LLM-based system that enables screen-reader users to interact with geovisualizations through natural language queries, validated by user studies demonstrating its effectiveness in bridging accessibility gaps and revealing distinct interaction patterns.

The Gist

Imagine trying to read a map, but instead of seeing lines, colors, and shapes, you only hear a robotic voice reading you a spreadsheet. That is the current reality for millions of blind or low-vision people trying to understand geovisualizations (maps that show data, like population density or election results). They can get the raw numbers, but they miss the "big picture"—the patterns, the trends, and the stories the map is trying to tell.

Enter GeoVisA11y (pronounced "Geo-Vis-Accessibility"). Think of it as a super-smart, conversational tour guide that lives inside a map.

The Core Idea: A Map That Talks Back

Most maps are like a painting in a museum: you look at it, and that's it. If you can't see, you can't enjoy the art. GeoVisA11y turns the map into a conversation.

Instead of staring at a screen, a user can ask, "Show me the states with the highest population density," or "What does the map look like around here?" The system doesn't just read numbers; it analyzes the map, finds patterns, and explains them in plain English. It's like having a friend who can see the map, point out the interesting parts, and explain why they matter, all while you navigate using just your keyboard or voice.

How It Works: The "Brain" Behind the Map

The system uses a team of AI specialists (Large Language Models) working together in a pipeline:

1. The Translator: When you ask a question, the system first figures out if you want to move the map (e.g., "Go to Texas") or ask something (e.g., "Is Texas crowded?").
2. The Clarifier: If you say, "How is it here?", the system looks at where your cursor is focused to know that "it" means "population density" and "here" means "Texas." It fixes the confusion before answering.
3. The Detective: The system runs mathematical detective work (using real statistics) to find "hot spots" (areas with high values) and "cold spots" (areas with low values). It doesn't just guess; it calculates.
4. The Storyteller: Finally, it turns those cold numbers into a warm, natural language story. Instead of saying "Iowa has a value of 45," it might say, "Iowa has a moderate population density, similar to its neighbors, but much lower than the coastal states."

The User Experience: A Two-Way Street

The system is designed to work for two types of people:

• For Screen-Reader Users (Blind/Low-Vision): They use keyboard arrows to "jump" from state to state (like hopping from island to island) and ask questions. The system highlights the area they are talking about and describes the shape of the state or its neighbors. It gives them the freedom to explore the map on their own terms, rather than just listening to a pre-recorded description.
• For Sighted Users: Even people who can see the map found it helpful. Sometimes, a map can be misleading (a big state might look important just because it's big, not because it has high data). The AI helps sighted users double-check their visual intuition with hard facts.

What the Study Found

The researchers tested this with 6 blind users and 6 sighted users. Here is what they discovered:

• It Works: The blind users could successfully find patterns and make decisions (like deciding where to give funding) just as well as the sighted users.
• Different Styles, Same Goal: Blind users tended to ask more questions about shapes and relationships (e.g., "What states touch Florida?"), while sighted users often looked at the map first and then asked specific questions. But in the end, both groups found the same answers.
• The "Aha!" Moment: One blind participant said the system made the data "come to life." Instead of struggling to guess connections between numbers, the AI described the patterns, making the data feel real and understandable.

Why This Matters

Think of GeoVisA11y as a universal translator for data. Just as subtitles help deaf people enjoy movies, and curb ramps help wheelchair users enter buildings, this system helps blind people enter the world of spatial data.

But here is the twist: It helps everyone.
Just as a curb ramp is useful for a parent with a stroller, this system helps sighted people who might be bad at reading maps or who want to double-check their assumptions. It proves that when we design technology to be accessible for everyone, we often end up making it better for everyone.

In a Nutshell

GeoVisA11y is a tool that turns a silent, visual map into a chatty, helpful guide. It allows blind users to "see" the world through data by listening, and it helps sighted users understand the world by asking the right questions. It's not just about accessibility; it's about shared understanding.

Technical Summary

Here is a detailed technical summary of the paper "GeoVisA11y: An AI-based Geovisualization Question-Answering System for Screen-Reader Users."

1. Problem Statement

Geovisualizations (interactive maps) are critical tools for spatial analysis in fields like urban planning, public health, and news media. However, they are largely inaccessible to screen-reader users (blind and low-vision, or BLV).

• Current Limitations: Existing accessibility features (e.g., static alt-text, data tables) are largely descriptive ("map reading") rather than analytical. They fail to convey high-level spatial patterns, relationships, or geometric characteristics.
• Gap in Technology: Previous question-answering (QA) systems for maps relied on keyword matching or simple rule-based approaches, lacking the ability to handle complex, context-aware, or geostatistical queries.
• Research Goal: To bridge the accessibility gap by enabling screen-reader users to perform deep spatial analysis (pattern recognition, interpretation, and navigation) through natural language interaction, while also investigating if these tools benefit sighted users.

2. Methodology

A. System Design: GeoVisA11y

GeoVisA11y is an open-source, LLM-based system designed with an iterative, human-centered approach involving co-design sessions with two blind participants. It integrates a screen-reader compatible UI with a custom AI pipeline.

Key Components:

1. User Interface (UI):

   • Interactive Map: Supports discrete keyboard navigation (arrow keys) between states/counties and zooming, rather than continuous panning.
   • AI Chat: Supports text and voice input. It synchronizes bidirectionally with the map (e.g., asking "Focus on Ohio" updates the map; focusing on Ohio allows context-aware queries like "What is the density here?").
   • Accessibility: Adheres to WCAG guidelines, providing clear focus states, audio announcements, and keyboard shortcuts.

2. QA Pipeline (Technical Architecture):
   The system uses a four-stage pipeline powered by GPT-4o-mini (via few-shot prompting) to process queries without sending raw map images to the LLM, ensuring accuracy and speed.

   • Input Classifier: Distinguishes between action commands (e.g., "Go to Texas") and information queries.
   • Query Refiner: Resolves ambiguities (deictic references like "here" or "it") by ingesting the current map focus and chat history.
   • Scope Assessor: Determines if a query can be answered using local geospatial data or requires external knowledge (LLM knowledge base).
   • Query Processor: Classifies queries into 14 categories (e.g., Retrieve, Compare, Pattern, Shape) and triggers specific operations:
     • Local Operations: Direct database queries (DuckDB) and geostatistical analysis (Moran's I for global patterns, LISA for local clusters).
     • LLM Operations: Handling general knowledge, visualization concepts, and summarizing statistical results into natural language.

Supported Query Types:

• Analytical: Retrieval, comparison, extremum, aggregation, filtering, sorting, clustering.
• Geospatial: Pattern identification (hot/cold spots), outlier detection, shape description, spatial relationships (neighbors).
• Contextual: General knowledge and visualization concepts.

B. User Study

• Participants: 12 total (6 BLV screen-reader users, 6 sighted participants with varying geography knowledge).
• Tasks:
  1. Digital Equity Funding: Distribute grants based on underserved populations and digital access (using choropleth maps).
  2. Energy Sources: Identify predominant heating fuel sources across US regions (using dot density maps to remove immediate visual patterns).
• Evaluation: Mixed-method approach including quantitative metrics (query accuracy, task completion time) and qualitative analysis (interviews, open coding of interaction patterns).

3. Key Contributions

1. GeoVisA11y System: An open-source, accessible geovisualization QA system that moves beyond descriptive alt-text to support complex spatial analysis and navigation via natural language.
2. Empirical Findings on Interaction: A study revealing distinct interaction patterns between BLV and sighted users, challenging the assumption that accessibility tools only serve one group.
3. Geospatial Query Dataset: A dataset of 346 queries from both user groups to inform future research in accessible data visualization.
4. Design Principles: Synthesized guidelines for accessible map exploration, including discrete navigation, context-aware disambiguation, and synchronized map-chat interfaces.

4. Results

System Performance

• Query Support: The system successfully supported 92% of all user queries.
• Accuracy: 83.8% of responses were factually correct.
  • Error Sources: The primary failure points were the Query Refiner (26.8% of errors, failing to resolve "here/it") and the Scope Assessor (21.4% of errors, misclassifying local vs. external scope).
  • Limitations: Errors also stemmed from LLM hallucinations, unimplemented backend calculations (e.g., standard deviation), and requests for new visualizations beyond current capabilities.

User Engagement & Strategies

• BLV Users:
  • Relied heavily on verbal queries combined with keyboard navigation.
  • Frequently asked for visual descriptions (shapes, neighbors) and extremum/retrieval data.
  • Valued the system for enabling self-directed spatial exploration and recalling geographic knowledge.
  • Showed high trust in system descriptions but expressed a need for source verification.
• Sighted Users:
  • Used visual assessment first, then queried specific details.
  • Preferred aggregate and sort queries.
  • Used the system more when visual patterns were obscured (e.g., dot density maps) or when they lacked domain expertise.
  • Requested more sophisticated statistical analysis and direct manipulation (e.g., selecting multiple states via mouse).

Convergence of Understanding

Despite different interaction strategies, both groups identified similar spatial patterns in the data (e.g., identifying Mississippi as a high-need area for digital equity). This suggests GeoVisA11y successfully creates a "shared understanding" of geovisualizations across ability levels.

5. Significance and Future Directions

• Universal Design: The study supports the hypothesis that accessibility features designed for BLV users (like conversational analysis and clear pattern descriptions) significantly benefit sighted users, particularly those with limited geographic literacy or when visualizations are complex.
• Beyond "Map Reading": GeoVisA11y demonstrates that AI can facilitate "map analysis" and "interpretation" for screen-reader users, moving them from passive consumers of data to active analysts.
• Future Work:
  • Dynamic Prompting: Implementing guided questions to clarify user intent (e.g., distinguishing "fuel" from "heating fuel").
  • Direct Manipulation: Integrating mouse/touch selection with conversational interfaces to reduce semantic distance.
  • Scalability: Automating metadata ingestion to deploy the system as a plugin for existing geovisualization platforms globally.
  • Diverse Abilities: Expanding research to include low-vision users who may use hybrid strategies (screen magnification + voice).

In conclusion, GeoVisA11y represents a significant step toward inclusive geospatial data analysis, proving that LLM-driven natural language interfaces can democratize access to complex spatial insights for all users.