MCP-Driven Accessibility Tree Standardization for AI-Powered Screen Reader Agents
This paper proposes a conceptual framework that leverages the Model Context Protocol (MCP) to unify heterogeneous platform-specific accessibility APIs into a standardized, semantic-rich layer for LLM-based screen reader agents, thereby reducing integration complexity while enabling persistent user preferences and consistent cross-platform interaction.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
For decades, the way computers talk to people who cannot see the screen has relied on a hidden translation layer. When a user opens a window on a computer or a page on a phone, the software does not just draw pixels; it also builds a secret map of what those pixels represent. This map lists every button, link, and text box, assigning each a name and a role, such as "submit button" or "heading." Screen reader software reads this map aloud, allowing a blind user to navigate the digital world by listening rather than looking. However, a new kind of computer program has recently emerged: artificial intelligence agents that can look at a screen and click buttons on their own, just like a human would. These agents are designed to help people with disabilities by performing tasks autonomously. To do this, they need to understand the screen, but they currently face a difficult choice. They can either take a picture of the screen and try to guess what everything is, or they can try to read the secret map. Both methods have flaws. Taking a picture misses the precise names and roles that screen readers rely on, while reading the map is often slow and requires building a different translator for every single type of computer or phone.
A team of researchers from companies and universities in the United States has proposed a new way to solve this problem. They suggest building a universal translator that sits between the computer's operating system and the artificial intelligence agent. Instead of forcing the agent to learn how to read the secret maps of Windows, macOS, Android, and web browsers separately, this new system would act as a single, standard bridge. The researchers call this bridge the Model Context Protocol. They designed a system where the computer's native accessibility features feed into a central server, which then cleans up the information and presents it to the artificial intelligence in a neat, consistent format. This server would also remember the user's specific needs, such as how fast they want information read or how large buttons must be, and keep that information safe across different sessions and devices. The goal is to let the artificial intelligence agent ask for exactly what it needs—a specific part of the screen or a list of actions—without having to download the entire map of the interface every time.
The researchers did not build a fully working version of this system to test it on real people. Instead, they created a detailed architectural plan and compared it against the methods currently used in the field. They analyzed how existing artificial intelligence agents perceive screens, looking at the trade-offs between speed, accuracy, and the effort required to build the software. Their analysis suggests that their proposed system would significantly reduce the engineering work needed to support different platforms. Currently, developers must write unique code to extract information from every operating system. With this new standard, they would only need to write one set of instructions to connect to the bridge, and the bridge would handle the rest. The researchers also found that by asking for only small, specific parts of the screen map rather than the whole thing, the system could save a great deal of computer memory, which is a critical constraint for artificial intelligence models.
However, the study also makes it clear that this new bridge cannot fix a fundamental problem that exists in the current technology. The speed at which the artificial intelligence can see the screen is still limited by how fast the computer's own operating system can provide the information. If the native map on a phone or computer is slow to generate, the new system will be slow too. The researchers emphasize that their proposal does not magically speed up the underlying technology; it simply organizes the flow of information more efficiently. They also noted that the system relies on the quality of the information provided by the computer. If the original map is missing labels or has errors, the new system cannot invent the missing details. It can only translate what is there.
The team identified three main areas where their proposal offers the most value. First, it creates a shared language for artificial intelligence agents to talk to any screen, removing the need for custom-built solutions for every device. Second, it allows the system to remember a user's disability profile, such as their preferred reading speed or navigation style, so the agent can adapt to them automatically every time they log in. Third, it provides a structured way to ask for information in small chunks, preventing the artificial intelligence from being overwhelmed by too much data at once. The researchers argue that while their idea is not a finished product, it provides a clear path forward. They suggest that the next step is to build a working prototype for at least two different types of computers to measure the actual speed and accuracy. Until then, their work serves as a blueprint for how to make artificial intelligence agents more reliable and easier to build for the people who need them most.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.