💻 computer science

LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents

This paper introduces LUMOS, a semantic operating-system layer that bridges AI agents and native interfaces by converting accessibility metadata into machine-readable blueprints, thereby reducing reliance on inefficient visual interpretation methods like screenshots and OCR.

Original authors: Yogeswar Reddy Thota

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yogeswar Reddy Thota

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but very literal, robot how to use your computer.

The Problem: The "Pixel" Language
Right now, operating systems (like Windows or macOS) are designed for humans. We see a blue button and know it's clickable because of its color, shape, and shadow. If you ask a standard AI agent to "click the blue button," it has to take a picture of your screen (a screenshot), look at the pixels, guess which ones make up the button, and then calculate exactly where to move the mouse.

This is like trying to give directions to a friend by describing the exact shade of blue of their shirt and the number of steps they need to take, rather than just saying, "Walk to the red door." It's slow, expensive (in terms of computer power), and prone to mistakes. The AI might click the wrong thing because it misread a shadow as a button.

The Solution: LUMOS (The "Translator" Layer)
The paper introduces LUMOS, which acts like a universal translator sitting between the AI and your computer.

Instead of showing the AI a picture of the screen, LUMOS reads the computer's internal "ID card" for every single item on the screen. Modern computers already have this information built-in for accessibility tools (like screen readers for the blind). LUMOS grabs this data and turns it into a clean, simple list for the AI.

The Analogy: The Restaurant Menu vs. The Kitchen Window
Think of the computer screen as a restaurant kitchen.

The Human Way (Screenshots): The AI is a customer looking through a dirty window at the kitchen. It sees a blur of shapes and has to guess, "Is that a burger or a sandwich? Is the chef holding a knife or a spoon?"
The LUMOS Way: LUMOS is the waiter who walks into the kitchen, reads the order ticket, and comes back to the AI with a clear list: "Item A2 is a 'Submit' button. It is currently active. You can click it."

How It Works (The "Observe-Act" Loop)
LUMOS doesn't just give the AI a static list; it keeps the conversation going in a loop:

Observe: LUMOS asks the computer, "What is on the screen right now?" It gets a compact blueprint (e.g., "A2 is a text box with the word 'Hello'").
Plan: The AI (the brain) reads this blueprint and decides what to do next. It sends a simple command like: "Type 'World' into A2."
Act: LUMOS executes that command exactly where it belongs.
Repeat: LUMOS looks again to see if the text changed, and the cycle continues.

Key Features Explained Simply

No More Guessing Coordinates: Instead of saying "Click at pixel 450, 200," LUMOS says "Click the 'Save' button." If the window moves, the button's ID stays the same, so the AI doesn't get lost.
The "Magic Pointer": If you move your mouse cursor over an item, LUMOS can instantly tell the AI, "The cursor is currently hovering over the 'Delete' button." It doesn't need to take a picture of the cursor; it just asks the computer, "What is under the pointer?"
Safety First: LUMOS acts like a responsible supervisor. If the AI tries to do something dangerous (like delete a file), LUMOS can stop it or ask for confirmation, ensuring the AI acts like a human user, not a rogue hacker.

What This Paper Actually Claims
The authors are not saying LUMOS can solve every computer problem yet. They are demonstrating that:

We don't need to reinvent the wheel; we can use the accessibility tools already built into computers.
This method is much more efficient and accurate than forcing AI to "look" at screenshots.
It creates a safer, more reliable way for AI to interact with software without needing to rewrite the software itself.

In short, LUMOS turns the messy, visual world of a computer screen into a clean, logical conversation that an AI can actually understand and follow.

Technical Summary: LUMOS – A Semantic Operating-System Layer for Accessibility-Grounded AI Agents

1. Problem Statement

Current operating systems (Windows, macOS, Linux) are optimized for human users, relying on visual metaphors (pixels, icons, colors, animations) rather than machine-readable structures. While humans benefit from visual grouping and depth, AI agents require compact semantic state, grounded actions, and reliable feedback.

Consequently, existing computer-use agents typically rely on screenshots and Visual-Language Models (VLMs) to interpret interfaces. This approach introduces several critical inefficiencies:

High Token Costs: Parsing high-entropy visual data consumes significant context window space.
Visual Ambiguity & Latency: Agents must infer UI semantics (e.g., "is this button clickable?") from pixels, leading to coordinate uncertainty and slower inference.
Brittleness: Agents struggle with long-horizon workflows spanning multiple applications because they lack a stable, symbolic representation of the system state.

The paper argues that operating systems already possess the necessary infrastructure to solve this: Accessibility APIs (such as Microsoft UI Automation on Windows and the DOM/Accessibility Trees in browsers). These APIs expose structured metadata (roles, names, values, bounds) originally designed for screen readers and assistive technologies, yet they remain underutilized by AI agents.

2. Methodology: The LUMOS Architecture

LUMOS (Language-Model Unified Machine-Readable Operating-System Semantics) is proposed as a middleware layer that sits between the AI agent and the operating system. It does not replace the OS kernel or the visible interface; instead, it creates a parallel, machine-readable interaction plane.

The system operates on an observe–plan–act loop with the following components:

A. Perception Layer (Semantic Blueprinting)

Instead of capturing a screenshot, LUMOS queries the OS for native UI structures:

Native Apps: Queries the UI Automation (UIA) tree for foreground windows, extracting elements as structured objects with properties like role, name, value, bounds, and supported patterns.
Web Apps: Extracts DOM and accessibility tree information.
Output: A compact "semantic blueprint" containing stable element identifiers (e.g., A2), roles, and current states, excluding decorative visual details.

B. Live Semantic Pointer Grounding

To address the ambiguity of "what is under the cursor," LUMOS implements a semantic pointer mechanism.

Instead of cropping a screenshot around the cursor and using OCR/Vision to guess the target, LUMOS queries the OS directly (e.g., ElementFromPoint in Windows).
This maps physical coordinates $(x, y)$ directly to a semantic element ID, providing immediate access to the element's role, name, and state without visual inference.

C. Planner Layer

The LLM planner receives the user goal, recent memory, and the current semantic blueprint.

Constraint: The planner must emit a single JSON action from a constrained schema.
Discipline: This prevents the generation of hallucinated long scripts. The agent must re-observe the system state after every action, mimicking human behavior (click, check, type, check).

D. Universal Action Schema

LUMOS exposes a small, application-neutral set of visible UI primitives:

observe: Refresh perception.
open_windows_search, open_app: Launch applications.
click, double_click, drag: Interact with identified elements.
type_text, set_text: Input or replace text.
press_key: Keyboard navigation.
finish: Explicitly signal task completion.

E. Memory and Safety Layers

Memory: Tracks recent actions and text states to prevent errors like appending text that should replace existing content or repeating failed actions.
Safety: Enforces an allowlist and confirmation policy. Risky operations (e.g., sending emails, deleting data) require explicit confirmation. The agent acts as a "visible user" via the UI, not a backend process, ensuring actions are inspectable and reversible.

3. Key Contributions

The paper outlines four primary contributions:

Semantic Operating-System Layer: A middleware that transforms native accessibility metadata into machine-readable semantic blueprints for AI agents.
Live Semantic Pointer Grounding: A mechanism for cursor-position-aware interaction using live UI queries (e.g., ElementFromPoint), bridging physical coordinates and semantic elements.
Accessibility-Grounded Observe–Act Loops: An alternative to screenshot/OCR-centric approaches where agents navigate via semantic roles, values, and bounds.
Path Toward AI-Native Computing: Evidence that existing OSs can expose a machine-readable interaction plane without requiring application redesign.

4. Results and Evaluation

The paper presents a prototype implementation in Python running on Windows, validated through specific case studies and diagnostic traces rather than large-scale human-subject benchmarks.

Case Studies:
- Notepad: Successfully demonstrated opening Notepad, generating text, and handling text replacement (set_text) vs. appending. The system identified that "stopping" is an action, requiring an explicit finish command to prevent infinite revision loops.
- Windows Search Handoff: Demonstrated a robust workflow for launching applications (e.g., Outlook) via the Windows Search overlay. The system preserves the pending query ("outlook") and ensures it is typed and submitted correctly before proceeding, preventing context hijacking.
Diagnostic Findings:
- Debug logs revealed common LLM failure modes, such as copying instructions literally, appending text incorrectly, or repeating fragments.
- LUMOS's memory and repair layers successfully mitigated these by rejecting non-content, enforcing set_text for corrections, and requiring explicit completion signals.
Proposed Evaluation Plan: The paper outlines a future evaluation framework comparing LUMOS against screenshot-based agents across four metrics: task success, latency, token count, and observation size. It hypothesizes that semantic grounding will reduce token usage and latency while improving action grounding accuracy.

5. Significance and Claims

The paper positions LUMOS as a step toward AI-native computing, arguing that accessibility APIs should be re-purposed as the "cognitive infrastructure" for AI agents.

Modest Claims: The authors explicitly state that LUMOS does not replace visual agents; rather, it reduces dependence on screenshots when semantic structure is available. It is not a claim of human-level autonomy, but a demonstration of a specific interaction architecture.
Core Argument: The significance lies in the architectural shift from "perception-heavy" (pixels) to "semantics-first" (structured state). By treating accessibility trees as the primary interaction substrate, the system separates the OS's role (providing structured state) from the LLM's role (planning).
Future Implications: The paper suggests that future operating systems could expose richer semantic states directly (e.g., application intentions, security boundaries) to facilitate safer and more efficient agent interaction, moving beyond the current model where agents must infer everything from visual surfaces.

In conclusion, LUMOS demonstrates that leveraging existing OS accessibility metadata can create a safer, more efficient, and more grounded interface for AI agents, offering a viable middle ground between brittle scripts, expensive visual agents, and hidden backend APIs.