💻 computer science

DocOS: Towards Proactive Document-Guided Actions in GUI Agents

The paper introduces "Proactive Document-Guided Action" and proposes the DocOS benchmark to address the limitations of GUI agents in handling long-tailed tasks by enabling them to autonomously search for and ground external documentation into executable actions, revealing that current progress is hindered by challenges in information retrieval and instruction grounding.

Original authors: Jingjing Liu, Ziye Huang, Zihao Cheng, Zeming Liu, Jiahong Wu, Yuhang Guo, Kehai Chen, Yunhong Wang, Haifeng Wang

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingjing Liu, Ziye Huang, Zihao Cheng, Zeming Liu, Jiahong Wu, Yuhang Guo, Kehai Chen, Yunhong Wang, Haifeng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "All-Knowing" Robot That Gets Stuck

Imagine you have a very smart robot assistant (a GUI Agent) designed to help you use your computer. It has read millions of books and knows how to use common apps like Word or Chrome. It's great at everyday tasks.

But, what happens when you ask it to do something weird or very specific, like "Create a special run configuration for a Python template in PyCharm"? The robot freezes. Why? Because that specific instruction isn't in its memory. It tries to guess, fails, tries again, and eventually gives up or does the wrong thing. It's like asking a chef who knows how to make a burger to suddenly make a dish they've never seen, without a recipe. They might guess the ingredients, but the result will likely be a mess.

The Solution: Teaching the Robot to "Google It"

The authors of this paper propose a new way for these robots to work. Instead of relying only on what's inside their brain, they should be allowed to stop, open a web browser, search for the official manual, read it, and then follow the instructions.

They call this "Proactive Document-Guided Action."

Old Way: The robot guesses based on memory.
New Way: The robot realizes it doesn't know, searches the web for the "Official Guide," reads the steps, and then performs the task exactly as the guide says.

This mimics how humans solve problems: when we don't know how to fix a weird error on our computer, we don't guess; we search for a tutorial and follow it.

The Test: Introducing "DocOS"

To see if robots can actually do this, the team built a test called DocOS. Think of DocOS as a giant obstacle course for robots.

The Course: It contains 817 different tasks across 20 different computer programs (like PyCharm, Blender, VS Code, etc.).
The Challenge: The tasks are "long-tailed," meaning they are rare, specific, and difficult. You can't just guess the answer; you must find the right documentation to solve them.
The Rules: The robot has to:
1. Open a web browser.
2. Search for the right manual.
3. Read and understand the steps in that manual.
4. Go back to the computer and click/type exactly what the manual says.

What Happened? (The Results)

The researchers tested several top-tier robot agents on this course. The results were a bit of a reality check: The robots are still struggling.

They found two main "bottlenecks" (traffic jams) where the robots fail:

The "Lost in the Library" Problem (Search Failure):
Even when the robot knows it needs to search, it often can't find the right page. It might find the main website for the software but get lost in the thousands of other pages, never finding the specific manual it needs. It's like walking into a massive library and trying to find one specific book, but you keep picking up the wrong ones.
The "Bad Translator" Problem (Execution Failure):
Sometimes, the robot finds the right manual and reads the instructions. But when it tries to do the task, it messes up. It might understand the sentence "Click the blue button" but click the wrong blue button, or it might get confused by the complex layout of the screen. It fails to translate the words in the manual into the clicks on the screen.

The Takeaway

The paper concludes that while these AI agents are getting smarter, they aren't ready to be fully independent workers yet. They are like a student who has a great textbook but is terrible at navigating the library to find it, and even worse at following the instructions once they find them.

DocOS is a new tool to measure exactly how good (or bad) these robots are at this specific skill: finding a manual and following it. The authors hope that by using this test, researchers can build better robots that can truly help us with complex, real-world computer tasks without needing a human to hold their hand.

Technical Summary: DocOS – Towards Proactive Document-Guided Actions in GUI Agents

1. Problem Statement

Current Graphical User Interface (GUI) agents, driven by Multimodal Large Language Models (MLLMs), have demonstrated significant progress in automating device interactions across web, desktop, and mobile environments. However, a fundamental limitation persists: these agents rely heavily on static parametric knowledge acquired during pre-training or instruction tuning. While sufficient for common interactions, this reliance renders agents brittle when confronted with long-tailed, application-specific tasks that require explicit procedural knowledge absent from model parameters.

When faced with such tasks (e.g., configuring a specific run environment in PyCharm), agents often resort to inefficient and error-prone trial-and-error exploration, leading to high rates of execution failure and hallucinations. Existing benchmarks and retrieval-augmented methods typically assume that relevant documentation is pre-collected or provided, failing to address the challenge of autonomous knowledge acquisition in open, dynamic environments. There is a critical gap in evaluating an agent's ability to proactively search for, comprehend, and ground external documentation into executable GUI actions.

2. Methodology

2.1 Proposed Paradigm: Proactive Document-Guided Action

The authors introduce Proactive Document-Guided Action, a novel paradigm that mirrors human problem-solving. Instead of relying solely on internal knowledge, agents are empowered to autonomously navigate the open web to locate relevant documentation, comprehend procedural instructions, and faithfully ground these instructions into precise GUI actions. This paradigm decouples reasoning capabilities from the storage of vast, domain-specific knowledge, enabling adaptation to evolving tasks without extensive retraining.

2.2 The DocOS Benchmark

To evaluate this paradigm, the authors propose DocOS, a benchmark designed for fully interactive environments.

Task Formulation: The interaction is modeled as a Partially Observable Markov Decision Process (POMDP) decomposed into two sequential phases:
1. Proactive Knowledge Retrieval: The agent interacts with a web browser to search for and retrieve a target document ( $D$ ) containing necessary procedural knowledge.
2. Document-Grounded Execution: The agent executes the user instruction ( $I$ ) by grounding actions in the retrieved context ( $D$ ).
Data Construction: DocOS comprises 817 high-quality tasks across 20 applications (e.g., PyCharm, Blender, VS Code, Postman). Tasks are categorized into three difficulty levels (Easy, Medium, Hard) based on the number of execution steps (ranging from ~3 to ~11 steps on average).
Environment: The benchmark utilizes a dynamic execution environment (Docker-based) where agents must autonomously navigate a web browser (Microsoft Edge) to find documentation. The environment supports realistic visual perception and a comprehensive action space (mouse clicks, typing, scrolling, hotkeys, etc.).
Quality Control: A multi-stage pipeline involving task construction, automated document collection (crawling official sites), and rigorous filtering ensures semantic consistency, unambiguity, and executability.

2.3 Evaluation Metrics

The evaluation is split into two phases with specific metrics:

Documentation Search:
- Target URL Inclusion (TUI): Measures if the ground-truth URL is a substring of the retrieved URL, accounting for dynamic parameters.
- Hierarchical Path Progress (HPP): Evaluates the depth of the correct navigation path achieved, distinguishing between agents that are "lost" and those that reach high-level sections.
Task Execution:
- Task Completion Rate (TCR): The ratio of successfully completed tasks to the total number of tasks, serving as the decisive metric for practical utility.

3. Key Contributions

Novel Paradigm: The paper is the first to propose Proactive Document-Guided Action for GUI agents, shifting the focus from static knowledge to autonomous document retrieval and application.
DocOS Benchmark: Introduction of a comprehensive benchmark that evaluates the full pipeline of retrieval, understanding, and execution in realistic desktop environments, covering 20 applications and 817 tasks.
Systematic Evaluation: The authors conduct extensive experiments under two complementary settings: Proactive Document Search (evaluating autonomous retrieval) and Document-Grounded Execution (evaluating instruction implementation given documents).

4. Experimental Results and Analysis

Extensive experiments on DocOS using various state-of-the-art GUI agents (e.g., Qwen3-VL, UI-TARS, MAI-UI) reveal significant limitations:

Dual Bottlenecks: Progress is strictly constrained by two primary failures:
1. Inaccurate Information Localization: Agents struggle to reliably locate the exact relevant information within documentation websites during the proactive search phase. While they often reach high-level sections (stable HPP), they fail to pinpoint specific actionable pages (low TCR).
2. Unfaithful Grounding: Even when provided with "Oracle" (ground-truth) documents, agents frequently fail to translate retrieved instructions into precise, executable GUI actions.
Impact of Documents: Providing external documents generally improves performance across models (e.g., MAI-UI-8B saw a ~7.66% relative improvement), validating the necessity of the paradigm. However, the gains are not uniform; smaller models or those with limited context windows may struggle to utilize long-context documents effectively.
Step Complexity: There is a significant negative correlation between task step count and success rate. For tasks requiring more than 4 steps, the pass rate drops to nearly 0% for most models, even with document assistance, highlighting severe limitations in long-horizon planning and context retention.
Error Analysis: Common failure modes include:
- Imprecise Localization: Stopping at loosely relevant sections.
- Non-official Reference: Retrieving third-party blogs instead of authoritative docs.
- Execute Before Retrieval: Bypassing search and guessing based on parametric knowledge.
- Action Grounding Failure: Failing to translate high-level instructions into specific UI interactions (e.g., missing parameter adjustments).

5. Significance and Claims

The paper claims that DocOS provides a foundational understanding of the challenges facing future GUI agents. By exposing the dual bottlenecks of document retrieval and document-conditioned execution, the work underscores that current agents are not yet robust enough for self-guided deployment in dynamic environments.

The authors position DocOS not as a solved problem, but as a systematic tool to help researchers better understand the strengths and limitations of GUI agents. The findings suggest that enabling reliable, self-evolving GUI agents requires addressing the gap between retrieving procedural knowledge and faithfully grounding it into complex, multi-step actions. The paper concludes that while progress has been made in perception and reasoning, the ability to autonomously acquire and apply external knowledge remains a critical hurdle for generalist agents.