From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents

This paper introduces the Declarative Model Interface (DMI), a novel OS abstraction that separates high-level semantic planning from low-level interaction mechanisms to transform human-oriented GUIs into LLM-friendly primitives, thereby significantly boosting computer-use agent success rates and reducing interaction steps without requiring application source code modifications.

The Gist

Imagine you are trying to teach a brilliant but very literal robot how to use a computer. This robot (the AI) is incredibly smart at understanding complex instructions, but it has two major weaknesses:

1. It's bad at "fine motor skills": It struggles with the tiny, precise movements needed to click a specific button, drag a scrollbar, or scroll through a menu.
2. It gets tired easily: Every time it has to look at the screen, figure out where to click, and then click, it uses up a huge amount of its "brain power" (computing cost) and time.

Currently, we force this robot to use Graphical User Interfaces (GUIs)—the same screens with menus, icons, and buttons that humans use. This is like asking a master chef to cook a meal by manually turning every single valve on a gas stove, one by one, while also trying to read the recipe. It's slow, prone to mistakes, and exhausting.

The Problem: The "Human" Interface

The paper argues that current computer screens are designed for humans, not robots.

• For Humans: We are great at seeing a picture and saying, "Oh, that's a blue button, I'll click it." We don't need to know the code behind it.
• For Robots: They are terrible at guessing where a button is on a messy screen. They have to plan a long, step-by-step journey: "Move mouse to X, click, wait for menu to open, move to Y, click..." If they miss one step, the whole plan fails.

The Solution: The "Declarative Model Interface" (DMI)

The authors propose a new way for computers to talk to robots called DMI. Think of DMI as a specialized translator or a concierge that sits between the robot and the computer screen.

Instead of the robot giving a long list of physical instructions, it simply tells the concierge what it wants to happen.

The Three Magic Tools of DMI

DMI turns the messy computer screen into three simple, magic commands:

1. Access (The "Find Me" Button):

   • Old Way: "Move mouse to the top left, click 'File', wait, move to 'New', click 'Word Document'..."
   • DMI Way: "Go to the 'New Word Document' button."
   • Analogy: Instead of giving the robot a map and telling it to turn left at the bakery, right at the library, and then knock on the door, you just say, "Go to the library." The robot (via DMI) knows exactly how to get there without getting lost.

2. State (The "Set It" Button):

   • Old Way: "Click the scrollbar, drag it down a little, look at the screen, drag it a bit more, look again..."
   • DMI Way: "Set the scrollbar to 80%."
   • Analogy: Instead of telling a driver to "press the gas pedal until you see the mountain," you just say, "Drive to the mountain." The car (DMI) handles the steering and speed automatically.

3. Observation (The "Tell Me" Button):

   • Old Way: "Take a picture of the screen, read the text, tell me what it says."
   • DMI Way: "What is the text in this box?"
   • Analogy: Instead of asking the robot to squint at a blurry sign and guess the words, you just ask the sign itself, "What do you say?"

Why This Changes Everything

The paper calls this "Policy vs. Mechanism Separation."

• Policy (The Plan): This is the "What." The robot decides, "I need to make the background blue."
• Mechanism (The How): This is the "How." The robot used to have to figure out, "Click here, then there, then drag this."

DMI takes the "How" away from the robot and gives it to the computer system. The robot only has to worry about the "What."

The Results:
When the researchers tested this with Microsoft Office (Word, Excel, PowerPoint):

• Success Rate: The robots got the job done 67% more often.
• Speed: They finished tasks 43% faster because they didn't have to take 20 tiny steps to do one big thing.
• Efficiency: In over 60% of cases, the robot could finish the whole task with just one single instruction to the computer, rather than a long back-and-forth conversation.

The Bottom Line

This paper suggests that we shouldn't force AI to act like a human clicking a mouse. Instead, we should build computer interfaces that speak the AI's language: direct, declarative, and high-level.

It's the difference between telling a genie, "Rub the lamp, then say 'I wish for a sandwich,' then wait for the smoke to clear," versus just saying, "I wish for a sandwich." The genie (DMI) handles the magic; the wisher (the AI) just focuses on the goal.

Technical Summary

1. Problem Statement

Computer-Use Agents (CUAs) powered by Large Language Models (LLMs) aim to automate complex computer tasks. However, they struggle significantly when interacting with existing Graphical User Interfaces (GUIs), which are designed for humans, not AI agents.

The core issues stem from the imperative nature of GUIs:

• Procedural Navigation: GUIs hide functionality behind hierarchical menus and tabs. LLMs must generate long, error-prone sequences of navigation actions (e.g., clicking menus, tabs, dropdowns) just to make a target control visible. This increases token costs and the risk of cascading failures.
• Iterative Interaction: Many GUI tasks (e.g., dragging a scrollbar, selecting text) require high-frequency "observe-act" loops. LLMs suffer from high inference latency (10–120s per round-trip) and weaker visual acuity compared to humans, making these loops prohibitively expensive and unreliable.
• Policy-Mechanism Coupling: In current GUIs, the policy (what the user wants to achieve) is tightly coupled with the mechanism (how to navigate and click). This forces LLMs to handle low-level spatial reasoning and visual processing, which are not their strengths, rather than focusing on high-level semantic planning.

2. Methodology: Declarative Model Interface (DMI)

The authors propose DMI, an abstraction layer that transforms imperative GUI interactions into declarative primitives. DMI separates policy (LLM's role) from mechanism (system's role).

Core Design Principles

1. Policy-Mechanism Separation: The LLM specifies the desired outcome (e.g., "set scrollbar to 50%"), while DMI handles the deterministic execution (navigation, clicking, dragging).
2. Three Declarative Primitives:
   • Access: Navigate to a target control and perform a primitive interaction (e.g., click) based on a unique identifier.
   • State: Set a control to a specific state (e.g., set_scrollbar_pos(80%), select_lines(start, end)).
   • Observation: Retrieve structured data (e.g., text content) without relying on pixel-level OCR or visual recognition.

Technical Implementation

• Navigation Topology Modeling:
  • The system constructs a UI Navigation Graph (UNG) where nodes are controls and edges represent click-induced transitions.
  • To resolve path ambiguity (cycles and merge nodes where one control is reachable via multiple paths with different semantics), the graph is transformed into a path-unambiguous forest.
  • A cost-based selective externalization algorithm is used: it clones substructures only when necessary to maintain unique paths, otherwise using shared subtrees with reference IDs. This prevents exponential node growth while preserving semantic correctness.
• Context-Efficient Representation:
  • Instead of dumping the entire UI tree into the LLM context, DMI provides a compressed, hierarchical description of the navigation topology.
  • It employs a "Query on Demand" mechanism: the LLM receives a core topology by default and can request specific branches or the full forest if needed, balancing context window limits with task requirements.
• Robust Execution:
  • Fuzzy Matching: Handles UIA identifier variations by matching control types, hierarchies, and names.
  • Error Handling: Provides structured error feedback and retry mechanisms for unstable UI states.
  • Instruction Filtering: Automatically filters out non-leaf (navigation) nodes from LLM outputs, ensuring the system only executes commands targeting functional (leaf) controls, even if the LLM includes navigation steps in its reasoning.

3. Key Contributions

1. Identification of the Mismatch: The paper formally identifies that human-centric OS interfaces (GUIs) are fundamentally ill-suited for LLMs due to the mismatch between LLM capabilities (strong reasoning, weak visual perception) and GUI design (visual-heavy, procedural).
2. DMI Abstraction: The introduction of a novel OS interface layer that decouples policy from mechanism, allowing LLMs to focus on semantic planning while the system handles deterministic navigation and interaction.
3. Algorithmic Solutions: Development of a cost-aware algorithm to transform complex UI graphs into path-unambiguous forests and a context-efficient querying mechanism to manage LLM token limits.
4. Comprehensive Evaluation: A rigorous evaluation across Microsoft Office Suite (Word, Excel, PowerPoint) demonstrating significant performance gains over state-of-the-art baselines.

4. Experimental Results

The system was evaluated using the OSWorld-W benchmark (27 single-app scenarios) with GPT-5 and GPT-5-mini.

• Success Rate: DMI improved the task success rate by 67% (from 44.4% to 74.1% with GPT-5 medium reasoning).
• Efficiency: It reduced the number of interaction steps (LLM calls) by 43.5% (from 8.16 to 4.61).
• Latency: Task completion time decreased by 39%.
• One-Shot Completion: Over 61% of successful tasks were completed in a single LLM call (4 steps total, including framework overhead), compared to the baseline which required multiple rounds of observation and action.
• Failure Analysis:
  • Baseline (GUI-only): Failures were dominated by mechanism-level errors (navigation errors, visual recognition failures, interaction mistakes).
  • DMI: Failures shifted almost entirely to policy-level errors (semantic misunderstandings, task planning). This confirms that DMI successfully offloaded the mechanical burden from the LLM.

5. Significance and Impact

• Paradigm Shift: The paper argues for a fundamental shift in OS interface design from Imperative (human-centric, action-based) to Declarative (agent-centric, state-based).
• Scalability: By removing the need for LLMs to plan fine-grained navigation, DMI makes computer-use agents more scalable, cost-effective, and reliable.
• Generalizability: While implemented on Windows using UIA, the principles apply to other OSes (macOS, Linux, Android) that offer accessibility APIs. It offers a path forward for automating complex workflows without requiring application-specific APIs or fine-tuning of the LLM.
• Future of Agent-OS Interaction: It establishes that operating systems must evolve to serve a new class of users (LLMs) with distinct constraints, rather than forcing agents to mimic human limitations.

In summary, DMI acts as a "fast path" for LLM agents, handling the deterministic, mechanical aspects of GUI interaction so that the LLM can focus on the "slow path" of high-level semantic reasoning and planning.