💬 NLP

CoAct-1: Computer-using Multi-Agent System with Coding Actions

CoAct-1 is a novel multi-agent system that enhances computer automation by synergistically combining GUI-based control with direct programmatic execution, achieving a new state-of-the-art success rate of 60.76% and significantly improved efficiency on the OSWorld benchmark.

Original authors: Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, Caiming Xiong

Published 2026-02-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, Caiming Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, helpful assistant named CoAct-1. Its job is to do things on your computer for you, like organizing files, editing spreadsheets, or sending emails.

Before CoAct-1, most computer assistants were like clumsy toddlers. They could only interact with your computer by looking at the screen and physically "clicking" buttons or "typing" with a mouse, just like a human would. If you asked them to "find all the photos in a folder, resize them, and zip them up," the toddler-assistant would have to:

Click the folder.
Scroll down.
Click the first photo.
Hold Shift, click the last photo.
Right-click, hover over "Send to," click "Compressed folder."
Type a name.
Hit Enter.

If the toddler clicked the wrong icon, or if the screen loaded slowly, the whole task would fail. It was slow, frustrating, and prone to mistakes.

The New Idea: The "Magic Wand" vs. The "Mouse"

The authors of this paper realized that while clicking buttons is good for some things, it's terrible for others. So, they built CoAct-1, a team of three specialized workers who can switch between two modes of operation:

The Mouse Mode (The GUI Operator): This worker is great at visual tasks. They look at the screen, find the "Save" button, and click it. They are good at navigating menus and seeing what's on the screen.
The Magic Wand Mode (The Programmer): This worker doesn't click buttons. Instead, they write a short piece of code (a script) that tells the computer's brain exactly what to do. Instead of clicking 50 times to rename 1,000 files, they just say, "Rename all files in this folder to 'Vacation_1.jpg', 'Vacation_2.jpg'..." and the computer does it instantly.

The Boss: The Orchestrator

The secret sauce of CoAct-1 is the Orchestrator. Think of the Orchestrator as a smart project manager or a conductor of an orchestra.

When you give the system a big, complicated task (like "Prepare my tax report"), the Orchestrator breaks it down:

"Okay, I need to open Excel and find a specific table." -> That's visual. Orchestrator says: "Hey, Mouse Worker, you handle this."
"Now I need to copy that data into a text file and rename it." -> That's repetitive and precise. Orchestrator says: "No, Magic Wand Worker, you write a quick script for this. It's faster and won't make mistakes."
"Finally, I need to email the file." -> Orchestrator says: "Back to Mouse Worker, you need to click the 'Send' button."

Why This Changes Everything

The paper tested this new team against the old "clumsy toddler" assistants on two very difficult computer tests (OSWorld and WindowsAgentArena). Here is what happened:

Speed: The old assistants took about 15 steps to finish a task. CoAct-1 finished the same tasks in just 10 steps. It's like taking a shortcut through a forest instead of walking around the edge.
Success Rate: The old assistants succeeded about 50-55% of the time. CoAct-1 succeeded 60% to 65% of the time.
Reliability: When a task required doing something complex with files or data, the "Magic Wand" (coding) was a game-changer. It didn't get tired, it didn't click the wrong button, and it didn't get confused by a slow screen.

The Big Picture

Think of it this way:

Old Agents were like trying to build a house by only using a hammer and a screwdriver. You can do it, but it takes forever, and you might break a window.
CoAct-1 is like having a construction crew that knows when to use the hammer (for the nails) and when to use a power drill or a crane (for the heavy lifting).

By letting the computer "speak code" when it makes sense, and "click buttons" when it's necessary, CoAct-1 creates a much smarter, faster, and more reliable way to automate our computers. It's a giant leap toward having a true digital assistant that can actually get your work done without you having to babysit it.

1. Problem Statement

Current autonomous computer-using agents primarily rely on Graphical User Interface (GUI) manipulation (e.g., mouse clicks, keyboard typing) to interact with operating systems. While Vision-Language Models (VLMs) have improved these agents' capabilities, they face significant limitations in efficiency and reliability, particularly for complex, long-horizon tasks.

Brittleness: GUI agents are prone to visual grounding errors (e.g., misidentifying icons) and cumulative error propagation over long action sequences. A single mis-click can derail an entire task.
Inefficiency: Routine tasks involving file management, data processing, or system configuration often require hundreds of precise GUI steps that could be executed in a single line of code.
Limitations of Planners: While adding high-level planners (e.g., GTA-1) helps decompose tasks, it does not solve the fundamental inefficiency of executing every subtask via GUI interaction.

2. Methodology: The CoAct-1 Framework

The authors propose CoAct-1, a novel multi-agent system that introduces coding as a core action space. Instead of relying solely on GUI manipulation, CoAct-1 employs a hybrid approach where agents can dynamically choose between visual interaction and direct programmatic execution.

Core Architecture

The system consists of three specialized agents working in a hierarchical structure:

Orchestrator (Meta-Planner):
- Acts as the central controller ( $\pi_{Orch}$ ).
- Analyzes the user's high-level goal and the current system state (screenshot + history).
- Dynamic Delegation: Decides whether to assign a subtask to the Programmer (for backend/OS tasks) or the GUI Operator (for visual/frontend tasks).
- Maintains long-term memory of the task history and receives summarized updates from executors.
Programmer Agent:
- Implements the coding policy ( $\pi_{Code}$ ).
- Generates and executes Python or Bash scripts via a code interpreter.
- Operates in a multi-round loop: generate code $\rightarrow$ execute $\rightarrow$ observe feedback $\rightarrow$ refine code.
- Ideal for file management, data processing, and environment configuration.
GUI Operator:
- Implements the visual policy ( $\pi_{GUI}$ ).
- A VLM-based agent that performs standard GUI actions (clicks, typing, navigation).
- Operates in a perception-action loop, using screenshots to ground actions.

Workflow & Memory Design

Hierarchical Memory: The Orchestrator maintains long-term context, while the Programmer and GUI Operator maintain isolated, short-term working memory specific to their current subtask. This prevents context overload and ensures modularity.
Summarization: Once an executor completes a subtask, a summarizer model condenses the interaction history into a concise summary and a new screenshot, which is passed back to the Orchestrator for the next decision.
Hybrid Action Space: The action space is defined as $A = A_{GUI} \cup A_{Code}$ , allowing the system to bypass inefficient GUI sequences with robust, single-shot code execution when appropriate.

3. Key Contributions

Paradigm Shift: Introduces a hybrid action space that integrates direct programmatic execution (coding) with GUI manipulation, moving beyond the "GUI-only" constraint of previous agents.
Multi-Agent Orchestration: Designs a dynamic delegation system where an Orchestrator intelligently routes tasks to the most suitable agent (Programmer vs. GUI Operator) based on task requirements.
Efficiency & Robustness: Demonstrates that replacing long, error-prone click sequences with concise scripts significantly reduces the number of steps required and minimizes failure rates.
State-of-the-Art Performance: Establishes new benchmarks on major computer-use datasets, proving the scalability of this approach across different operating systems (Linux/Ubuntu and Windows).

4. Experimental Results

The system was evaluated on two challenging benchmarks: OSWorld (369 tasks) and WindowsAgentArena (154 tasks).

Performance Metrics

OSWorld: CoAct-1 achieved a 60.76% success rate (within 150 steps), surpassing the previous best (Agent S2.5 at 55.98%) and other strong baselines like GTA-1 (53.07%).
- Domain Highlights: Exceptional performance in OS tasks (75.00%), Office tasks (64.80%), and Multiple Apps (47.87%).
WindowsAgentArena: CoAct-1 achieved a 52.5% success rate, significantly outperforming Agent S2 (29.8%) and NAVI (19.5%).
- Domain Highlights: Dominated Windows System tasks (83.3%) and Windows Utils (77.7%).

Efficiency Analysis

Step Reduction: CoAct-1 completed tasks in an average of 10.15 steps on OSWorld, compared to 15.22 steps for GTA-1 and 14.90 for UI-TARS.
Ablation Studies:
- Pure Coding: Achieved high efficiency (1.14 steps/task) but lower success (35.73%) due to inability to handle visual tasks.
- Pure GUI: Achieved moderate success (50.68%) but required many more steps (11.20).
- Hybrid (CoAct-1): Combined the strengths of both, achieving the highest success rate (60.76%) with high efficiency (10.15 steps).
Model Sensitivity: Performance was highly sensitive to the reasoning capabilities of the Orchestrator. Using a powerful model (OpenAI o3) for the Orchestrator significantly boosted results.

5. Significance

CoAct-1 represents a critical advancement in the field of Generalized Computer Automation.

Scalability: By leveraging code for backend operations, the system scales effectively to complex, long-horizon tasks that were previously brittle for pure GUI agents.
Reliability: The hybrid approach mitigates the "cumulative probability of errors" inherent in long GUI sequences, making agents more robust for real-world deployment.
Future Direction: The paper suggests that the future of computer-using agents lies not just in better vision models, but in multi-modal action spaces that combine the intuition of GUI interaction with the precision and power of programmatic control.

In conclusion, CoAct-1 demonstrates that integrating coding as a primary action modality creates a more powerful, efficient, and scalable path toward fully autonomous computer agents.