🤖 AI

IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents

IntentCUA is a multi-agent framework that enhances long-horizon computer-use automation by leveraging shared memory to abstract multi-view intent representations and reusable skills, thereby stabilizing execution and significantly improving task success rates and efficiency compared to existing approaches.

Original authors: Seoyoung Lee, Seobin Yoon, Seongbeen Lee, Yoojung Chun, Dayoung Park, Doyeon Kim, Joo Yong Sim

Published 2026-03-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seoyoung Lee, Seobin Yoon, Seongbeen Lee, Yoojung Chun, Dayoung Park, Doyeon Kim, Joo Yong Sim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly scatterbrained robot assistant how to use your computer. You want it to do a complex task, like "Find the video I watched yesterday about machine learning, summarize it, and save the notes in my work folder."

If you just tell a standard AI to do this, it might get lost. It might click the wrong button, forget what it was doing halfway through, or keep trying to open a website it already visited. It's like a tourist in a huge city who keeps asking for directions, getting confused, and walking in circles.

IntentCUA is a new system designed to stop that robot from getting lost. Here is how it works, explained with everyday analogies:

1. The Problem: The "Forgetful Tourist"

Current computer agents are like tourists who only look at the street sign right in front of them. They don't remember where they've been or what their final destination is.

The Drift: They start with a plan but slowly wander off course.
The Loop: If they make a mistake, they often panic and try the same thing over and over again, wasting time.
The Noise: Computer screens are messy (pop-ups, changing windows), and these agents often get confused by the visual clutter.

2. The Solution: The "Experienced Project Manager"

IntentCUA acts less like a tourist and more like an experienced Project Manager who has a massive, organized filing cabinet of past experiences. It uses three main tricks to stay on track:

A. The "Intent Translator" (Learning the Why, not just the What)

Instead of just memorizing that "I clicked the blue button," IntentCUA learns the intent behind the action.

Analogy: Imagine you are teaching a child to cook. A standard robot memorizes: "Move hand to spoon, lift spoon, pour."
IntentCUA learns: "The goal is to add salt to the soup."
It looks at the action from four different angles (what app was open, what button was pressed, what keywords were used, and what the description said) to understand the true meaning of the step. This allows it to recognize that "clicking the search bar" and "typing a URL" are both part of the same goal: "Go to a website."

B. The "Skill Library" (Building a Toolbox)

Once the system understands the intent, it groups similar actions together into reusable "skills."

Analogy: Think of a carpenter. They don't reinvent how to hammer a nail every time they build a house. They have a "Hammering Skill" in their toolbox.
IntentCUA takes thousands of past user actions and turns them into Skill Hints. If it needs to "open a file," it doesn't guess; it pulls the "Open File" skill from its toolbox. This stops it from wasting time figuring out basic steps it already knows how to do.

C. The "Three-Person Team" (Planner, Optimizer, Critic)

The system doesn't rely on one brain; it uses a team of three specialized agents working together:

The Planner: The "Architect." It looks at your request and checks the "Skill Library" to build a high-level blueprint. It asks, "Do I have a plan for this? If not, let's build one using our skills."
The Optimizer: The "Foreman." It takes the blueprint and fills in the specific details for the current screen. If the screen looks slightly different than expected, it adjusts the steps on the fly.
The Critic: The "Quality Inspector." It watches the robot work. If the robot clicks the wrong thing, the Critic immediately says, "Stop! That didn't work. Let's try the backup plan," preventing the robot from getting stuck in a loop.

3. How It Saves the Day

When you give IntentCUA a long, complicated task:

It checks its memory first: "Have I done something like this before?" If yes, it reuses the old plan (saving huge amounts of time).
It fills in the gaps: If the old plan is 90% right, it uses its "Skill Hints" to fill in the missing 10% instead of starting over.
It stays focused: Because it understands the intent (the goal), it doesn't get distracted by a pop-up ad or a weirdly named button. It knows, "I need to get to the file," so it ignores the noise.

The Result

In tests, this system was a massive success:

Success Rate: It completed about 75% of complex tasks, while other systems only managed about 40-50%.
Efficiency: It took fewer steps to finish the job. It didn't waste time re-doing things.
Speed: It finished tasks 4.5 times faster than the competition because it didn't get stuck in loops.

In short: IntentCUA is like giving your computer assistant a map, a toolbox of pre-built solutions, and a team of supervisors. Instead of wandering aimlessly, it knows exactly where it's going, how to get there, and how to fix mistakes before they become disasters.

1. Problem Statement

Computer-use agents operating in desktop environments face significant challenges in long-horizon automation due to:

Noisy Perception & Dynamic Contexts: Desktop environments involve multi-window operations, OS-level shortcuts, and frequent UI changes, leading to local perception errors.
Intent Drift: Existing agents (RL-based or trajectory-retrieval based) often lose alignment with the original user intent over time, causing them to redundantly re-solve completed sub-problems.
Error Accumulation: Small perception errors compound over sequential steps, leading to cascading retries, inefficient loops, and task failure.
Lack of Abstraction: Current systems often operate at the raw action or trajectory level rather than abstracting interactions into reusable, intent-aligned skills, hindering generalization across heterogeneous applications.

2. Methodology: IntentCUA Framework

IntentCUA is a multi-agent framework designed to stabilize long-horizon execution through intent-aligned plan memory and hierarchical skill abstraction. It consists of three main components: a Planner, a Plan-Optimizer, and a Critic, all coordinating via a shared memory system.

A. Intent-Level Representation Learning & Skill Abstraction (Offline Phase)

The system transforms raw user interaction traces into structured, reusable knowledge:

Multi-View Labeling: Raw traces are labeled across four views: Environment ( $E$ ), Action ( $A$ ), Keyword ( $K$ ), and Description ( $D$ ).
- Control traces use $[E, A, D]$ .
- Browsing traces use $[E, K, D]$ .
Multi-View Embedding: A multi-view encoder maps these heterogeneous views into a shared latent space ( $z_i$ $z_{i}$ ). The model is trained using a composite loss function:
- Contrastive Loss ( $L_{con}$ ): Aligns embeddings from different views of the same intent while separating different instances.
- Prediction Loss ( $L_{pred}$ ): Ensures cross-view consistency (predicting one view from another).
- Reconstruction Loss ( $L_{rec}$ ): Ensures the shared embedding retains view-specific semantics.
Hierarchical Clustering:
- Intent Groups (IG): High-level clusters driven by environment/context (e.g., "Web Browser").
- Subgroups (SG): Finer clusters within IGs based on action patterns and descriptions (e.g., "Search and Download").
Skill Abstraction:
- Within each SG, low-level action traces are consolidated into a Skill Hint.
- Atomic actions are mapped to a canonical verb-predicate structure with typed arguments (e.g., text_input with <url>).
- A Skill Prototype (medoid) is induced for each SG, representing the most common action pattern.

B. Intent-Aware Planning & Execution (Online Phase)

The system executes tasks using a "Cache-First" planning strategy:

Planning (Planner Agent):
- The Planner queries the shared memory using the user's intent.
- Case 1 (Cache Miss): Synthesizes a new plan by retrieving relevant Skill Hints and filling gaps.
- Case 2 (Exact Hit): Reuses a previously approved global plan ( $G$ ).
- Case 3 (Partial Hit): Aligns with a stored plan and injects Skill Hints to fill missing steps (gap filling), reducing the need for full re-planning.
Optimization (Plan-Optimizer Agent):
- Refines specific plan units by referencing similar traces stored in memory.
- Adapts substeps dynamically based on the current screen context (GUI grounding).
Verification (Critic Agent):
- Monitors execution state after each step.
- Provides feedback signals: Success, Retryable, or Blocked.
- If "Retryable," the Plan-Optimizer adjusts the subplan based on the new state; if "Blocked," the task terminates or triggers higher-level replanning.

3. Key Contributions

IntentCUA Framework: A novel multi-agent system that stabilizes long-horizon desktop automation through coordinated planning and intent-aligned memory.
Trace-to-Skill Pipeline: A method to learn multi-view intent representations and induce hierarchical, parameterized skills from raw interaction traces, enabling cross-application transfer.
Memory-Grounded Planning: A mechanism that retrieves subgroup-aligned skills to augment partial plans, significantly reducing intent drift and redundant re-planning.
Empirical Validation: Extensive ablation studies and end-to-end evaluations demonstrating that intent abstraction and memory coordination are critical for execution stability.

4. Experimental Results

The system was evaluated on 286 real-world GUI tasks across diverse domains (local apps, web, cross-application workflows), including tasks from WebVoyager, ScreenAgent, and an in-house suite.

Task Success Rate: IntentCUA achieved 74.83%, significantly outperforming:
- UFO2 (Trajectory-centric): 51.2%
- UI-TARS-1.5 (RL-based): 38.8%
Step Efficiency Ratio (SER): IntentCUA achieved 0.91 (ratio of successful steps to total steps), compared to 0.85 (UI-TARS) and 0.82 (UFO2).
Latency: IntentCUA reduced execution time by approximately 4.5x compared to baselines (1.46 mins vs. ~6.6–9.8 mins), demonstrating superior scalability.
Long-Horizon Robustness:
- IntentCUA maintained stable performance even for tasks exceeding 30 steps (42.9% success), whereas baselines dropped below 20%.
- Success rates remained high across varying step lengths (85.9% for 10–15 steps), indicating resistance to error accumulation.
Ablation Studies:
- Adding Intent Subgroup (SG) gating and Intent Representations ( $Z$ ) provided the largest gains in stability.
- The combination of Skill Hints ($SSG$) and Plan Memory ($PM$) alone improved completion rates to 81.23%, proving the value of skill abstraction even without full representation learning.

5. Significance

IntentCUA addresses the critical bottleneck in desktop automation: maintaining goal coherence over long sequences in dynamic environments.

Shift from Replay to Abstraction: Unlike previous methods that rely on replaying raw trajectories, IntentCUA abstracts interactions into semantic skills, allowing for better generalization to unseen domains.
System-Level Coordination: It demonstrates that a cooperative multi-agent loop (Planner-Optimizer-Critic) grounded in shared memory is more effective than monolithic planners.
Practical Impact: The framework offers a path toward reliable, efficient, and scalable desktop automation agents capable of handling complex, multi-application workflows without constant human intervention or suffering from "drift."

In conclusion, the paper establishes that system-level intent abstraction combined with memory-grounded coordination is the key to achieving robust, long-horizon computer-use agents.