💬 NLP

Autonomous Continual Learning of Computer-Use Agents for Environment Adaptation

ACuRL is an autonomous curriculum reinforcement learning framework that enables computer-use agents to continually adapt to diverse digital environments without human annotation by using a task generator to synthesize tailored training tasks and a robust automatic evaluator for reliable feedback.

Original authors: Tianci Xue, Zeyi Liao, Tianneng Shi, Zilu Wang, Kai Zhang, Dawn Song, Yu Su, Huan Sun

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianci Xue, Zeyi Liao, Tianneng Shi, Zilu Wang, Kai Zhang, Dawn Song, Yu Su, Huan Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you just hired a highly skilled personal assistant to help you manage your digital life—someone who can use Excel, send emails, and navigate complex scientific software.

At first, this assistant is great at general tasks, but the moment you hand them a specific, niche piece of software you use for work (like a specialized astronomy tool or a specific company spreadsheet), they freeze. They don't know where the buttons are, they don't understand the "logic" of that specific app, and if you update the software, they suddenly forget everything they just learned.

This paper introduces ACuRL, a way to give that assistant a "brain" that learns and adapts to any new software automatically, without you ever having to sit down and teach them.

Here is how it works, broken down into three simple parts:

1. The "Curiosity" Phase (Environment Exploration)

Instead of you giving the assistant a manual, the researchers let the assistant "play" with the new software first.

The Analogy: Imagine dropping a toddler into a new playground. They don't read a map; they run around, touch the swings, climb the slide, and see what happens when they push the buttons. By "playing," the assistant learns, "Okay, this is a spreadsheet app, and clicking this icon opens a menu." This builds a foundation of "common sense" about the new environment.

2. The "Smart Teacher" (Curriculum Learning)

Once the assistant starts learning, they need a teacher. But in this system, the teacher is an AI that creates a customized workout plan (a curriculum) that evolves as the assistant gets stronger.

The Analogy: Think of it like a video game that adjusts its difficulty in real-time.

If the task is too easy: The teacher gives them a "Boss Level" task (e.g., "Don't just make a chart; make a complex, multi-colored chart with three different data sets").
If the task is too hard: The teacher breaks it down into "Mini-Games" (e.g., "Before you make the whole chart, just practice clicking the 'Insert' button").
If the task is medium: The teacher gives them variety (e.g., "You know how to edit a document; now try doing it with a different type of file").

This ensures the assistant is always challenged but never overwhelmed, preventing them from getting frustrated and giving up.

3. The "Strict Judge" (CUAJudge)

In many AI training methods, the "teacher" is a bit lazy—they just look at the final result and say "Good job" or "Bad job." But with computer tasks, that's not enough. If you ask an assistant to "Change the font to blue," and they change it to red, they didn't just fail; they did the wrong thing.

The Analogy: Imagine a cooking competition judge. A lazy judge just tastes the final dish and says, "It's salty." A CUAJudge is a Master Chef who watches the entire process. They check: "Did you use the right salt? Did you chop the onions correctly? Did you actually turn the stove on?"

CUAJudge looks at the "before" and "after" screenshots and the history of every click to make sure the assistant didn't just get lucky, but actually followed the instructions perfectly.

Why does this matter?

The researchers found that this method is incredibly efficient. Even as the assistant learns new, complex software, they don't "forget" how to use their old tools (this is called avoiding Catastrophic Forgetting).

It’s like a student who learns Calculus but doesn't suddenly forget how to do basic Addition. Because the assistant's brain updates in a very "sparse" and targeted way, they keep their old skills sharp while constantly adding new ones to their toolkit.

In short: ACuRL turns a "generalist" AI into a "specialist" that can master any digital tool it encounters, entirely on its own.

Technical Summary: Autonomous Continual Learning of Computer-Use Agents for Environment Adaptation (ACuRL)

1. Problem Statement

Current Large Language Model (LLM)-based Computer-Use Agents (CUAs) face two critical limitations when transitioning from benchmarks to real-world applications:

Environment Diversity: Real-world digital spaces contain millions of applications. Agents frequently encounter "out-of-distribution" environments where their pre-trained knowledge of UI logic and functionalities is insufficient.
Environment Dynamism: Software interfaces and functionalities evolve over time, causing distribution shifts that render static models obsolete.

Existing solutions struggle because obtaining high-quality, environment-grounded training data is prohibitively expensive via human annotation, and evaluating long-horizon, multimodal trajectories (screenshots + actions) is difficult to automate without introducing reward hacking or inaccuracies.

2. Methodology: The ACuRL Framework

The authors propose ACuRL (Autonomous Curriculum Reinforcement Learning), a zero-human-data framework that enables agents to adapt to new environments through an iterative, self-evolving loop. The framework consists of three core components:

A. Environment-Grounded Task Generation
To ensure tasks are realistic and executable, the agent undergoes two preliminary stages:

Environment Exploration: The agent autonomously interacts with the target environment to collect "exploration trajectories" (actions and observations) to understand the UI and available functions.
Context Review: The agent reviews diverse, real-world contexts (e.g., specific documents, emails, or spreadsheets) crawled from the web. This ensures generated tasks are grounded in meaningful, diverse user scenarios rather than abstract button-clicking.

B. Iterative Curriculum Reinforcement Learning
Instead of a fixed task pool, ACuRL uses a Curriculum Task Generator that adjusts task difficulty based on the agent's current success rate ($SR$):

Easy Tasks ( $SR > 70\%$ ): The generator synthesizes more complex variants (e.g., longer horizons or additional constraints).
Medium Tasks ( $30\% \le SR \le 70\%$ ): The generator maintains difficulty but increases diversity by changing user personas or scenarios.
Hard Tasks ( $SR < 30\%$ ): The generator performs hierarchical decomposition, breaking complex tasks into simpler, skill-focused subtasks to provide a learning scaffold.

C. CUAJudge: Robust Automatic Evaluation
To provide reliable reward signals for RL, the authors introduce CUAJudge. Unlike simple state-comparison tools, CUAJudge uses a three-stage pipeline:

Key Point Identification: Extracts critical requirements from the task.
Key Screenshot Identification: Selects informative frames from the long trajectory.
Outcome Judgment: Performs state-difference analysis (comparing initial vs. final states) and evidence-grounded verification (requiring the model to cite specific screenshots/actions to prove key points were met).

3. Key Contributions

First Autonomous Continual Learning Framework for CUAs: Enables adaptation to new software without any human-annotated data.
Adaptive Curriculum Mechanism: A novel way to co-evolve task difficulty with agent capability, preventing both overfitting and stagnation.
CUAJudge Evaluator: A high-fidelity automatic evaluator achieving 93% agreement with human judgments, specifically designed for the complexities of computer-use trajectories.
Infrastructure Optimization: A unified environment management protocol and asynchronous preloading/evaluation system that accelerates training by 3–5x.

4. Experimental Results

The framework was tested across six environments (LibreOffice suite, Thunderbird, and scientific tools like Celestia and KAlgebra):

Performance Gains: Achieved 4–22% improvement in success rates on target environments across iterations.
Mitigation of Catastrophic Forgetting: The agent preserved (and sometimes improved) performance on previously learned environments, demonstrating effective cross-environment continual learning.
Parameter Sparsity: Analysis revealed that only ~20% of parameters are substantially updated during adaptation. This high sparsity explains how the agent learns new skills without overwriting old ones.
Task Validity: 94% of the autonomously generated tasks were found to be meaningful and executable by human evaluators.

5. Significance

ACuRL represents a significant step toward general-purpose, self-improving digital assistants. By removing the "human-in-the-loop" requirement for data collection and evaluation, it provides a scalable pathway for agents to master niche, professional, or rapidly changing software environments. The research demonstrates that through structured curriculum learning and robust automated feedback, agents can achieve specialized expertise while maintaining generalist capabilities.