Computers have become powerful tools for handling complex information, yet asking a machine to perform a multi-step office job often feels like trying to teach a child to drive by only describing the destination. For years, researchers have tried to build artificial intelligence agents that can look at a screen, understand what needs to be done, and click the right buttons to get there. These systems, known as GUI agents, have made progress, but they struggle with two persistent problems. First, they lack enough high-quality practice data to learn the subtle, unspoken rules of how people actually use software. Second, they are often too rigid; if a human gives a vague instruction, the machine might succeed once by luck but fail the next time because it cannot adapt to small changes in the screen or the user's specific habits. The goal has been to create a digital worker that is not just smart enough to follow a script, but reliable enough to handle the messy reality of a real desktop.

A team from Tencent has introduced a new system called UI-Mate that tackles these issues by changing how the agent learns and how it receives help. Instead of just memorizing a list of clicks, the system is trained in a simulated environment where it can practice thousands of tasks, from organizing spreadsheets to managing email, while a specialized engine checks its work in real time. This process allows the system to learn not just what the final result should look like, but how to recover when it makes a mistake. The researchers found that by training the agent in this rigorous, closed-loop way, they could create a model that performs significantly better than previous open-source systems, even matching the capabilities of much larger, proprietary models.

The core of this improvement lies in a method the researchers call "environment-grounded training." Imagine a student learning to cook not just by reading a recipe, but by actually standing in a kitchen, chopping vegetables, and having a teacher immediately point out if the knife slipped or the heat was too high. In the same way, UI-Mate is trained in a digital environment where every action it takes is tested against a real, working computer screen. If the agent tries to save a file in the wrong place, the system knows immediately and records that failure. This cycle of trying, failing, and correcting happens millions of times across different applications, allowing the agent to build a deep understanding of how software behaves. The researchers built a massive library of these practice tasks, covering everything from simple file management to complex workflows that require moving information between a calendar, an email client, and a project management tool. By carefully balancing the types of tasks the agent sees, they ensured it learned to handle both short, simple jobs and long, complicated sequences without getting confused.

However, even a well-trained agent can struggle when a human gives a vague instruction, such as "process this candidate's offer." The agent might not know which specific files to open or which exact buttons to click because the instruction lacks the necessary detail. To solve this, the researchers developed a new way for the agent to learn from a demonstration. Instead of forcing the agent to blindly copy a recorded sequence of mouse clicks, the system breaks the demonstration down into a flexible plan of subtasks. It looks at a video of a human or a stronger AI completing a similar job and extracts the goals: "find the offer," "verify the salary," "send the email." During the actual work, the agent uses this plan as a guide but still looks at the live screen to decide exactly how to move the mouse or type the text. This means that if the screen looks slightly different than in the video, the agent can adjust its actions rather than getting stuck.

The results of this approach are striking. When tested on a standard set of computer-use challenges, the new system, specifically the 27-billion-parameter version, achieved a success rate of 77.0 percent, outperforming other open-source models and coming very close to the best closed-source systems available. More importantly, the researchers created a new test called OSWorkerBench, which simulates realistic office work across dozens of different applications. On this test, the system improved its performance by nearly 18 percentage points compared to its base model. The most significant finding, however, came when they added a single demonstration to the agent's instructions. For a set of 33 long, complex tasks, providing just one example of how to do the job raised the agent's success rate from 17.2 percent to 35.4 percent. This shows that the system does not just memorize the example; it understands the underlying logic and can apply it to new situations, making it much more reliable for real-world use.

The researchers also discovered that the agent's ability to handle long tasks is not just about raw computing power. They tested models of different sizes and found that while larger models generally performed better, the specific training method mattered more than the size alone. A smaller model trained with their rigorous, environment-grounded approach outperformed a much larger model that had not received the same training. This suggests that the quality of the practice data and the feedback loop are more critical than simply making the brain of the agent bigger. The system also proved capable of handling tasks that require remembering information from the beginning of a workflow and using it at the very end, a common failure point for previous AI agents.

To ensure these results were not just a fluke, the team built a benchmark that included 100 distinct office tasks, ranging from human resources to sales and engineering. They evaluated the system on its ability to complete these tasks without any help, and then again with a demonstration. The data showed that the agent consistently improved its performance when given a visual guide, but it did not become dependent on it. It remained capable of working from instructions alone, proving that the demonstration acted as a helpful map rather than a rigid set of rules. This balance is crucial for practical deployment, as it means the system can be used in situations where no prior example exists, while still being able to learn quickly when a user provides one.

The implications of this work extend beyond just better test scores. By demonstrating that an agent can learn to navigate complex, multi-application workflows with high reliability, the researchers have taken a significant step toward making digital automation a practical reality for everyday office work. The system does not require a human to micromanage every click; instead, it can take a high-level goal and figure out the necessary steps, correcting its own course when it encounters unexpected screen layouts or missing information. The ability to learn from a single demonstration suggests that these agents could eventually be taught new, specific company procedures in minutes rather than months.

The study also highlights the importance of how we evaluate these systems. The researchers found that simply counting whether a task was completed or not often hides the fact that an agent might have made significant progress before failing at the very last step. By measuring partial progress, they were able to see that the agent was often 80 percent of the way through a complex task, even if it didn't finish. This nuance is important for understanding where the technology stands today and where it needs to improve. The team identified that the most common remaining failures were not in the early stages of a task, but in the final steps, such as forgetting to send a confirmation email or missing a specific field in a form.

In the end, the work presents a clear path forward for building more capable computer agents. It shows that the combination of rigorous, simulated training and the ability to learn from visual examples creates a system that is both powerful and adaptable. The researchers have made their training data and benchmarks available to the community, allowing others to build upon these findings. As these systems continue to evolve, they move closer to a future where artificial intelligence can handle the routine, repetitive, and complex digital tasks that currently consume so much of the human workday, leaving people free to focus on the decisions that truly require human judgment.