Goal Completion and State Preservation: Case Studies of GUI-Derived Agent Tools
This paper demonstrates that reusing recorded GUI operations as parameterized tools significantly improves task completion rates and reduces model calls for software agents compared to raw code, while also highlighting the critical need to evaluate persistent application state and collateral changes to ensure reliable workflow execution.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital world, software agents are programs designed to perform tasks on our behalf, acting much like a digital employee that can click buttons, fill out forms, and move data between different applications. To teach these agents how to do complex jobs, researchers often record a human performing the task on a screen, capturing every mouse click and keystroke. This recording becomes a map, a set of instructions that the agent can follow to repeat the work. However, a critical question remains: does following a recorded map guarantee that the job is truly finished correctly? In software engineering, "completing" a task often means more than just reaching the final screen; it requires that the underlying system remembers the change accurately. If an agent updates a price but fails to update the hidden calculations that determine discounts, the task is technically done but functionally broken. This distinction between finishing the visible steps and preserving the invisible state of the system is the central puzzle this research addresses.
A researcher named Dekuan Li investigated this problem by testing two different ways to guide software agents through complex business tasks. The study focused on a specific family of software services used for managing customer relationships, billing, and event tickets. The researcher compared a "raw code" approach, where an agent tries to figure out the steps from scratch using general programming knowledge, against a "GUI-derived workflow" approach. In the latter method, the agent was given a frozen, pre-recorded map of the exact steps a human took on a graphical interface, which the agent then simply filled in with new details like names and dates. The goal was to see which method was better at completing the entire request and, perhaps more importantly, which method left the software system in a healthy, consistent state afterward.
The results of the experiments were striking. When the agents were given three different complex business requests to solve, the ones using the pre-recorded workflow succeeded in hitting every single core requirement on all three attempts. In contrast, the agents trying to write their own code from scratch managed to complete only a fraction of the necessary steps, scoring as low as eight out of thirty-three points on the hardest tasks. The agents using the recorded map also required far fewer attempts to finish the job. While the raw code agents hit token limits or stopped due to invalid commands, the workflow agents finished the same tasks in just five or six attempts. This suggests that providing an agent with a concrete, recorded path of human actions is significantly more reliable than asking it to invent the path on its own, at least for these specific types of business operations.
However, the study revealed a deeper layer of complexity that goes beyond simple success or failure. Even when the agents successfully completed the visible parts of the task, the way they did it mattered for the long-term health of the system. In a separate test involving an online store platform, researchers found that different methods of saving a price change produced different hidden side effects. One method, which was similar to the raw code approach, updated the price but left a cached version of a discount calculation stuck at an old value. This meant that while the price looked correct on the screen, the system's internal math was wrong. Another method, which used a more thorough, repository-based approach, correctly updated both the price and the hidden discount calculations. This finding highlights that an agent can "finish" a task by reaching the right number, yet still leave the system in a broken state if it doesn't trigger the right internal updates.
The research also uncovered a subtle issue with how information is inherited in these systems. In one experiment, researchers tested whether updating a default description for a product would automatically update that description for specific variations of the product. They found that in some cases, if an agent had previously made a specific override for a product, that old override would block the new update from taking effect. The agents using the repository-based tools were able to navigate this correctly, propagating the description normally, whereas the raw code and GUI-hint conditions showed blocked propagation. This demonstrates that the path an agent takes to reach a goal can determine whether the system remains flexible for future changes or becomes rigid and prone to errors.
Ultimately, the study concludes that while recording human actions and turning them into tools for agents is a powerful way to improve task completion rates, it is not a perfect solution. The workflow tools were far superior to raw code in getting the job done, but they still left some gaps in the final output, such as failing to meet exact event cardinality checks or issues with the complete task-body string. More importantly, the research shows that evaluating an agent's performance cannot stop at checking if the final number is correct. One must also examine the "collateral" changes—the side effects left behind in the system's memory. A truly successful agent must not only reach the destination but also ensure that the road it traveled remains clear and consistent for the next traveler. This work suggests that the future of reliable software agents lies in combining the efficiency of recorded workflows with rigorous checks on the state of the systems they modify.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.