💻 computer science

AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions

AutoRPA is a framework that bridges the gap between inefficient LLM-based ReAct agents and traditional RPA by automatically distilling interaction trajectories into robust, reusable RPA functions, thereby achieving significant reductions in token usage and runtime costs while maintaining task-solving capabilities.

Original authors: Minghao Chen, Xinyi Hu, Zhou Yu, Yufei Yin

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minghao Chen, Xinyi Hu, Zhou Yu, Yufei Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly expensive, personal assistant named "LLM" (Large Language Model). This assistant is brilliant at figuring out how to use a computer or phone screen for the first time. If you ask them to "book a flight" or "delete a note," they can look at the screen, think about what to do, click the right buttons, and get the job done.

However, there's a catch: They are slow and expensive to use for repetitive tasks.

Every time you ask them to book a flight, they have to "think" from scratch. They read the screen, plan the steps, and click. If you need to do this 100 times a day, you are paying for 100 hours of "thinking," which is a waste of money and time.

On the other hand, before these smart assistants existed, people used "Robotic Process Automation" (RPA). Think of RPA as a pre-recorded script. It's like a robot that blindly clicks "Button 3, then Type 'Hello', then Click 'Save'." It's incredibly fast and cheap to run. But it's also fragile. If the designer of the app moves "Button 3" to a different spot, the robot crashes because it doesn't know how to adapt. It needs a human to manually rewrite the script every time the screen changes.

The Problem

We want the speed and cheapness of the robot (RPA) but the smart adaptability of the assistant (LLM). We want a solution that works for repetitive tasks (like booking flights every day) without needing a human to rewrite the code every time the app updates.

The Solution: AutoRPA

The paper introduces AutoRPA, a system that acts like a master chef who turns a messy cooking demonstration into a perfect, reusable recipe.

Here is how it works, step-by-step:

1. The "Exploration" (The Chef Watching)

First, AutoRPA uses the smart LLM assistant to perform the task (e.g., booking a flight) just once or a few times. The assistant looks at the screen, thinks, and clicks. This is the "ReAct" phase.

Analogy: The chef watches a sous-chef cook a dish for the first time, noting every move.

2. The "Translation" (Turning Notes into a Recipe)

The assistant's actions are usually "hard-coded" (e.g., "Click the button at pixel coordinates 144, 278"). This is bad because if the screen shifts, the coordinates are wrong.
AutoRPA has a special Translator Agent that rewrites these actions. Instead of saying "Click at 144, 278," it says "Click the button that says 'Book Flight'."

Analogy: The chef takes the sous-chef's messy notes ("Hit the red dot at the top right") and rewrites them as a clear instruction ("Press the red 'Start' button"). This makes the recipe work even if the kitchen layout changes slightly.

3. The "Building" (Creating the Master Script)

A Builder Agent takes these translated, flexible instructions and combines them into a single, robust computer program (an RPA function). It looks at many different attempts (trajectories) to figure out the best way to handle variations.

Analogy: The chef writes the final, professional recipe card that can be used by anyone, anywhere, to make that dish perfectly.

4. The "Hybrid Repair" (The Safety Net)

Sometimes, the new recipe might have a bug. If the robot tries to run the code and fails, AutoRPA doesn't just give up. It has a Hybrid Repair Strategy:

It pauses the robot.
It calls the smart LLM assistant back in to figure out why it failed and how to fix it from that exact point.
It uses the assistant's fix to update the recipe.

Analogy: If the robot burns the toast, the chef steps in, fixes the toast, and then updates the recipe card so the robot doesn't burn it again next time.

The Results

The paper tested this on three different environments (Android apps, websites, and simulated web tasks).

Efficiency: The new AutoRPA code is 82% to 96% cheaper to run than asking the smart assistant to do the task every time. It saves a massive amount of "token" usage (the currency of AI thinking).
Success Rate: It solves tasks just as well as, or sometimes better than, the smart assistant alone, because the generated code is stable and doesn't get confused by minor screen changes.
Reusability: Once the code is built for a "type" of task (like "booking a flight"), it can be reused for hundreds of specific instances (booking to New York, booking to London) without needing to think again.

Summary

AutoRPA is a system that watches a smart AI learn a task, translates that learning into a flexible, reusable script, and then polishes that script until it's perfect. It gives us the best of both worlds: the intelligence to handle new situations and the efficiency of a robot to handle repetitive work.

Technical Summary: AutoRPA

Problem Statement

While Large Language Model (LLM) agents have demonstrated proficiency in interacting with Graphical User Interfaces (GUIs) via the ReAct (Reasoning and Acting) paradigm, they face significant inefficiencies in practical deployment. Most existing research optimizes for single-task performance on novel instances, often invoking expensive LLM reasoning for every interaction. However, real-world scenarios frequently involve repetitive task types (e.g., daily report filing, flight booking) where invoking LLM reasoning for each instance is computationally wasteful and costly.

Conversely, traditional Robotic Process Automation (RPA) offers high runtime efficiency for repetitive tasks but suffers from brittleness. Traditional RPA relies on manually crafted scripts that are rigid; they fail when GUI layouts change and require significant human effort to develop and maintain. The core challenge addressed by this paper is how to automatically synthesize robust, token-efficient RPA functions for specific task types using LLM assistance, bridging the gap between the flexibility of LLM agents and the efficiency of traditional RPA.

Methodology: The AutoRPA Framework

AutoRPA is a framework designed to "distill" the decision logic of ReAct-style agents into executable, reusable RPA code. The process is divided into three main phases:

1. Exploration Phase

ReAct Trajectory Generation: A ReAct-style agent interacts with the GUI environment to solve a task instance. It observes screenshots and UI structures (DOM/accessibility trees), reasons through steps, and executes actions.
Reflection: If a trajectory fails, a reflection mechanism allows the agent to analyze the failure and retry (up to $N_{ref}$ times) to ensure successful trajectories are collected.
Translation (Hard-to-Soft Coding): A dedicated Translator Agent converts the agent's "hard-coded" actions (e.g., click(index=2)) into "soft-coded" procedures. Instead of relying on fixed indices or coordinates, the translator generates code that locates elements based on semantic attributes (e.g., text content, element type, target descriptions) using a find_element function. This ensures the resulting logic is robust to minor layout changes.

2. RPA Generation Phase

Builder Agent with RAG: A Builder Agent synthesizes the final RPA function ( $F_k$ ) for a specific task type ( $G_k$ ).
Retrieval-Augmented Generation (RAG): To prevent the builder from making erroneous assumptions about the interface state due to missing context, the framework employs a tree-organized retrieval mechanism. The builder can query a trajectory database to fetch specific interaction blocks (observations, actions, summaries) from past successful explorations.
Code Synthesis: The builder generates a Python function that includes logical structures (loops, conditionals) to handle task variations. It prunes redundant steps from the translated trajectory and incorporates necessary error handling. The code may also include calls to Multi-Modal LLMs (ask_mllm) for complex reasoning tasks that cannot be resolved via static rules.

3. Verification Phase with Hybrid Repair

Execution and Analysis: The generated RPA code is executed on seen task instances. If execution fails, an Analyzer Agent identifies the breakpoint, analyzes the failure cause, and determines if the task can be resumed or must be restarted.
Hybrid Repair Strategy: Rather than relying solely on code refinement, the system employs a hybrid approach. If the code fails, the ReAct agent resumes execution from the breakpoint (or restarts) to produce a corrective demonstration. This "hybrid trajectory" (code execution + ReAct repair) is fed back to the Builder Agent to refine the code iteratively.
Termination: The process continues until the code passes all validation tasks or a maximum number of refinement attempts ( $M$ ) is reached. If the task remains unautomatable, the system defaults to using the ReAct agent for that specific instance.

Key Contributions

AutoRPA Framework: A novel system that automatically generates robust RPA functions for task types, targeting repetitive GUI automation scenarios where traditional RPA is too brittle and ReAct agents are too costly.
Translator-Builder Pipeline: A two-stage process where a translator converts hard-coded ReAct actions into soft-coded, attribute-based procedures, and a builder synthesizes robust RPA functions using tree-structured trajectory retrieval (RAG).
Hybrid Repair Strategy: A verification mechanism that combines direct code execution with ReAct-based fallback. This allows the system to iteratively refine code by incorporating corrective demonstrations from the ReAct agent when the synthesized code fails.
Efficiency and Robustness: The framework successfully reduces token consumption significantly while maintaining or improving success rates compared to pure LLM-based agents.

Experimental Results

The authors evaluated AutoRPA on three benchmarks: AndroidWorld (real-world Android apps), WebArena (realistic web tasks), and MiniWoB++ (simulated web tasks).

Token Efficiency: AutoRPA reduced token usage by 82% to 96% compared to ReAct-style agents. For instance, on AndroidWorld with GPT-4o, AutoRPA consumed only 14.7k tokens versus 79.9k for ReAct.
Success Rates:
- On AndroidWorld, AutoRPA achieved a 37.0% success rate (GPT-4o), outperforming ReAct (35.2%) and SeeAct (15.3%), while using significantly fewer tokens.
- On MiniWoB++, AutoRPA achieved 91.1% success on "hard" tasks and 95.4% on all tasks, outperforming state-of-the-art methods like AdaPlanner and AutoManual, while consuming less than 10% of the tokens required by ReAct.
- On WebArena, AutoRPA matched the performance of prior methods but with a drastic reduction in token consumption.
Ablation Studies: Experiments confirmed that removing the ReAct exploration phase, the translator agent, the RAG mechanism, or the hybrid repair strategy all led to significant drops in success rates, validating the necessity of each component.

Significance and Claims

The paper claims that AutoRPA addresses a critical practical problem: the high cost and inefficiency of running LLM agents on repetitive tasks versus the high maintenance cost of traditional RPA. By distilling the decision logic of LLM agents into verified, reusable code, AutoRPA enables:

Token-Efficient Automation: Drastically reducing the computational cost of GUI automation for recurring task types.
Robustness: Generating code that handles variations in environment and task instructions better than static scripts or single-shot LLM generation.
Scalability: Providing a pathway to automate diverse GUI tasks without requiring extensive manual scripting or curated expert knowledge for every new task type.

The authors emphasize that their method does not rely on specific ReAct agents and can be adapted to various LLM backbones, making it a generalizable solution for efficient GUI automation.