OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations
This paper introduces OmegaUse-SOP, a human-in-the-loop system that transforms expert demonstrations into reusable, domain-specific Standard Operating Procedure (SOP) skills for GUI agents, significantly improving their reliability in complex professional environments like photovoltaic simulation workflows.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a computer program that can look at a screen, understand what it sees, and click buttons or type text just like a human would. This is the promise of the "GUI agent," a type of artificial intelligence designed to navigate the digital world not by reading code, but by watching and interacting with the graphical interfaces we use every day. While these agents have become quite good at simple tasks, they often stumble when faced with complex, professional jobs. Real-world work is rarely a straight line; it involves hidden rules, specific software habits, and steps that must be checked and rechecked. When an agent tries to perform a professional task without understanding these nuances, it often gets lost, misses a crucial detail, or performs the wrong action. The challenge, then, is not just making the agent smarter, but teaching it the specific, unwritten procedures that experts use to get the job done correctly.
A researcher from Baidu and a power engineering company in China has developed a new system to solve this problem, called OmegaUse-SOP. The core idea is to treat the way humans perform complex software tasks as a set of instructions that can be captured, refined, and reused. They call this process "SOP Engineering," which stands for Standard Operating Procedure Engineering. Think of it as taking a master craftsman's workflow and turning it into a reliable guide that a machine can follow. Instead of simply recording a video of someone clicking buttons, the system watches the human expert, understands what they are doing, and then rewrites those actions into clear, logical steps that a computer can understand and adapt to new situations.
The system works through four distinct stages that happen in a loop. First, the "Observe" module watches a human expert perform a task on a computer. It records every mouse click, keyboard press, and screen change, saving a picture of the screen right before each action. This creates a detailed log of exactly what happened and what the screen looked like at that moment. Next, the "Reason" module takes this raw log and translates it into plain language. It looks at a click on a specific spot and figures out that the human was actually selecting a "project name" field, not just clicking at a random coordinate. It turns a list of technical events into a story: "Click the project name box, then type the name." This step is crucial because it helps the computer understand the meaning of the action, not just the movement.
Once the steps are understood, the "Configure" stage allows a human user to add specific rules and variables. In professional work, the same task might need to be done with different numbers or settings each time. This module lets the user mark certain parts of the instructions as changeable, so the agent knows which values to swap out for a new job. Finally, the "Execute" module puts the plan into action. The agent looks at the current screen, finds the right step in its instructions, and performs the action. After every step, it checks to see if the screen changed the way it was supposed to. If something looks wrong, it can stop and ask a human for help, or try again. This cycle of watching, understanding, adjusting, and acting turns a single human demonstration into a reusable skill that can be applied to many different tasks.
To test if this approach actually works, the researcher partnered with a client in the power industry to tackle a very specific challenge: designing solar power systems using a complex software package called PVsyst. They chose five difficult tasks that experts perform regularly, such as importing weather data, setting the angle of solar panels, and calculating energy losses. They tested three different artificial intelligence models on these tasks. First, they let the models try to do the jobs just by listening to a simple instruction, without any help from the new system. The results were mixed; the models struggled significantly, completing only one to three out of the five tasks depending on the model used. They often got confused by the software's specific layout or missed subtle requirements.
Then, the researcher applied the OmegaUse-SOP system. They recorded human experts performing the same five tasks, used the system to turn those recordings into refined, step-by-step skills, and let the models try again. The difference was immediate and total. With the help of the engineered SOPs, every single model, including the open-source ones, completed all five tasks successfully. The system didn't just make the agents slightly better; it turned them from unreliable novices into competent operators. The researcher also ran a specific test to see which part of their system was most important. They found that the "Reason" module, the part that translates raw clicks into meaningful instructions, was the key. When they removed this step and let the agents try to follow the raw, unexplained logs, the success rate dropped dramatically. This proved that simply recording what a human does is not enough; the computer must understand why they are doing it.
The findings suggest that for artificial intelligence to truly master professional software, it needs more than just powerful vision or memory. It needs a way to capture the implicit knowledge that human experts hold—the habits, the checks, and the specific ways they navigate complex menus. By building a system that can turn human demonstrations into structured, editable, and verified skills, the researcher has shown a practical path forward. The OmegaUse-SOP system does not replace human judgment; instead, it creates a bridge between human expertise and machine execution, allowing computers to reliably handle the intricate, rule-bound workflows that define professional work in fields like engineering and design.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.