SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents
The paper introduces SeekJudge, a practical model-based reward framework utilizing four role-specialized agents and a distilled 9B backbone that matches or surpasses rule-based supervision in training computer-use agents while offering step-level judgments, lower costs, and scalability for long-horizon tasks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a robot to navigate a video game world, but instead of pressing buttons on a controller, the robot has to click, type, and scroll on a real computer screen just like a human would. This is the world of "computer-use agents." For a long time, scientists have tried to teach these robots by showing them a goal—like "book a hotel"—and then checking if they succeeded. The old way of checking was like using a rigid checklist: if the robot clicked the right button at the right time, it got a point; if it clicked the wrong one, it got zero. But computers are messy and change all the time. A checklist might say "you failed" because the robot clicked a button that looked slightly different than the one on the checklist, even though the robot actually finished the job perfectly. It's like a teacher failing a student for writing the word "color" instead of "colour," even though the essay was brilliant.
To fix this, researchers started using AI models to act as judges, hoping they could understand the spirit of the task rather than just the strict rules. However, these AI judges often get confused when they have to look at a whole movie of the robot's actions all at once. It's like trying to find a specific typo in a 500-page novel by reading the entire book in one second; the important details get lost in the noise. The big question became: How do we build a judge that is smart enough to understand human goals, cheap enough to run thousands of times while the robot learns, and fast enough to handle long, complicated stories without getting a headache?
Enter SeekJudge, a new framework that acts like a team of four specialized detectives working together to solve a mystery, rather than one overworked detective trying to do everything at once. The researchers found that when you force an AI to look at too many screenshots of a computer screen at the same time, it starts to hallucinate or miss the most important clues. It's as if you asked someone to identify a suspect in a crowd of 100 people; they might get distracted by a stranger's bright hat and miss the actual criminal. But if you showed them just the one photo of the suspect, they would spot them instantly.
SeekJudge solves this by splitting the job. First, two "scouts" (called the Condense and Ground agents) quickly skim through the robot's long journey and write a short, easy-to-read summary of what happened, step by step. Then, a "detective" (the Seek agent) reads this summary and decides, "Hmm, I need to see the photo from step 42 to be sure." It then calls a fourth agent, the Analyze agent, to look only at that single, specific screenshot and answer a precise question about it. This back-and-forth loop continues until the detective is confident enough to give a final verdict.
The paper shows that this "seek and analyze" approach is a game-changer. When they tested it, SeekJudge was the first AI-based judge to perform as well as, or even better than, the old rigid rule-checkers at helping robots learn new tasks. In fact, when used to train a robot, the robots learned faster and succeeded more often with SeekJudge than with the traditional methods.
What's really cool is how efficient this is. Because the AI only looks at one image at a time instead of a whole stack, it doesn't need a super-expensive, massive computer to run. The researchers found that their system costs a tiny fraction of what other AI judges cost—hundreds of times cheaper than using a closed-source giant model. It's like the difference between hiring a team of local experts who each look at one clue for a few dollars, versus hiring one celebrity detective who charges a fortune to read the entire case file at once.
The team also built a new "training ground" called CUAStepBench, which is a collection of 278 different computer tasks where humans have carefully labeled every single step of the robot's journey. This helped them train their "detective" team to be incredibly sharp. They proved that by breaking the problem down into finding the right moment (localization) and then reading the details (extraction), they could handle much longer and more complex tasks than previous methods.
In short, SeekJudge doesn't try to be a super-intelligent all-seeing eye; instead, it acts like a smart, efficient team that knows exactly when to zoom in on the most important clue. This makes it possible to train computer-using robots in the real world, where tasks are long, screens change, and strict rulebooks just don't work. The paper suggests that this approach could be the key to making AI assistants that can actually help us with our daily computer chores, from booking flights to organizing files, without needing a supercomputer to grade their homework.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.