GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents
This paper introduces a controlled benchmark demonstrating that while GUI agents currently outperform CLI agents due to better interaction reliability, the CLI performance gap is primarily driven by incomplete skill coverage rather than inherent model limitations, as verifier-guided skill augmentation significantly boosts CLI success rates.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very smart robot assistant that needs to do chores on your computer. You can give this robot two different ways to do the job:
- The "Human" Way (GUI): The robot looks at your computer screen like a human does. It sees icons, buttons, and menus. It uses a mouse to click, drag, and type, just like you would.
- The "Code" Way (CLI): The robot doesn't look at the screen. Instead, it talks to the computer through a special list of pre-programmed commands (skills). It says, "Add a track," and the computer does it instantly, without the robot ever seeing the "Add Track" button.
This paper is a big experiment to figure out which way is better. But here's the catch: in the past, people compared these two methods unfairly. They often gave the "Code" robot easier tasks or different goals than the "Human" robot. It was like comparing a race car to a bicycle, but then giving the bicycle a downhill track and the race car a mountain.
The Big Experiment: A Fair Race
The researchers built a fair playing field. They created 440 different computer tasks (like editing a video, organizing music, or making a spreadsheet).
- Same Goal: Both robots got the exact same instruction (e.g., "Rename these three songs").
- Same Starting Point: Both robots started with the computer in the exact same state.
- Same Finish Line: A computer program checked the results to see if the job was done right.
- Different Tools: The only difference was how they were allowed to do the work. One had to click buttons; the other had to use the command list.
The Results: Who Won?
Round 1: The Original Race
- The "Human" Robot (GUI): It won with a 59.1% success rate. It was pretty good at following the visual clues on the screen.
- The "Code" Robot (CLI): It lost with a 48.2% success rate. It struggled more.
Why did the Code robot lose?
The researchers dug into the failures and found a surprising reason. The Code robot wasn't necessarily "dumber." It just had a broken instruction manual.
- Imagine the Code robot has a list of 100 skills, but the task required "Skill #42," which didn't exist in its list. The robot couldn't do the job, not because it was confused, but because the tool was missing.
- Only about 37% of the tasks could be done with the original list of skills the Code robot had.
Round 2: The "Fixed Manual" Race
The researchers then fixed the Code robot's manual. They added the missing skills that the robot needed to pass the tests.
- New Score: The Code robot's success rate jumped to 69.3%.
- The Takeaway: Once the Code robot had the right tools, it actually became better than the Human robot! This proves that the Code robot's original failure wasn't because it couldn't think; it was because it didn't have the right buttons to push.
Where Each Robot Shines (and Stumbled)
The paper found that each robot has a different "superpower" and a different "weakness":
The Human Robot (GUI):
- Superpower: Great at tasks where the steps are obvious on the screen, like navigating a website or organizing a video timeline.
- Weakness: It gets lost easily. If a menu is hidden, or if it has to click 20 times in a row, it often forgets what it was doing or clicks the wrong thing. It's like trying to navigate a maze while blindfolded, except you can see the walls but you keep tripping over them.
The Code Robot (CLI):
- Superpower: Great at tasks that are like building with Lego blocks, where the structure is clear (like organizing files or 3D modeling). If it has the right command, it's fast and precise.
- Weakness: It's terrible at guessing. If the computer has a "default" setting that a human would just click without thinking, the Code robot has to be told exactly what that default is. If the manual doesn't say "The default name is 'Track 1'," the robot might name it "Track 2" and fail. It's like a chef who can cook a perfect steak but doesn't know you usually salt it before cooking unless you write it down.
The Bottom Line
The paper concludes that comparing these two methods isn't about saying one is "better" than the other. It's about where the logic lives.
- In the Human (GUI) method, the logic is built into the visible interface. The computer shows you the steps, but you have to physically find and click them.
- In the Code (CLI) method, the logic is built into a hidden layer of skills. The computer does the heavy lifting, but only if someone has written down every single step in the manual.
The Lesson: If you want a robot to do computer tasks, you have to decide: Do you want it to "see" and "click" (which is hard to get right over long tasks), or do you want it to "command" (which is fast, but only works if you've built a perfect, complete library of commands for it)? The best system might need a mix of both.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.