💻 computer science

Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs

This paper presents a systematic empirical study demonstrating that while inference-time scaling techniques like contextual and temporal expansion can stabilize local computer-use agents, they often yield diminishing returns and shift failure modes toward premature successes, indicating that efficient local deployment requires selective compute allocation and failure-aware control mechanisms rather than indiscriminate scaling.

Original authors: Woongkyu Lee, Jungwook Choi

Published 2026-07-31

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Woongkyu Lee, Jungwook Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who lives entirely inside your computer. This robot can look at your screen, read what's on it, and even click buttons or type words to help you finish tasks, like organizing your files or booking a flight. We call these "Computer-Use Agents." For a long time, scientists thought the best way to make these robots smarter was just to give them more brainpower while they were working. It's like thinking, "If I get stuck on a puzzle, I'll just stare at it longer or try to remember every single thing I've ever seen to solve it." This idea is called "inference-time scaling"—basically, spending more time and computer energy while the robot is trying to do a job to see if it gets better at the job.

But here's the catch: most of these super-smart robots live on giant, expensive servers in the cloud. What if we want to put a robot like this on your own laptop or phone, where the battery is small and the processor isn't a giant supercomputer? That's where "local" agents come in. The big question is: if you have a smaller, cheaper robot running on your own device, does giving it more time to think or more history to remember actually help it finish the job? Or does it just make the robot spin its wheels, get confused, or waste your battery? This paper dives into that exact mystery, testing whether "thinking harder" actually helps small, local robots, or if it just leads to new kinds of mistakes.

The Great Local Robot Experiment: Does More Thinking Mean More Success?

The authors of this paper decided to put this idea to the test. They set up a series of experiments with three different "local" computer agents (robots that run on your own hardware) and asked them to solve real-world tasks on a computer screen. They wanted to see what happened when they tweaked four specific "knobs" to give the robots more resources:

Contextual Scaling (The Memory Knob): How many past screenshots does the robot remember?
Temporal Scaling (The Time Knob): How many steps (clicks or types) is the robot allowed to take before it has to stop?
Structural Scaling (The Teamwork Knob): Should the robot do everything itself, or should it split the job into two parts: one part that plans the steps and another part that actually clicks the buttons?
Parallel Scaling (The Crowd-Sourcing Knob): Should the robot try to come up with several different plans at once and pick the best one?

The results were surprising and a little bit funny. It turns out that for these local robots, just "trying harder" doesn't always mean "doing better." In fact, sometimes it makes things worse.

The Memory Trap: Remembering Too Much

First, they tested the Memory Knob. They found that a robot with no memory (it only sees the current screen) is a disaster. It gets stuck in loops, like a hamster running on a wheel, clicking the same button over and over because it doesn't remember what it just did. Giving it just a tiny bit of history (one previous screen) was a huge game-changer. It stabilized the robot.

However, the authors discovered a "sweet spot." When they gave the robot too much history (remembering 8 past screens instead of 4), the robot didn't get smarter; it just got more expensive to run. The extra memory didn't help it solve the task better. Instead, it started making a new kind of mistake: premature false successes. Imagine a student taking a test who, instead of finishing the last question, just guesses "I'm done!" and hands in the paper early because they are tired of reading. The robot, overwhelmed by too much history, would sometimes think it had finished the task when it actually hadn't. So, the paper suggests that for local robots, a moderate amount of memory is best—enough to avoid loops, but not so much that it confuses the robot into giving up early.

The Time Trap: More Steps, Same Problems

Next, they turned the Time Knob. They let the robots take more steps to finish a task, thinking, "If it's stuck, just let it try more times!" The result? The robots didn't get much better at finishing the tasks. They just took longer.

The authors found that giving a robot more time didn't fix its bad logic; it just let it make more of the same bad moves. If a robot was going down the wrong path, giving it 100 steps instead of 15 just meant it walked further down the wrong path before realizing it was lost. The main benefit was that it stopped the robot from hitting a "time limit" wall, but it didn't stop the robot from making mistakes. In fact, it often led to those same "premature false successes" where the robot thought it was done when it wasn't. The paper suggests that for local models, just giving them more time isn't a magic fix; it's mostly just burning more battery.

The Teamwork Trap: Too Many Cooks

Then they tried Structural Scaling. They split the robot into two: a "Planner" that thinks about what to do, and a "Doer" that actually clicks the buttons. They hoped this would be like having a general and a soldier, where the general makes a great plan. But on local computers, this backfired.

The "Planner" part often wrote plans that were messy or incomplete, and the "Doer" part couldn't understand them. It was like a general shouting orders in a language the soldier didn't speak. This added a lot of extra work (computational cost) and actually made the robots worse at finishing tasks than the single robot that did everything itself. The only time this helped was if they used Parallel Scaling—asking the Planner to write multiple different plans at once and picking the best one. This did help a little bit, but it cost a huge amount of computer power to generate all those extra plans. The paper suggests that for local robots, keeping things simple (one robot doing both jobs) is usually better than trying to split the work, unless you have a lot of extra power to burn.

The Big Takeaway: Quality Over Quantity

So, what's the final verdict from this study? The authors suggest that for local computer agents, the old idea of "more compute is always better" is a trap.

Instead of just throwing more memory, time, or complex team structures at the problem, we need to be smarter about how we use what we have.

Don't overfeed the memory: A little history is great, but too much just confuses the robot.
Don't just wait longer: Giving a confused robot more time usually just means it gets confused for longer.
Keep it simple: Splitting the job into planning and doing often creates more headaches than it solves on local devices.

The paper concludes that the future of local computer agents isn't about making them "think harder" in a brute-force way. It's about designing them to be aware of their own limits, knowing when to stop, and using just the right amount of resources to stay on track without burning out. It's a reminder that sometimes, a small, focused robot is better than a giant, overworked one.

Technical Summary: Rethinking Inference-Time Scaling in Local Computer-Use Agents

Problem Statement

The deployment of autonomous Computer-Use Agents (CUAs) on local hardware is critical for privacy, cost efficiency, and security. However, improving the performance of these agents under strict hardware constraints remains challenging. While recent studies demonstrate that inference-time scaling (allocating additional computation during execution) significantly boosts the performance of frontier, proprietary agents, its efficacy for resource-constrained local models is poorly understood.

There is a fundamental tension between model capability and operational efficiency. State-of-the-art agents often incur high costs in step counts and token consumption compared to human efficiency. It remains unclear whether the marginal utility of adding computation (e.g., longer histories, more steps, or parallel planning) justifies the cost for local models, or if such scaling merely alters failure patterns without improving task success. This paper addresses the need for a systematic study of the marginal utility and failure behaviors of inference-time scaling in local settings.

Methodology

The authors conducted a systematic empirical study evaluating local CUAs on the OSWorld benchmark (361 real-world Ubuntu tasks). The study focused on four specific dimensions of inference-time scaling:

Contextual Scaling: Varying the length of the screenshot and action history ( $H \in \{0, 1, 4, 8\}$ ) provided to the agent.
Temporal Scaling: Varying the maximum decoding steps ( $S \in \{15, 50, 100\}$ ) allowed for task completion.
Structural Decomposition: Comparing Single-Agent architectures (one model handles observation and action) against Two-Stage Agents (decoupled Planning and Grounding stages).
Parallel Scaling: Varying the number of candidate plans ( $P$ ) generated in the planning stage of two-stage agents.

Models Evaluated:

Single-Agent: Qwen3-VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct (MoE), UI-TARS-1.5-7B, and OpenCUA-7B.
Two-Stage: Grounding fixed as GTA-1-7B; Planning/Judging performed by Qwen3-8B and Qwen3-30B-A3B.

Metrics:

Task Success Rate: The primary measure of accuracy.
Efficiency: Measured by average steps per task and cumulative prompt token usage.
Failure Modes: Categorized into repetitive loops, max-step stalls, premature false successes, and format-parsing errors.

Key Contributions and Results

1. Contextual Scaling: Stability vs. Saturation

Minimal History is Critical: Providing even a single previous screenshot ( $H=1$ ) yields a substantial performance leap (from ~18% to >25% success) by preventing repetitive loops and redundant actions.
Diminishing Returns: Expanding history to $H=4$ further improves step efficiency and trajectory stability. However, gains saturate beyond this point. Increasing history to $H=8$ adds significant token cost without improving accuracy.
Shift in Failure Modes: As history increases, failures shift from "max-step stalls" (running out of steps) to premature false successes (the agent terminates believing the task is done when it is not). This suggests that while context stabilizes the trajectory, it does not necessarily improve the agent's ability to judge task completion correctly.

2. Temporal Scaling: Ineffectiveness under Limited Reasoning

Linear Cost, Flat Accuracy: Increasing the maximum step budget ( $S$ ) from 15 to 100 yields little to no improvement in task success for local models.
Redistribution of Failures: Additional steps reduce max-step stalls but fail to correct erroneous trajectories. Instead, they allow agents to continue incorrect behaviors for longer, shifting failures toward premature false successes.
Conclusion: Temporal scaling is inefficient for local models; it increases operational costs linearly without raising the success ceiling.

3. Structural Decomposition: Overhead in Local Settings

Performance Drop: Decoupling agents into Planning and Grounding stages (Two-Stage) consistently underperforms Single-Agent baselines, even with additional planning steps.
New Failure Modes: Local models struggle with the structural requirements of two-stage agents. Common failures include:
- Planning Limitations: Generated plans are often incomplete or misaligned with visual states.
- Formatting Errors: Planners frequently omit required fields or generate actions as plain strings rather than valid code blocks, causing the grounding agent to fail parsing.
Cost: Two-stage agents consume substantially more tokens than single agents while achieving lower accuracy.

4. Parallel Scaling: Partial Recovery at High Cost

Mitigation: Increasing the number of parallel plans ( $P$ ) in two-stage agents partially recovers performance by providing more candidate plans and bypassing format errors (if at least one plan is valid).
Sub-linear Gains: The improvement in accuracy is sub-linear relative to the exponential increase in token consumption. While it reduces format-related failures, it does not fully close the gap with single-agent baselines.

Significance and Practical Implications

The paper argues that efficient local CUAs require selective compute allocation rather than uniform scaling. The findings challenge the assumption that "more compute" automatically leads to better performance in local settings.

Failure-Aware Control: Additional computation often changes the type of failure rather than resolving it. Future systems need control mechanisms that detect repetitive loops, stalled progress, or premature termination, rather than simply extending the horizon.
Structural Simplicity: Complex agentic frameworks designed for powerful proprietary models (e.g., hierarchical planning) may introduce prohibitive overhead for local models. Local agents should favor structurally simple designs or be specifically fine-tuned to handle decomposition.
Context Management: Instead of unbounded context expansion, local agents should employ selective memory retention to maintain task-relevant history while filtering noise, balancing stability with token costs.

In conclusion, the authors posit that the future of local CUAs depends less on increasing inference-time compute and more on designing agents that align with the specific capabilities and limitations of local models, utilizing failure-aware control and optimized resource allocation.