💻 computer science

Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents

This paper reveals that while full-image visual memory reduces state-level failures in GUI agents, it exacerbates action-level errors, prompting the proposal of Action-Grounded Visual Memory (AGMem), which stores localized image crops related to successful actions to significantly improve task success rates.

Original authors: Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kunwoong Kim, Woomin Song, Chanseok Jeon, Jinwoo Shin

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kunwoong Kim, Woomin Song, Chanseok Jeon, Jinwoo Shin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Overwhelmed Intern

Imagine you hire a super-smart intern (the GUI Agent) to do complex tasks on your computer, like editing a presentation or organizing files. You give them a list of instructions, and they look at the screen to figure out what to do next.

Recently, researchers tried to help this intern by giving them a photo album of everything they've done in the past. The idea was: "If you get stuck, look at your photo album to remember how you solved similar problems before."

The researchers thought this would make the intern perfect. Instead, they found that giving the intern a giant, unedited photo album actually made them mess up more often.

The Problem: The "Cluttered Desk" Effect

The paper discovered that when you show the intern a full screenshot of the entire screen from a past task, it creates two specific problems:

The Good News: It helps the intern understand the big picture. They stop getting confused about which page they are on or what the overall goal is.
The Bad News: The full screenshot is too cluttered. It's like handing someone a map of the entire world when they only need to find a coffee shop three blocks away. The important details (like a tiny button or a hidden menu) get lost in the noise.

Because of this clutter, the intern starts making two new types of mistakes:

Hidden Operation Blindness: They miss the "secret" steps. For example, to save a file, you often have to click a tiny "three dots" menu that isn't visible until you click something else. The full-screen photo is so busy that the intern ignores the tiny menu and tries to guess.
Grounding Errors: The intern knows what to click (e.g., "Click the Save button"), but because the photo is so crowded, they click the wrong spot, just a few pixels off.

The Four Ways Agents Fail

The authors categorized the mistakes the intern makes into four buckets, like a mechanic diagnosing a car:

Cognitive Failure (The Wrong Plan): The intern misunderstands the goal entirely. Example: "I need to save the file," but they decide to print it instead.
Visual State Misunderstanding (The Wrong Reading): The intern looks at the screen and misreads what is happening. Example: They think a pop-up window is closed when it's actually open.
Hidden Operation Blindness (The Invisible Step): The correct action requires doing something not immediately visible, like opening a hidden menu. Example: They don't know they need to press a keyboard shortcut to reveal a hidden toolbar.
Grounding Error (The Missed Target): The intern knows exactly what to do, but their hand (the mouse click) is clumsy. Example: They aim for the "Submit" button but click the "Cancel" button right next to it.

The Solution: AGMem (The "Highlighter" Approach)

The researchers realized that the problem wasn't the memory itself, but how the memory was stored. Storing the whole screen was like storing a whole library book just to remember one specific sentence.

They proposed a new system called AGMem (Action-Grounded Visual Memory).

The Analogy:
Instead of giving the intern a full photo of the past screen, AGMem acts like a smart highlighter.

It looks at what the intern did in the past.
It cuts out (crops) only the tiny part of the screen where the action happened.
It throws away the rest of the clutter (the background, the other windows, the irrelevant icons).

How it works:
If the task was "Click the 'Save' button," AGMem doesn't show the whole desktop. It shows a zoomed-in, cropped image of just that button and the area around it.

The Results: A Sharper Focus

When they tested this new "cropped memory" system:

It fixed the "clutter" problem. The intern could finally see the tiny hidden menus and click the right buttons.
It kept the "big picture" benefits because the intern could still see the sequence of cropped steps.
The Outcome: The system improved the success rate of these computer agents by 33% compared to using full-screen photos.

Summary

The paper teaches us that more information isn't always better. For computer agents trying to control screens, dumping a full screenshot into their memory is like shouting instructions over a loud crowd. Instead, giving them a focused, zoomed-in view of exactly what they need to do (Action-Grounded Visual Memory) helps them ignore the noise and get the job done right.

Technical Summary: Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents

Problem Statement

Graphical User Interface (GUI) agents, powered by Vision-Language Models (VLMs), are increasingly tasked with automating complex computer operations. While recent advancements have introduced experiential memory—retrieving past trajectories to guide decision-making—and extended this to visual memory (storing screenshots), the efficacy of these mechanisms remains poorly understood. Specifically, it is unclear which specific failure modes visual memory mitigates and which it might exacerbate.

The authors observe that naive approaches, which prepend full-screen screenshots of past interactions to the agent's context, introduce a "divergent effect." While these full-image memories provide global context that helps with state-level understanding, they simultaneously crowd the visual field with task-irrelevant cues. This noise distracts the agent from subtle operations and degrades the precision of coordinate grounding, leading to a trade-off where improvements in one area cause regressions in another.

Methodology

1. Failure Mode Taxonomy

To systematically analyze agent performance, the authors introduce a four-class taxonomy of GUI agent failures, mapping to distinct stages of the perception-reasoning-action pipeline:

Cognitive Failure: Misunderstanding the task goal, subtask, required next step, or success condition (e.g., following a wrong plan or terminating early).
Visual State Misunderstanding: Misreading the visible screen state (e.g., believing a dialog is open when it is not, or misidentifying selected objects).
Hidden Operation Blindness: Failing to execute an action that requires a non-visible operation (e.g., opening a context menu, using a keyboard shortcut, or navigating a hidden overflow menu).
Grounding Error: Intending the correct visible target but executing a low-level action (e.g., clicking coordinates) that misses the target.

The authors evaluated this taxonomy on three benchmarks: OSWorld (desktop tasks), WebForge (web tasks), and AgentNetBench (offline action prediction). They found that different benchmarks exhibit distinct dominant failure modes: OSWorld is dominated by Hidden Operation Blindness and Cognitive Failure; WebForge by Grounding Error; and AgentNetBench by Visual State Misunderstanding.

2. Analysis of Full-Image Memory

The study first quantified the impact of adding full-image memory (retrieving and prepending full screenshots of relevant history) to the agent context.

Result: Full-image memory reduced state-level failures (Cognitive Failure and Visual State Misunderstanding) by providing global context.
Side Effect: However, it significantly worsened action-level failures. Hidden Operation Blindness increased by 11.7 percentage points, and Grounding Error increased by 8.6 percentage points. The dense visual context of full screenshots obscured the specific, often small, UI elements required for precise actions.

3. Proposed Solution: Action-Grounded Visual Memory (AGMem)

Motivated by the finding that "which pixels are stored matters as much as whether memory is used," the authors propose AGMem, a framework designed to store and retrieve focused, action-relevant visual evidence rather than monolithic screenshots.

Core Components:

Action-Relevant View (Cropping): Instead of storing full screenshots, AGMem constructs memory entries as tightly cropped images ( $C_t$ ) that isolate the specific GUI region where an action $a_t$ takes effect. This is achieved by computing pixel-wise differences between consecutive screenshots ( $o_t$ and $o_{t+1}$ ), applying morphological operations to create a change mask, and bounding the result to the smallest UI container.
Two-Stage Retrieval:
1. Trajectory-Level: A retriever selects a small pool of memory trajectories whose post-hoc subtask labels match the current subtask list.
2. Step-Level: Within the selected pool, a step-level retriever matches the current state (self-reported subtask and previous view) against memory steps using CLIP encoders (combining text and image embeddings).
Recovery-Aware Memory: To address error propagation, AGMem maintains a separate "recovery memory." An LLM-based detector identifies when an agent enters an erroneous state. If an error is flagged, retrieval is redirected to this recovery memory, which contains examples of corrective behaviors (faulty sub-trajectories paired with the steps that restored validity), rather than just successful executions.

Key Results

Experiments conducted on OSWorld using GPT-5.4-mini demonstrate the effectiveness of AGMem:

Failure Reduction: AGMem is the only configuration that consistently reduces all four failure modes compared to the vanilla agent.
- Visual State Misunderstanding: Reduced by 37.3 percentage points (from 73.1% to 32.3%).
- Hidden Operation Blindness: Reduced by 26.3 percentage points (from 67.1% to 52.5%), reversing the trend seen with full-image memory.
- Grounding Error: Reduced by 22.5 percentage points (from 27.5% to 22.5%).
Task Success: AGMem improved the overall task success rate by 33.3% over the full-image memory baseline (27.2% vs. 20.4%) and by 9.1% over the vanilla agent (27.2% vs. 18.3%).
Generalization: The improvements were consistent across OSWorld, WebForge, and AgentNetBench, indicating that the approach generalizes across web and desktop environments.

Significance and Claims

The paper claims that naive visual memory is insufficient for GUI agents because it fails to distinguish between global state context and local action cues. The primary contribution is the empirical demonstration that full-image memory trades state-level errors for action-level errors.

The significance of this work lies in:

Taxonomy: Providing a structured classification of GUI agent failures that reveals the specific bottlenecks of different environments.
Mechanism Analysis: Showing that the "divergent effect" of visual memory is a fundamental issue of information density, not just a lack of data.
AGMem Framework: Proposing that action-grounded memory (focusing on cropped, relevant regions) and recovery-aware retrieval are necessary to mitigate the side effects of visual memory. The authors argue that effective memory for GUI agents requires not just more history, but compacted and selective visual evidence aligned with the current subtask and action.

The authors conclude that this analysis-design-reanalysis loop offers a useful lens for future work on memory-augmented GUI agents, emphasizing that the representation of visual memory is as critical as its existence.