🤖 machine learning

Mem-W: Latent Memory-Native GUI Agents

Mem-W introduces a novel class of GUI agents that integrate historical and working memory directly into the model's latent context as compact tokens, eliminating the mismatch between external symbolic memory and internal representations to significantly improve long-horizon task performance across web and mobile benchmarks.

Original authors: Guibin Zhang, Yaohui Ling, Fanci Meng, Kun Wang, Shuicheng Yan

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guibin Zhang, Yaohui Ling, Fanci Meng, Kun Wang, Shuicheng Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Notebook" vs. The "Brain"

Imagine you are trying to teach a robot to navigate a complex video game world (like a website or a phone app). The robot needs to remember things: "I clicked the wrong button three screens ago," or "I saw a menu that looked like this in a previous game."

The Old Way (The Notebook):
Most current AI agents work like a student with a physical notebook. When they need to remember something, they stop, write a summary in the notebook (e.g., "User wanted to buy shoes, but the cart was empty"), read it back, and then try to use that text to decide what to do next.

The Flaw: This is like translating a thought from English to French, writing it down, translating it back to English, and then thinking about it. It's slow, clunky, and you might lose the nuance of the original feeling. The robot is constantly translating its own history into "human-readable notes" before it can use them.

The Mem-W Solution (The Brain):
The authors of this paper, Mem-W, say: "Why translate at all?"
Instead of writing notes, Mem-W turns the robot's entire history into raw, continuous electrical signals (called "latent tokens") that fit directly into the robot's brain. It's like the robot doesn't need a notebook; it just feels the memory as part of its current thought process.

How Mem-W Works: The "Memory Weaver"

Think of Mem-W as a master weaver who takes two different types of thread and spins them into a single, seamless fabric that the robot can wear.

1. The Two Types of Threads

The robot needs two kinds of memory:

Working Memory (The "Right Now" Thread): This is what happened in the last few seconds of the current task. (e.g., "I just scrolled down and saw a red button.")
Experiential Memory (The "Past Experience" Thread): This is what happened in other tasks the robot did before. (e.g., "Last time I saw a red button, it was a 'Delete' button, so I should be careful.")

2. The Magic Compressor (The "Spinning Wheel")

In the past, these two threads were kept in separate boxes. Mem-W introduces a Compressor.

Imagine a machine that takes a long, messy video of a task and shrinks it down into a tiny, dense "memory chip."
It doesn't summarize the video into text; it turns the video into a compressed signal that the robot's brain can understand instantly.
Crucially, it compresses both the "Right Now" thread and the "Past Experience" thread using the same machine. This means they speak the same language.

3. The Weaving (The "Seamless Fabric")

Once the threads are compressed into these tiny chips, Mem-W weaves them together with the robot's current view of the screen.

Result: The robot looks at the screen and sees a single, continuous stream of information. It doesn't see "Current Screen" + "Old Notes." It sees "Current Screen + Past Lessons + Current Progress" all mixed together in one fluid thought.

How It Learns: "Learning by Doing"

The paper describes a two-step training process to teach this weaver how to work:

Stage 1: The "Shadow" Teacher (Self-Distillation)
- The robot is shown a long, perfect video of a human doing a task.
- The robot tries to copy the human using only the compressed memory chips.
- If the robot makes a mistake, it learns to adjust the "compression" so that the most important details (like "don't click the red button") are preserved in the tiny chip. It's like a student trying to memorize a whole book by only remembering the key plot points, then checking if they got the story right.
Stage 2: The "Outcome" Coach (Outcome-Aware Supervision)
- Now, the robot tries to solve tasks on its own.
- If it succeeds, the system says, "Great! Keep those memory chips exactly as they are."
- If it fails, the system says, "Bad! Those memory chips didn't help you avoid the trap. Change them."
- This teaches the robot to specifically remember what leads to success and what leads to failure, filtering out the noise.

The Results: Why It Matters

The paper tested Mem-W on four different "worlds" (websites and mobile apps). Here is what happened:

It's Faster and Smarter: By skipping the "translation to text" step, the robot made fewer mistakes and finished tasks more often.
It Works on Small Brains Too: Even when they used a smaller, less powerful AI model, adding Mem-W made it perform as well as (or better than) much larger models that didn't have this memory system.
The "Sweet Spot": They found that retrieving about 5 past experiences and compressing the current history was the perfect balance. Too much memory slowed it down; too little made it forgetful.

The Bottom Line

Mem-W is a new way for AI agents to remember. Instead of keeping a diary of what they did, they turn their entire history into a native language that their brain speaks fluently. This allows them to learn from past mistakes and current progress simultaneously, making them much better at navigating complex digital worlds like the internet and smartphones.

In short: Mem-W stops the AI from writing a diary and starts letting it dream in its own native language.

Technical Summary: Mem-W: Latent Memory-Native GUI Agents

Problem Statement

Graphical User Interface (GUI) agents are increasingly tasked with operating complex, long-horizon environments across web, mobile, and desktop platforms. Successful control in these settings requires the agent to retain visual, procedural, and task-level evidence that extends far beyond the immediate screen. However, current approaches treat memory as an external, human-readable artifact. Existing systems typically summarize, categorize, and retrieve interaction histories as text or structured records (e.g., graphs, lists, or layered states) before re-encoding them into the model's policy.

This creates a fundamental representational mismatch: the experience is stored in a symbolic format (text/structured records) that differs from the continuous latent embedding sequence over which modern Vision-Language Models (VLMs) actually operate. This necessitates repeated translation between formats and forces designers to pre-define memory ontologies (categories, representation formats, and interface rules) that may not align with the specific structural needs of downstream tasks. The authors posit that this "ontology-first" paradigm limits the agent's ability to organically learn what information is relevant for task success.

Methodology: Mem-W

The paper introduces Mem-W, a series of latent-memory-native GUI agents designed to eliminate the gap between memory storage and policy execution. Instead of external symbolic scaffolds, Mem-W treats memory as a continuous part of the agent's context, woven directly into the embedding space.

Core Architecture

Unified Latent Space: Mem-W unifies two distinct memory scales into a single learnable space:
- Experiential Memory: Long-horizon, cross-session historical trajectories retrieved from an external bank.
- Working Memory: Short-horizon, in-session history (the "expired prefix" of the current episode).
Trajectory-to-Latent Compressor ( $C_\phi$ ): A shared, lightweight compressor (implemented as a query-based transformer following the Q-Former design) maps variable-length observation-action segments into fixed-size latent blocks ( $K \times d$ $K \times d$ ).
- It processes both retrieved historical trajectories and the current episode's expired prefix.
- It incorporates an outcome embedding ( $\gamma(y)$ ) that encodes whether a trajectory resulted in success or failure, allowing the agent to distinguish procedural polarity without explicit filtering rules.
- For online working memory where the outcome is unknown, a special "unknown" marker is used.
Dual-Scale Weaving: At inference, the compressed memory tokens are woven with the current GUI observation and local context into a single continuous embedding sequence. The frozen GUI agent policy ( $\Pi_\theta$ ) then attends to this unified sequence as native context, rather than as external notes.

Training Strategy

Mem-W employs a two-stage training procedure to optimize the compressor parameters ( $\phi$ ) while keeping the backbone GUI agent ( $\Pi_\theta$ ) frozen:

Stage 1: Self-Distillation: The compressor learns to replace raw historical context. A "teacher" agent observes an extended raw window of history, while a "student" agent observes the latent-augmented input. The objective minimizes the divergence between the teacher's and student's action distributions, forcing the latent tokens to capture decision-relevant state.
Stage 2: Outcome-Aware Supervision: The system optimizes the compressor using environment feedback (terminal rewards). Using a policy-gradient objective (specifically RLOO-based), the compressor learns to encode evidence that leads to task success and filter out failure-inducing patterns, guided by the outcome embeddings.

Key Contributions

Unified Formulation: The paper recasts the intricate hierarchy of GUI memory as a single latent representation problem. It argues that working memory and experiential memory can be compactly grounded in one learnable space where task-relevant structure emerges via end-to-end optimization, removing the need for prescribed taxonomies.
Learnable Instantiation: Mem-W provides a concrete implementation using a shared trajectory-to-latent compressor. This allows both retrieved historical data and online episode prefixes to be projected into a unified latent space, trained via self-distillation and outcome-aware supervision.
Empirical Evidence: Extensive experiments demonstrate that this approach consistently improves diverse backbones and outperforms agents equipped with hand-engineered memory architectures.

Experimental Results

Mem-W was evaluated across four benchmarks: Multimodal-Mind2Web, MMInA (web navigation), GUI-Odyssey, and AndroidControl-v2 (mobile navigation).

Performance Gains: Mem-W consistently improved performance across all tested backbones (including UI-TARS, Qwen3-VL, and UI-Venus).
- On MMInA, Mem-W achieved gains of up to +30.0 percentage points over baseline models (e.g., UI-Venus-1.5-8B improved from 18.5% to 48.5%).
- On Multimodal-Mind2Web, improvements ranged from +8.83 to +20.58 points depending on the backbone and task subset.
- On AndroidControl-v2, Mem-W-8B achieved a Pass@4 score of 81.16, exceeding proprietary models like GPT-5+UGround and larger open-source baselines.
Comparison with Memory-Augmented Agents: Mem-W outperformed existing memory-augmented agents (e.g., ReasoningBank, AWM, CoMEM, HyMEM). The authors attribute this to Mem-W's ability to dynamically distill trajectories into compact, decision-relevant latent summaries and its use of outcome-aware supervision, whereas competitors often rely on coarse truncation or expert-trajectory imitation without explicit success/failure conditioning.
Efficiency: Despite the additional compression step, Mem-W maintained a favorable accuracy-efficiency trade-off, often reducing the number of steps required to complete tasks and lowering the rate of hitting maximum step limits compared to baselines.
Scalability: Performance scaled positively with the size of the external memory bank (up to 50K trajectories) and the number of retrieved trajectories, indicating the system effectively exploits larger reservoirs of experience.

Significance and Claims

The paper claims that latent-context-native memory serves as a scalable foundation for long-horizon GUI agency. By treating memory as a continuous substrate rather than an external symbolic scaffold, Mem-W allows the agent to "read" successes, failures, and unfinished progress through the same machine-native interface used for current observations.

The authors conclude that the bottleneck in long-horizon GUI agency is not merely model scale, but how interaction history and prior experience are represented and exposed to the policy. Mem-W demonstrates that a unified, end-to-end learnable latent interface can replace prescribed memory hierarchies, enabling task-relevant memory structures to emerge naturally from training. This perspective suggests a path toward interactive multimodal agents whose competence is built on learned latent context interfaces rather than manually engineered memory modules.