💻 computer science

AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents

The paper introduces AndroTMem, a diagnostic framework and benchmark that identifies memory failures as the primary bottleneck in long-horizon GUI agents, leading to the proposal of Anchored State Memory (ASM), a method that significantly improves task completion by representing interaction sequences as causally linked intermediate-state anchors rather than full replays or summaries.

Original authors: Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, Hanqian Li, Yu Huang, Song Dai, Yonghua Hei, Yue Ding, Xiang Li, Shikang Wang

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, Hanqian Li, Yu Huang, Song Dai, Yonghua Hei, Yue Ding, Xiang Li, Shikang Wang, Chengdong Xu, Jingqi Liu, Xueying Ma, Zhiwen Zheng, Xiaofei Zhang, Bincheng Wang, Nichen Yang, Jie Wu, Lihua Tian, Chen Li, Xuming Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Goldfish" Phone Assistant

Imagine you hire a very smart, super-intelligent robot assistant to help you do a complex errand on your phone.
The Task: "Go to the JD app, find the price of AirPods. Then go to Taobao, find the price there. Compare them, buy the cheaper one, and then text your friend Alice on WeChat saying, 'I found a deal!'"

This sounds simple, right? But for a computer, this is a nightmare.

Step 1-10: You are in the JD app. You find the price ($100).
Step 11-20: You switch to Taobao. You find the price ($120).
Step 21: You have to remember that $100 was the cheaper one.
Step 22: You have to remember which item it was so you can buy it.
Step 23: You switch to WeChat to text Alice.

The Issue: Current AI assistants are like goldfish with a 3-second memory. As the task gets longer (more steps), they forget the important details. They might get to the WeChat app and say, "Okay, I'm ready to text Alice," but they have forgotten what to say or which phone they just bought. They get lost in the middle of the journey.

The researchers call this the "Memory Bottleneck." The AI isn't bad at seeing the screen or tapping the right button; it's bad at remembering the story of what it just did.

The Solution: The "Anchored Memory" System

The paper introduces a new system called AndroTMem (Android Trajectory Memory). To fix the goldfish problem, they didn't just give the AI a bigger brain; they gave it a better filing system.

1. The Old Way: The "Scroll of Doom"

Previously, if an AI wanted to remember what happened, it would try to read its entire history from start to finish every single time it needed to make a decision.

Analogy: Imagine trying to find a specific sentence in a 500-page novel by reading the whole book from page 1 every time you want to know what happened in Chapter 3. It's slow, confusing, and you get distracted by irrelevant details (like the description of a tree in Chapter 1 that has nothing to do with the plot).

2. The New Way: The "Anchored State Memory" (ASM)

The researchers propose Anchored State Memory. Instead of reading the whole book, the AI creates a list of "Anchors."

What is an Anchor? Think of an anchor as a milestone or a sticky note you put on a map.
- Anchor 1: "I found the AirPods on JD for $100." (This is a State Anchor).
- Anchor 2: "I found the AirPods on Taobao for $120."
- Anchor 3: "I decided the JD one is cheaper."
- Anchor 4: "I added the JD one to the cart."
How it works: When the AI needs to decide what to do next, it doesn't read the whole history. It looks at its Anchors.
- AI asks: "What did I decide about the price?"
- AI looks at Anchor 3: "Oh right, JD is cheaper."
- AI acts: "Okay, I will buy the JD one."

This is like a hiker on a long trail. Instead of remembering every single step they took (the texture of the dirt, the color of the grass), they remember the landmarks: "I passed the big red rock," "I crossed the bridge," "I reached the summit." These landmarks (Anchors) are enough to navigate the rest of the journey.

The Experiment: The "AndroTMem-Bench"

To prove this works, the researchers built a giant test track called AndroTMem-Bench.

They created 1,069 difficult tasks that require switching between apps and remembering details over 30 to 65 steps.
They tested the best AI models available (like GPT-4o, Gemini, and open-source models).

The Results:

Without Anchors: The AI got lost. As the tasks got longer, the success rate dropped like a stone.
With Anchors: The AI stayed on track. It remembered the "cheap price" and the "friend's name" even after 50 steps.
The Improvement: The new system improved success rates by 5% to 30%. In the world of AI, that is a massive jump.

Why This Matters

Think of the future of AI assistants. We want them to do our taxes, plan our vacations, or manage our entire digital lives. These are long-horizon tasks (they take a long time and have many steps).

If we don't fix the memory problem, our AI assistants will always be clumsy, forgetful helpers who need constant reminders. AndroTMem shows that by organizing memory into structured, connected "anchors" rather than a messy pile of history, we can build agents that are actually reliable enough to work in the real world.

Summary in One Sentence

AndroTMem teaches AI assistants to stop trying to remember every single step of a long journey and instead focus on remembering the key landmarks (Anchors) that connect the beginning of the task to the end.

1. Problem Statement

Long-horizon Graphical User Interface (GUI) agents, which aim to execute complex, multi-step tasks across mobile applications, face a critical bottleneck: interaction memory.

Current Limitations: Existing paradigms rely on either replaying full interaction sequences (which introduces noise and redundancy as sequences grow) or using coarse summaries (which often erase fine-grained, dependency-critical intermediate states).
The Core Issue: Real-world mobile tasks often span dozens of steps across multiple apps. Success depends on preserving and reusing sparse intermediate states (e.g., extracted prices, selected items, or permission grants) that occur early in the trajectory but are crucial for decisions made much later.
Evaluation Gap: Current benchmarks often fail to explicitly test memory retention because their tasks lack strong step-to-step causal dependencies. Consequently, agents may succeed by luck or local perception without truly "remembering" the task state, obscuring the fundamental memory failure modes.

2. Methodology

A. AndroTMem-Bench (Diagnostic Benchmark)

The authors introduce AndroTMem-Bench, a diagnostic framework designed to rigorously evaluate memory in long-horizon Android GUI agents.

Dataset Scale: 1,069 realistic tasks across 50 applications, totaling 34,473 interaction steps (average 32.1 steps/task, max 65).
Design Philosophy: Tasks are constructed with strong step-to-step causal dependencies. Later actions strictly depend on intermediate outcomes from earlier steps (e.g., comparing prices in App A and App B before adding the cheaper one to a cart in App A, then sharing via App C).
Annotation: The dataset includes "State Anchors"—sparse, annotated intermediate states (e.g., Price_JD, Cheaper_Item) that serve as ground truth for memory retrieval.
Metrics:
- TCR (Task Complete Rate): The primary metric, measuring success based on whether the agent correctly preserves and reuses critical intermediate states to reach the final goal.
- AMS (Action Matching Score): Measures step-level action accuracy.

B. Diagnostic Findings

Evaluating 12 diverse agents (open-source, closed-source, and multi-agent) on the benchmark revealed:

Memory is the Bottleneck: Performance degradation as task length increases is driven primarily by within-task memory failures (forgetting or misbinding intermediate states), not by isolated perception errors or local action mistakes.
Failure Modes: Agents frequently suffer from:
1. State Loss: Forgetting extracted values.
2. State Mis-binding: Associating data with the wrong UI element.
3. Context Drift: Losing track of the current subgoal.
4. Unverified Progress: Proceeding without confirming a subgoal was completed.

C. Proposed Solution: Anchored State Memory (ASM)

To address these failures, the authors propose Anchored State Memory (ASM), a structured history representation mechanism.

Concept: Instead of a raw trace or a free-form summary, ASM organizes the interaction history into a compact set of causally linked intermediate-state anchors.
Structure: Each anchor ( $m_k$ $m_{k}$ ) contains:
- Type: Functional role (e.g., Subgoal, State Change, Dependency, Exception, Context Info, Finish).
- Content: Semantic information.
- Evidence: Grounding UI observations (screenshots/bboxes).
- Links: Causal dependency relations connecting it to previous anchors.
Operation: The agent operates in a Retrieve–Reason–Update loop:
1. Retrieve: Fetches relevant anchors based on the current UI state and instruction.
2. Reason: Uses retrieved anchors + current context to predict the next action.
3. Update: Analyzes the step to create new anchors, update existing ones, or establish new causal links.

3. Key Contributions

AndroTMem-Bench: A high-quality benchmark specifically designed to diagnose memory limitations in long-horizon GUI agents through causally dependent, cross-app workflows.
Diagnostic Insight: Empirical evidence proving that long-horizon performance drops are dominated by memory failures rather than perception or action prediction errors.
Anchored State Memory (ASM): A novel memory mechanism that structures history into sparse, causally linked anchors, enabling targeted retrieval and attribution-aware decision-making.
Comprehensive Evaluation: Systematic testing across 12 agents showing that structured memory outperforms both raw replay and summary-based baselines.

4. Experimental Results

The authors evaluated 12 agents (including GPT-4o, GPT-5, Gemini variants, Qwen, and UI-TARS) under three history strategies: Raw History, Coarse Summary, and ASM.

Performance Gains:
- TCR Improvement: ASM improved Task Completion Rates by 5% to 30.16% compared to baselines.
- AMS Improvement: Action Matching Scores improved by 4.93% to 24.66%.
- Example: On the Gemini-3-Flash model, TCR increased from 55.21% (Raw) to 65.05% (ASM). On UI-TARS-1.5-7B, TCR rose from 34.55% to 46.21%.
Robustness: ASM maintained high performance even as the number of interaction steps increased (60+ steps), whereas Raw and Summary methods showed significant degradation.
Efficiency: ASM achieved these gains while keeping token usage and inference time comparable to summary-based methods and significantly lower than raw trajectory replay.

5. Significance

Paradigm Shift: The paper shifts the focus from "perceiving the current screen" to "managing the interaction history." It demonstrates that for long-horizon tasks, structured memory is a core capability required for reliability.
Scalability: ASM provides a scalable solution for complex, real-world automation where tasks involve multiple apps and long dependency chains, addressing a major barrier to deploying GUI agents in production.
Future Research: By isolating memory failures and providing a structured representation (ASM), the work offers a clear path for improving agent reasoning, moving beyond simple context window expansion to semantic state management.

In conclusion, AndroTMem establishes that effective long-horizon GUI agents require more than just large context windows; they need anchored, structured memory that explicitly models causal dependencies between intermediate states to ensure task success.