Technical Summary: Me-Agent

1. Problem Statement

Large Language Model (LLM)-based mobile agents have advanced significantly in executing complex tasks across application scenarios. However, current agents face critical limitations in personalization and handling ambiguous instructions. Existing systems primarily rely on explicit user instructions, failing to account for latent user preferences or historical interaction patterns. This leads to three specific challenges:

1. Inability to interpret ambiguous instructions: Agents struggle when users omit application names (e.g., "Play Hey Jude" without specifying "QQ Music") or use vague references (e.g., "Play my favorite song").
2. Lack of learning from interaction history: Agents cannot continuously update their understanding of user behavioral patterns across multi-turn interactions.
3. Failure to handle personalized instructions: Current agents lack mechanisms to process preference configurations, hindering user-centric experiences.

Furthermore, existing personalization methods for LLMs are ill-suited for mobile agents. Fine-tuning approaches are impractical due to limited mobile computational resources and storage, while cloud-based training raises privacy concerns. Conversely, prompt-based methods that continuously expand context without filtering mechanisms degrade inference efficiency and complicate memory management.

2. Methodology: Me-Agent

To address these challenges, the authors propose Me-Agent, a learnable and memorable personalized mobile agent framework. Me-Agent operates without additional model training (parameter-free) and utilizes a two-level user habit learning approach:

A. User Preference Learning (UPL)

Inspired by training-free reinforcement learning, UPL is a parameter-free module designed to model user preferences through a Personal Reward Model. It operates in four stages:

1. Rollout: The agent performs $G$ independent rollouts for a given instruction to generate diverse execution trajectories.
2. Reward: A Vision-Language Model (VLM) evaluates these trajectories based on screenshot sequences, assessing goal achievement, step validity, result visibility, and error detection to generate a continuous reward signal ($r_g \in [0, 1]$).
3. Advantage: An LLM analyzes successful and failed trajectories to extract structured, natural language experiences. This includes a Comparative Critique phase that identifies generalizable insights across six categories: user preferences, UI navigation, action sequences, element identification, context awareness, and completion signals.
4. Optimization: New experiences are integrated into an experience pool ($E$) via an operation mechanism ($ADD, UPDATE, DELETE, KEEP$) to prevent redundancy and manage memory decay.

B. Hierarchical Preference Memory (HPM)

To address context overflow and attention dilution caused by injecting all memory into prompts, HPM stores application-specific knowledge externally in a two-level hierarchy:

• Level-1 Memory ($M_{L1}$): Organized by intent categories (e.g., Music, Navigation), mapping categories to sets of applications.
• Level-2 Memory ($M_{L2}$): Stores app-specific data, including workflows (instruction execution patterns, UI element positions) and content preferences (user-specific items within an app).

Inference Mechanism:
During execution, Me-Agent resolves ambiguity in two stages:

1. Application Resolution: If the instruction lacks an explicit app, the agent infers the target application ($a^*$) based on user preference probabilities derived from the experience pool.
2. Content Retrieval: For implicit content references (e.g., "my usual"), the agent retrieves top-$k$ candidates from the relevant app's memory based on semantic similarity and selects the final content ($c^*$) via LLM reasoning.
   The resolved application, content, and relevant experiences are then injected into the prompt to guide execution.

3. Key Contributions

The paper outlines four primary contributions:

1. Me-Agent Framework: A personalized mobile agent that adapts to user preferences via two-level habit modeling (prompt and memory levels) without requiring model parameter updates, making it suitable for API-based deployment.
2. User Preference Learning (UPL): A parameter-free module that models user preferences using a Personal Reward Model and structured experience accumulation.
3. Hierarchical Preference Memory (HPM): A memory architecture that separates long-term general preferences from app-specific knowledge to mitigate context overflow while enabling continuous behavioral modeling.
4. User FingerTip Benchmark: A new dataset designed to evaluate mobile agents on implicit preference reasoning. It features two types of ambiguous instructions:
   • Type I (App-Ambiguous): Omitting the application name (e.g., "Play Hey Jude").
   • Type II (Content-Ambiguous): Replacing specific content with vague references (e.g., "Play my favorite song").

4. Experimental Results

The authors evaluated Me-Agent on the User FingerTip benchmark and the general E-dataset.

• Personalization Performance (User FingerTip):
  • Me-Agent achieved a perfect App Selection Accuracy (ASA) of 1.000 for both Type I and Type II instructions, significantly outperforming baselines (Mobile-Agent-v2 and Mobile-Agent-E), which struggled to infer preferences without explicit modeling.
  • On Type II instructions, Me-Agent showed superior BERTScore (0.725 vs. 0.549) and Preference Score (PS) (0.534 vs. 0.275), demonstrating its ability to infer content preferences.
• General Task Performance (E-dataset):
  • Me-Agent achieved a Task Completion Rate (TCR) of 89.3%, a substantial improvement over the baseline Mobile-Agent-E (63.9%) and Mobile-Agent-v2 (41.4%).
  • It maintained high Action Fidelity (AF) (0.861) and Reflection Precision (RP) (0.978).
• Ablation Studies:
  • Experiments confirmed that both UPL and HPM are complementary. UPL improved task completion by shifting policy optimization to the context space, while HPM enhanced semantic quality and preference alignment through on-demand retrieval.
• Robustness:
  • The framework demonstrated stable performance across different backbone models (GPT-4o, Claude-3.5-Sonnet, Gemini-2.5-Pro), indicating that the modular design reduces dependence on specific backbone capabilities.

5. Significance and Limitations

Significance:
The paper claims that Me-Agent represents a significant step toward user-centric mobile agents. By decoupling personalization from parameter tuning, it offers a practical solution for mobile environments with limited resources. The framework effectively bridges the gap between explicit instructions and implicit user intent, allowing agents to handle real-world ambiguity (e.g., inferring "QQ Music" for "Play Hey Jude" based on history) without requiring retraining. The introduction of User FingerTip provides a necessary benchmark for evaluating these implicit reasoning capabilities, which previous datasets failed to assess.

Limitations:
The authors acknowledge two main limitations:

1. UI Dynamics: The system relies on stored UI positions and action sequences from successful executions. These learned experiences may become invalid if the application interface is updated or if unexpected pop-ups appear, potentially leading to execution failures.
2. Contextual Factors: The current personalization approach relies primarily on historical preferences and behavior patterns. It does not yet incorporate dynamic contextual factors such as the user's current location or emotional state, which could limit recommendation accuracy in situations deviating from usual patterns.