Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agents

1. Problem Statement

Mobile-use agents, powered by Multimodal Large Language Models (MLLMs), aim to automate tasks on graphical user interfaces (GUIs) by mimicking human interactions. While existing approaches utilize demonstration learning to improve agent performance, they suffer from two critical limitations:

• Neglect of Implicit Intent: Current methods focus solely on explicit intention flows (e.g., step sequences, operational logic). They fail to capture implicit intention flows, such as personal preferences, habits, and user-specific tastes, which are crucial for true personalization.
• Lack of Evaluation and Scalability:
  • There is a lack of benchmarks capable of measuring the Intention Alignment Rate (IAR)—the degree to which an agent's actions align with a specific user's hidden intent, distinct from mere task completion.
  • Fine-tuning a separate agent for every user is computationally impractical.

2. Methodology: IFRAgent Framework

The authors propose IFRAgent, a plug-and-play framework designed to enhance mobile-use agents by extracting and utilizing both explicit and implicit intention flows from human demonstrations. The framework operates in two phases:

A. Intention Flow Extraction Phase

This phase processes historical human demonstrations to build a user profile:

1. Explicit Flow Analysis: An agent ($A_e$) analyzes user trajectories to extract Standard Operating Procedures (SOPs). These SOPs are stored as a query-level vector library $(l_j, p_j)$, where $l_j$ is the embedded query and $p_j$ is the SOP.
2. Implicit Flow Analysis: An agent ($A_i$) analyzes interaction sequences to identify latent behavioral patterns. This information is incrementally updated into a user-level habit repository ($h_i$).

B. Deployment Phase

When a user issues a new (potentially ambiguous) query $q$:

1. Retrieval (RAG): The query is embedded and matched against the user's SOP library. If a similar past query $q'$ is found (similarity > threshold), its SOP $p'$ is retrieved.
2. SOP Extraction: A few-shot learning module (SOP Extractor $E$) uses the retrieved $(q', p')$ pair to generate a standard SOP $p$ for the current query $q$.
3. Personalized Rewriting: A Query Rewriter ($W$), trained via Knowledge Distillation from a powerful teacher model (GPT-4o), takes the original query $q$, the SOP $p$, and the user's habit repository $h_i$ as input. It rewrites them into a personalized query ($\hat{q}$) and a personalized SOP ($\hat{p}$) that reflect the user's specific habits and preferences.
4. Execution: The mobile-use agent ($F$) executes the action $a$ based on the rewritten inputs and the current screen state.

3. Key Contributions

A. MobileIAR Dataset

The authors introduced MobileIAR, the first dataset and benchmark specifically designed to evaluate Intention Alignment Rate.

• Content: Contains 945 instructions across 16 apps and 7 daily scenarios for both English and Chinese users.
• Dual Annotation: Unlike standard datasets, MobileIAR provides two types of ground truth for each task:
  1. Ground-Truth Action Lists: Actions that successfully complete the task (measuring general task completion).
  2. Human-Intent-Aligned Actions: The specific action a human user would take, reflecting their personal preference (measuring intent alignment).
• Significance: It enables the first quantitative assessment of how well an agent aligns with individual user intent, not just task success.

B. The IFRAgent Framework

A novel architecture that decouples explicit logic (SOPs) from implicit habits, allowing agents to "learn" user preferences without per-user fine-tuning.

C. Comprehensive Evaluation

Extensive experiments across diverse models (open-source like Qwen, UI-TARS, and closed-source like GPT-4o) and languages (English/Chinese).

4. Experimental Results

The study evaluated IFRAgent against state-of-the-art baselines on the MobileIAR dataset:

• Performance Gains:
  • Intention Alignment Rate (IAR): Improved by an average of 6.56% absolute (28.80% relative improvement).
  • Step-wise Success Rate (SR): Improved by an average of 5.50% absolute (24.20% relative improvement).
• Model Dependency:
  • General-Domain Models: Models like Qwen2.5-VL and GPT-4o showed the most significant gains. The authors argue that general models possess better world knowledge to understand implicit intent, whereas specialized mobile agents often "forget" general knowledge during post-training.
  • Scale Analysis: The most significant improvements were observed in 7B and 32B parameter models. Smaller models (3B) lacked instruction-following capability, while larger models (72B) already had high baseline performance.
• Ablation & Generalizability:
  • Both the SOP Extractor and the Query Rewriter are essential; removing either degrades performance.
  • The framework generalizes well to other datasets (e.g., modified OS-Kairos) and outperforms standard "SOP demonstration" methods (1-shot learning) by effectively abstracting implicit intent.
• Efficiency: The additional inference time is minimal because the rewriting and extraction are handled by a small, efficient model (Qwen3-4B) processed only once per query.

5. Significance

This work addresses a critical gap in the automation of mobile tasks by shifting the focus from task completion to user alignment.

• Paradigm Shift: It moves mobile agents from being "generic task solvers" to "personalized assistants" that understand user habits without requiring expensive per-user model training.
• Benchmarking: The introduction of MobileIAR establishes a new standard for evaluating the "human-likeness" and personalization capabilities of AI agents.
• Practicality: The plug-and-play nature of IFRAgent makes it feasible to deploy personalized agents on edge devices or within existing agent pipelines immediately.