Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization

1. Problem Definition

The paper addresses a critical gap in Mobile GUI Agents powered by Multimodal Large Language Models (MLLMs). While existing systems excel at completing tasks (e.g., booking tickets, managing emails), they typically optimize for task success rate or efficiency, ignoring user privacy personalization.

• The Core Issue: Different users have distinct privacy preferences (e.g., "Privacy-first" vs. "Utility-first"). These preferences induce structurally heterogeneous execution trajectories.
  • Privacy-first users may take longer paths involving permission denials, logging out, or clearing traces.
  • Utility-first users prefer direct paths, accepting defaults and sharing data for efficiency.
• The Challenge: Standard preference optimization methods (like Direct Preference Optimization, DPO) assume that chosen and rejected trajectories are structurally aligned and of similar length. However, in mobile GUI agents, privacy preferences cause variable-length trajectories with structural mismatches (e.g., one trajectory has extra defensive steps while the other does not).
  • Standard DPO fails here because it treats alignment placeholders (padding) as valid training signals, diluting the learning of critical privacy steps.
  • Key privacy actions are often sparse within long trajectories, making them hard for standard models to learn.

2. Methodology: Trajectory Induced Preference Optimization (TIPO)

To solve the instability caused by structural heterogeneity, the authors propose TIPO, a preference optimization framework designed specifically for variable-length, structurally different trajectories. TIPO modifies the standard step-level DPO loss function with two key mechanisms:

A. Preference-Intensity Weighting

Standard DPO treats all aligned steps equally. TIPO recognizes that not all steps contribute equally to privacy alignment.

• Mechanism: It assigns a weight ($\alpha_t$) to each step based on its "persona relevance."
• Implementation: A rule-based scoring system (assisted by an LLM for semantic normalization) calculates the difference in scores between the chosen ($s^+$) and rejected ($s^-$) steps. Steps with high differences (e.g., denying a permission vs. accepting it) receive higher weights.
• Goal: To amplify the learning signal for critical privacy decisions and prevent them from being drowned out by neutral task-execution steps.

B. Padding Gating

When aligning trajectories of different lengths, shorter trajectories are padded with no_action placeholders to match the length of the longer one.

• Mechanism: TIPO introduces a gating mechanism ($m_t$) that sets the weight of these padding steps to zero.
• Goal: To suppress "alignment noise." The model should not learn to prefer or reject no_action placeholders; it should only learn from genuine semantic differences.

The Objective Function:
The TIPO loss combines these mechanisms:
$$L_{TIPO} = \mathbb{E} \left[ \text{softplus} \left( - m_t \cdot \alpha_t \cdot z_t(\theta) \right) \right]$$
Where $z_t(\theta)$ is the step-wise preference score, $m_t$ is the padding gate, and $\alpha_t$ is the preference intensity weight.

3. Key Contributions

1. Problem Formulation: The paper redefines Mobile GUI agent personalization as a trajectory selection task, where the goal is to select a path consistent with user privacy preferences under the same task objective, rather than just maximizing task completion.
2. Dataset Construction: The authors created the Privacy Preference Dataset, containing 151 task instances across 8 categories (Shopping, Payment, etc.). It features paired trajectories (Privacy-first vs. Utility-first) with ~8.2k annotated steps, explicitly capturing structural heterogeneity and variable lengths.
3. Algorithm (TIPO): The proposal of a novel optimization method that handles structural heterogeneity via preference-intensity weighting and padding gating, outperforming standard DPO and other alignment baselines.

4. Experimental Results

Experiments were conducted using Qwen2.5VL-3B as the backbone, comparing TIPO against baselines like Frozen, SFT, DPO, ORPO, CPO, SimPO, and IPO.

Key Metrics:

• SR (Success Rate): Task completion capability.
• PAS-S / PAS-U: Persona Adherence (Security-oriented vs. Utility-oriented behaviors).
• Compliance: Average adherence to the target persona.
• PD (Persona Distinction): The ability to generate distinctly different trajectories for different personas on the same task.

Performance Highlights:

• Overall Success: TIPO achieved the highest Overall Success Rate (65.60%), proving it maintains task executability while aligning with preferences.
• Persona Adherence: TIPO achieved the highest Compliance score (46.22), significantly outperforming DPO (31.94).
• Persona Distinction: TIPO achieved the best PD score (66.67%), indicating it can effectively distinguish between privacy and utility personas, whereas other methods often converged to a single behavior.
• Ablation Study: Removing either the weighting or gating mechanism resulted in performance drops, confirming that both components are necessary and complementary.

Performance across Task Categories:

• Account & File Management: Highest PD (80.00%) due to explicit structural differences (e.g., local save vs. cloud sync).
• Transactional Tasks: Highest SR (72.17%), showing robustness in complex workflows.

5. Significance and Implications

• Beyond Privacy: While focused on privacy, the TIPO framework is applicable to any user preference that alters execution structure (e.g., cost sensitivity, efficiency, accessibility).
• Shift in Agent Design: The paper argues that personalization should not just be about what the agent outputs (content), but how it executes the task (process/trajectory).
• Practical Impact: For developers, this suggests that collecting paired trajectories under different user profiles and using trajectory-level preference optimization is a viable path to building trustworthy, user-aware mobile agents.
• Technical Advancement: It provides a solution to the "structural heterogeneity" problem in RLHF/DPO, demonstrating that standard alignment methods fail when trajectories differ in length and structure, and offering a robust alternative.

In summary, this paper bridges the gap between task execution and user privacy preferences in mobile agents, introducing a specialized optimization technique (TIPO) and a high-quality dataset to enable agents that respect user-defined privacy boundaries without sacrificing task success.