🤖 AI

OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis

OpenMobile is an open-source framework that synthesizes high-quality task instructions and agent trajectories through a scalable exploration pipeline and a policy-switching strategy, enabling agents trained on its data to achieve competitive performance on mobile benchmarks while ensuring results stem from broad functionality rather than overfitting.

Original authors: Kanzhi Cheng, Zehao Li, Zheng Ma, Nuo Chen, Jialin Cao, Qiushi Sun, Zichen Ding, Fangzhi Xu, Hang Yan, Jiajun Chen, Anh Tuan Luu, Jianbing Zhang, Lewei Lu, Dahua Lin

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kanzhi Cheng, Zehao Li, Zheng Ma, Nuo Chen, Jialin Cao, Qiushi Sun, Zichen Ding, Fangzhi Xu, Hang Yan, Jiajun Chen, Anh Tuan Luu, Jianbing Zhang, Lewei Lu, Dahua Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot butler how to use your smartphone to do things like "book a flight," "set a reminder," or "find a specific photo."

Right now, the most advanced robot butlers (like those from big tech companies) are incredibly smart. They can do these tasks almost 70% of the time. But there's a catch: they are black boxes. We don't know exactly how they were taught. They learned from secret, private data that no one else can see.

On the other hand, the "open-source" community (the hobbyists and researchers who share their work) is struggling. They are trying to teach their robots using old, messy textbooks (public datasets), and their robots only succeed about 30% of the time. They are stuck in the dark, trying to guess how the pros did it.

Enter "OpenMobile."

Think of OpenMobile as a transparent, open-source "Robot School" that anyone can join. It's a new framework designed to teach mobile agents (robots that use phones) by creating its own high-quality training materials from scratch.

Here is how it works, using some simple analogies:

1. The "Library of Everything" (Task Synthesis)

The Problem: Old methods tried to teach the robot by watching it wander around an app once and then guessing, "Oh, maybe it's trying to do this?" This is like trying to learn a whole city's map by walking down one street and guessing what's in the other neighborhoods. You miss a lot.

The OpenMobile Solution:
Instead of guessing, OpenMobile sends the robot on a scavenger hunt first.

Step 1: The robot wanders through 20 different Android apps, clicking buttons and swiping screens just to see what's possible. It builds a massive "Global Memory" (like a giant library catalog) of every feature every app has.
Step 2: Once the library is built, a teacher (an AI) looks at the catalog and writes new, creative homework assignments.
- Analogy: Instead of saying "Click the red button," the teacher says, "Find the recipe for 'Spicy Tacos' in the cooking app, then text the ingredients to your mom, and finally set a timer for 20 minutes."
- Because the teacher has the whole library, it can create complex, multi-step tasks that are grounded in reality, not just random guesses.

2. The "Coach and the Student" (Trajectory Synthesis)

The Problem: Usually, you teach a robot by showing it a perfect example (an "Expert") doing the task perfectly. The robot copies the expert. But what happens when the robot makes a mistake? If it only sees perfect examples, it doesn't know how to fix a slip-up. It panics and fails.

The OpenMobile Solution:
OpenMobile uses a Policy-Switching strategy. Imagine a student (the Learner) and a master coach (the Expert) working together.

The student tries to do the task.
The coach watches closely. As long as the student is doing okay, the student keeps going.
The Magic Moment: The moment the student starts to go off-track (e.g., clicks the wrong button), the coach immediately steps in, fixes the mistake, and guides the student back to the right path.
The Result: The robot learns not just how to succeed, but how to recover from failure. It learns that making a mistake isn't the end of the world; it's just a chance to correct course. This is the "secret sauce" that most other open-source methods miss.

3. The Results: Closing the Gap

The team used this method to create a dataset of 2,800 complex tasks and 34,000 action steps. They trained their robots on this data and tested them on three major benchmarks (like the "Olympics" of mobile robots).

Before: Open-source robots were scoring around 30%.
After: Their robots scored 51.7% and even 64.7% on the hardest tests.
The Verdict: They are now competitive with the secret, closed-source giants from big tech, proving you don't need a black box to build a super-smart robot.

Why This Matters

The paper also addresses a big fear: "Did they just cheat by memorizing the test questions?"
They ran a "plagiarism check" and found that while their training data was related to the test apps, it wasn't copying the specific test questions. The robots got better because they learned how to think and how to fix mistakes, not because they memorized the answers.

In short: OpenMobile is a toolkit that lets anyone build a robot butler that is smart, adaptable, and knows how to recover when things go wrong, all without needing access to secret corporate data. It's like giving everyone the recipe for the perfect cake, instead of just selling the cake.

1. Problem Statement

Despite significant progress in mobile agents powered by Vision-Language Models (VLMs), a critical gap exists between proprietary systems (e.g., Step-GUI, MAI-UI) and open-source approaches.

The Gap: Leading proprietary systems achieve ~70% success on benchmarks like AndroidWorld but rely on closed, undisclosed training data and synthesis recipes. In contrast, open-source models trained on public datasets (e.g., AndroidControl, AMEX) plateau at roughly 30%.
The Limitations of Current Open Methods:
- Task Synthesis: Existing open methods often couple environment exploration with instruction generation, limiting task diversity to what a single local trajectory reveals.
- Trajectory Collection: Standard "expert distillation" (imitating perfect trajectories) fails to teach agents how to recover from errors. Conversely, "self-evolution" (learning from one's own successes) suffers from slow convergence and capacity ceilings.
- Data Scarcity: There is a lack of large-scale, high-quality, open-source datasets that include both diverse task instructions and error-recovery signals.

2. Methodology: The OpenMobile Framework

OpenMobile is an open-source framework designed to synthesize high-quality task instructions and agent trajectories. It addresses the two core challenges through two novel components:

A. Scalable Task Synthesis (Decoupled Paradigm)

Instead of generating tasks from a single trajectory, OpenMobile decouples exploration from generation to build a Global Environment Memory ( $M$ ).

Environment Exploration: The agent performs exploration (e.g., random walks) across target apps to collect screen-action transitions.
Global Memory Construction:
- Screen Deduplication: Visually similar screens are clustered using perceptual hashing to identify unique screens ( $S$ ).
- Functionality Extraction: A strong VLM extracts natural language descriptions of UI functionalities ( $F$ ) for each unique screen.
- Neighborhood & Indexing: A graph structure ( $N$ ) is built linking screens, and semantic embeddings are created to enable cross-screen retrieval.
Memory-Augmented Instruction Generation: For a target screen, the system constructs a context ( $C$ $C$ ) comprising:
- Short-term Memory: Functionalities of neighboring screens (locally reachable).
- Long-term Memory: Semantically related functionalities from distant screens within the same app.
- Generation: A VLM uses this rich context to synthesize complex, multi-step, grounded instructions that encourage cross-feature composition.

B. Policy-Switching Rollout (Error-Recovery Synthesis)

To collect training trajectories that teach error recovery, OpenMobile introduces a Policy-Switching Strategy that alternates between a Learner ( $\pi_l$ ) and an Expert ( $\pi_e$ ).

The Problem with Alternatives: Pure expert distillation lacks error signals; pure self-evolution is slow and limited by the learner's current capability.
The Solution (Error-Intervention Switching):
1. The Learner executes the task.
2. A Monitor (the Expert model) observes the interaction in real-time.
3. If the Monitor detects the Learner has deviated from productive progress, it triggers a switch to the Expert to intervene and correct the trajectory.
4. The Expert executes a few steps to recover, then control returns to the Learner.
Outcome: This produces trajectories containing rare error-recovery demonstrations (mistake $\to$ correction) while ensuring the task is ultimately completed.

3. Key Contributions

OpenMobile Framework & Dataset: The release of an open-source framework and a dataset containing 2,800 task instructions and 34,000 action steps across 20 Android apps.
Novel Synthesis Techniques:
- Decoupled Task Synthesis: Utilizing global environment memory to generate diverse, complex instructions beyond local trajectory constraints.
- Policy-Switching Rollout: A strategy that effectively captures error-recovery signals, a critical component missing in standard imitation learning.
Transparent Analysis: The authors rigorously analyze the overlap between synthetic data and benchmark test sets, proving that performance gains stem from broad functionality coverage and enhanced error-recovery capabilities rather than data leakage or benchmark overfitting.

4. Experimental Results

The authors fine-tuned Qwen2.5-VL-7B and Qwen3-VL-8B on the OpenMobile dataset and evaluated them on three dynamic benchmarks: AndroidWorld, AndroidLab, and MobileWorld.

Performance Leap:
- AndroidWorld: The fine-tuned Qwen2.5-VL-7B achieved 51.7% success (up from ~25% baseline), and Qwen3-VL-8B reached 64.7%. This significantly outperforms existing open-data approaches (which hover around 30%) and rivals leading closed-data industrial systems (e.g., MAI-UI at 70.7%).
- Generalization: The models showed strong generalization to MobileWorld (a harder benchmark with long-horizon cross-app tasks), improving from 9.4% to 17.7% (Qwen3-VL-8B).
Ablation Studies:
- Task Synthesis: The decoupled memory-based approach yielded higher task complexity and soundness compared to coupled baselines (OS-Genesis).
- Rollout Strategy: The Error-Intervention Switching strategy produced the highest downstream performance (48.3% Pass@1) and the highest density of error-recovery instances (1.56 per trajectory) compared to random switching or pure distillation.
Data Contamination Check: Semantic similarity analysis showed only 3.5% of synthetic instructions had high similarity (>0.7) to test instructions. Removing these high-similarity samples caused only a marginal performance drop, confirming the gains are due to general capability improvements, not overfitting.

5. Significance

Bridging the Gap: OpenMobile demonstrates that open-source agents can compete with proprietary systems if provided with high-quality, synthesized data that covers diverse functionalities and includes error-recovery signals.
Reproducibility: By releasing the code, data, and synthesis recipes, the paper enables the research community to reproduce these results and build upon them, addressing the "black box" nature of current industry leaders.
Future Direction: The work highlights that functionality coverage and error-recovery training are more critical for mobile agent performance than simply scaling up model size or using static datasets. It sets a new standard for open data synthesis in the field of GUI agents.