💬 NLP

iOSWorld: A Benchmark for Personally Intelligent Phone Agents

This paper introduces iOSWorld, the first interactive native iOS simulator benchmark featuring 26 interconnected apps and a persistent user identity to evaluate personally intelligent phone agents, revealing that while privileged vision+XML access significantly boosts frontier model performance, current agents still struggle with complex multi-app and personalization tasks.

Original authors: Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital assistant that lives inside your phone. Right now, most of these assistants are like amnesiac tourists. They can follow a single instruction perfectly—like "turn on the flashlight"—but if you ask them to "plan my week based on my usual habits, my bank balance, and my text messages," they get lost. They don't know who you are, they don't remember your history, and they can't connect the dots between your different apps.

The paper "iOSWorld" introduces a new way to test if a phone assistant is actually smart enough to be your personal helper. Here is the breakdown in simple terms:

1. The Problem: The "Blank Slate" vs. The "Real Life"

Current tests for phone assistants are like giving someone a blank notebook and asking them to write a story. They can write a sentence, but they have no backstory.

The Reality: Your phone is full of a life's worth of data: your bank transactions, your travel plans, your friends' names, and your favorite restaurants. A truly useful assistant needs to know all this.
The Gap: No one had a way to test if an AI could actually navigate this messy, connected real life. Most tests only looked at isolated tasks on Android phones, ignoring the unique way iPhones (iOS) work.

2. The Solution: Building "iOSWorld"

The researchers built a giant, interactive video game that mimics a real iPhone.

The Character: They created a fictional person named Jordan Avery, a 31-year-old living in San Francisco.
The World: They built 26 custom apps for Jordan (like a fake Uber, a fake bank, a fake email, and a fake food delivery app).
The Magic: These apps are all connected. If Jordan orders a burrito on the "QuickBite" app, the "MyBank" app automatically shows a charge, and the "Mail" app gets a receipt. If Jordan books a flight on "SkyTrip," the "Notes" app has a reminder for it.
The Goal: They gave the AI 133 different missions. Some were simple (order a coffee), but many were complex (check your bank balance, find the receipt for that coffee, and tell your boss you're running late based on your calendar).

3. The Test: Two Ways to Look at the Phone

To see how smart the AI really is, they tested it in two different "modes," like giving a human a test with or without a cheat sheet:

Mode A: Vision-Only (The "Blind" Test)
The AI only sees screenshots of the screen, just like a human does. It has to guess where buttons are and figure out what's happening by looking at the pictures.
- Analogy: Like trying to solve a puzzle by only looking at a blurry photo of it.
Mode B: Vision + XML (The "Super-Sight" Test)
The AI sees the screenshots plus a hidden text list (called an accessibility tree) that tells it exactly what every button is called and where it is.
- Analogy: Like looking at the puzzle with a glowing highlighter that points to every piece and says, "This is the sky piece, this is the tree piece."

4. The Results: Who Passed the Test?

They tested the smartest AI models available (the "frontier" models) and one open-source model.

The Good News: When the AI had "Super-Sight" (Vision + XML), the smartest models got much better. They could finally navigate the apps correctly. One model (Opus) jumped from getting 26% of tasks right to 52% right.
The Bad News: Even with the best setup, the AI still struggled with the hardest tasks.
- Single App Tasks: They did great (82% success).
- Multi-App Tasks: They struggled (37% success). Connecting the dots between 3 or 4 different apps was too hard.
- Memory Tasks: They were okay (54% success), but often forgot the details.
The "Small Model" Problem: Smaller, cheaper AI models actually got worse when given the "Super-Sight" data. It was like giving a student too much information; they got overwhelmed and confused.

5. Why Did They Fail?

The researchers found three main reasons the AI got stuck:

Running Out of Time: The AI was given 50 steps to finish a task. On complex jobs, it would use up all 50 steps just trying to figure out where to click, leaving no time to finish the job.
Getting Lost: The AI would open the wrong app or get stuck in a loop, clicking the same button over and over.
The "Coordinate" Problem: In the "Vision-Only" mode, the AI often guessed the wrong spot on the screen to tap, like trying to catch a ball in the dark.

The Bottom Line

iOSWorld is the first time we've tested phone assistants in a realistic, connected environment that mimics a real human's digital life.

The paper concludes that while AI is getting better at looking at a screen and clicking buttons, it is not yet smart enough to be a truly "personally intelligent" assistant that understands your history, your money, and your relationships across different apps. It needs to get better at planning, remembering, and connecting the dots before it can truly replace a human assistant.

The good news? The researchers released all their code and the "game" for free, so other scientists can try to build better assistants using this new playground.

Technical Summary: iOSWorld

Problem Statement

Current benchmarks for mobile digital agents largely fail to evaluate personal intelligence. Existing frameworks (e.g., AndroidWorld, WebArena, OSWorld) typically operate in impersonal sandboxes where agents execute isolated instructions without access to a persistent user identity, history, or cross-application data continuity. A truly useful phone agent must reason over a user's specific identity, preferences, and accumulated data (transactions, messages, travel records) as they exist on the device. Furthermore, while interactive benchmarks exist for Android, there is a significant gap in native iOS evaluation. iOS differs fundamentally from Android in its UI framework, navigation patterns, and lack of open-source accessibility tools, making direct porting of Android benchmarks insufficient.

Methodology

The iOSWorld Environment

The authors introduce iOSWorld, the first interactive native iOS simulator benchmark built around a single, persistent user identity ("Jordan Avery") spanning 26 custom-built iOS applications.

Data Interconnectivity: The 26 apps (covering domains like finance, travel, food, messaging, and productivity) share connected data. For instance, a food order in one app generates a bank charge and a receipt email in others; a flight booking aligns with hotel reservations and calendar reminders across different apps.
Task Categories: The benchmark includes 133 tasks divided into three categories of increasing difficulty:
1. Single-App (27 tasks): Basic navigation and interaction within a single application.
2. Multi-App (60 tasks): Tasks requiring information transfer across 2 to 8 applications (e.g., checking an order in one app, finding the charge in another, and locating the receipt email in a third).
3. Memory & Personalization (46 tasks): Tasks requiring the agent to infer latent patterns from personal data without explicit instructions (e.g., "What is my most common commute route?" or "Find my most frequently ordered restaurant").
Observation Modalities: Agents are evaluated under two settings to separate visual grounding from privileged access:
- Vision-Only: The agent receives only screenshots (resized to 706×1536) and must visually identify UI elements and estimate coordinates.
- Vision+XML: The agent receives screenshots plus a cleaned accessibility tree (XML) extracted via Apple's XCUITest framework. This provides structured data on element types, labels, and normalized coordinates, simulating privileged access unavailable to deployed consumer agents.
Action Space: Actions are translated from provider-specific vocabularies (e.g., "click" becomes "tap xy"). In Vision+XML mode, agents can use accessibility identifiers for precise targeting and launch apps directly via bundle identifiers, bypassing visual home-screen navigation.

Evaluation Protocol

Models Evaluated: Five frontier computer-use models (Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, GPT-5.4 Mini, Gemini 3 Flash) and one open-source baseline (Qwen3.5 35B-A3B).
Scoring: A trajectory-level LLM-as-a-Judge (GPT-5.4 Mini) evaluates the full agent trajectory against rubric criteria. Human validation on 128 trajectories confirmed substantial agreement ( $\kappa=0.77$ ) between the automated judge and human annotators.
Constraints: All runs are limited to a 50-step budget.

Key Contributions

First Native iOS Benchmark with Persistent Identity: iOSWorld provides a dynamic simulator with 26 interconnected apps and a single user persona, addressing the lack of personalization in existing mobile benchmarks.
Comprehensive Task Suite: The release includes 133 tasks across single-app, multi-app, and memory/personalization categories, with a schema for adding new tasks and seeding personalized data.
Modality Comparison: The work evaluates the impact of privileged accessibility data (XML) versus pure vision, revealing distinct performance gaps between model sizes.
Open-Source Release: The authors release all 26 apps, seed data, tasks, rubrics, evaluation code, and an AWS-runner (EC2-managed Mac instances) to enable reproducibility for non-Mac researchers.

Results

Overall Performance

Best Configuration: The best overall configuration (Opus 4.6 with Vision+XML) achieves a 52% pass rate.
Category Breakdown:
- Single-App: 82% (Opus 4.6, Vision+XML)
- Memory/Personalization: 54% (Opus 4.6, Vision+XML)
- Multi-App: 37% (Opus 4.6, Vision+XML)
Baseline: The open-source Qwen3.5 35B-A3B achieves only 11% overall in the Vision+XML setting.

Impact of Privileged Access (Vision+XML)

Frontier Models: Access to the accessibility tree significantly improves performance for larger models. Opus 4.6 improves by +25.6 percentage points (26% $\to$ 52%), Sonnet 4.6 by +18.0 pp, and GPT-5.4 by +19.5 pp.
Smaller Models: Smaller models (GPT-5.4 Mini, Qwen3.5) do not benefit from the additional XML context; in fact, their performance often degrades (e.g., Qwen3.5 drops from 13% to 11%, and GPT-5.4 Mini drops from 26% to 16%). The authors attribute this to context overflow, where the added ~3,100 tokens per step exceed the model's effective capacity, leading to action loops or confusion.

Failure Analysis

Budget Exhaustion: 51% of frontier model failures are due to exhausting the 50-step budget, particularly in multi-app and memory tasks.
Common Failure Modes:
- Vision-Only: Agents frequently fail to estimate coordinates correctly, get stuck on small UI elements (e.g., payment toggles), or fail to switch apps reliably.
- Vision+XML: While coordinate errors are eliminated, agents still struggle with multi-step planning and often get stuck in action loops (especially Qwen3.5).
Step Efficiency: Gemini 3 Flash is the most step-efficient (21 steps/task on average), while Anthropic and OpenAI models average 42–45 steps.

Significance and Claims

The paper claims that iOSWorld establishes a necessary foundation for research into personally intelligent agents. It demonstrates that while frontier models can handle basic single-app interactions, they struggle significantly with the continuity and reasoning required for multi-app and personalization tasks.

The authors emphasize that the gap between current capabilities and useful personal agents is not just a matter of visual grounding but also of planning and memory. The results suggest that:

Privileged Access is a Double-Edged Sword: While accessibility trees solve visual grounding issues for large models, they can overwhelm smaller models, indicating a need for better context management or structured tool interfaces (as shown by Qwen3.5 improving with MCP tools).
Multi-App Reasoning is the Bottleneck: Even the best models only succeed in 37% of multi-app tasks, highlighting the difficulty of maintaining state and transferring information across disjoint application boundaries.
Personalization is Critical: The benchmark proves that agents cannot be evaluated solely on isolated tasks; their ability to reason over a user's history and preferences is a distinct and challenging capability that current benchmarks ignore.

The authors conclude that closing the gap to useful deployment will require stronger loop recovery, better visual grounding, and planning mechanisms that are explicitly aware of user history. They position iOSWorld as a tool to drive this research forward, offering a controlled environment to test personalization without the privacy risks of real user data.