Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots

1. Problem Statement

Mobile agents powered by Large Language Models (LLMs) aim to automate complex smartphone tasks by simulating human interactions with Graphical User Interfaces (GUIs). Despite advancements, these agents suffer from low accuracy, misinterpretation of user intent, and premature task termination. Two critical gaps exist in current research:

1. Lack of Failure Analysis: There is limited systematic understanding of why and where these agents fail in real-world, "in-the-wild" scenarios.
2. Input Modality Trade-offs: Most agents rely on screenshots (multimodal input) for visual context. However, screenshots raise significant privacy concerns and ecosystem pushback (e.g., restrictions by app developers). Conversely, screentext (UI tree/Accessibility API data) is less invasive but may lack crucial visual cues. It remains unclear if screenshots are strictly necessary for high performance or if text-only inputs suffice.

2. Methodology

A. The DailyDroid Benchmark

To address these gaps, the authors introduced DailyDroid, a comprehensive benchmark designed to evaluate mobile agents under realistic conditions.

• Scope: 75 tasks across 25 popular Android applications.
• Categories: Five scenarios: Productivity & Tools, Utilities & System, Information Access, Media & Entertainment, and Communication.
• Difficulty: Three levels (Simple, Medium, Hard) based on interaction complexity, navigation depth, and ambiguity.
• Design Principles: Tasks reflect everyday user intents, avoid sensitive data/payments, and are designed to expose friction points like dynamic content, permission prompts, and ambiguous UI labels.

B. Experimental Setup

• Models: Two OpenAI models were tested:
  • GPT-4o: A baseline multimodal model.
  • o4-mini: A reasoning-oriented model with advanced chain-of-thought capabilities.
• Input Modalities:
  • Text-only (Screentext): Structured UI tree data (converted to simplified HTML/XML) extracted via Android Accessibility Services.
  • Multimodal: Screentext + Screenshot (pixel representation).
• Framework: The experiments utilized an extended version of AutoTask, an open-source mobile agent framework. The authors disabled external memory and knowledge injection to isolate core agent capabilities.
• Metrics: Task Success Rate, System-Level vs. Agent-Level Failures, Execution Time (Total, Effective, API Delay), Step Count, Token Usage, and Monetary Cost.
• Scale: 300 total runs (75 tasks × 2 models × 2 modalities).

C. Failure Analysis Protocol

The authors employed an inductive coding approach (Grounded Theory) to analyze failed runs. Failures were categorized into:

• System-Level Failures: Infrastructure breakdowns (UI retrieval failure, parsing errors, logic ambiguity, execution errors) causing early termination.
• Agent-Level Failures: Logical errors despite functional infrastructure (LLM prediction errors, reflection failures, hitting step limits, impossible tasks).

3. Key Contributions

1. DailyDroid Benchmark: A standardized, human-centered evaluation suite focusing on everyday friction, privacy-aware modality trade-offs, and repeatable diagnostic evaluation.
2. Failure Handbook: A structured taxonomy of 10+ failure types (e.g., UI Retrieval, UI Parsing, LLM Prediction) with specific distributions across models and modalities.
3. Modality Comparison: A rigorous empirical comparison showing that while multimodal inputs offer marginal gains, they come at a high cost in privacy and computation.
4. Design Implications: Concrete recommendations for app developers (UI accessibility), OS designers (parsing protocols), and agent architects (action spaces and step constraints).

4. Key Results

A. Performance Overview

• Success Rates: Overall success rates were low, ranging from 26.7% to 33.3%.
  • o4-mini slightly outperformed GPT-4o (31.3% vs. 29.3% overall success).
  • Multimodal inputs yielded marginally higher success rates than Text-only (approx. 4-5% improvement), but the difference was not statistically massive.
• Cost Efficiency: Text-only inputs were significantly cheaper. Multimodal runs incurred ~25x higher costs due to token usage from images, despite similar success rates.
• Time: Multimodal inputs increased API latency significantly due to image processing.

B. Failure Analysis

• System-Level Dominance: System-level failures accounted for ~40-43% of all failures, primarily due to UI Retrieval (missing UI elements in the tree) and UI Parsing errors. This indicates that if the agent cannot "see" the UI via the API, the LLM's reasoning is irrelevant.
• Agent-Level Failures:
  • Prediction Errors: o4-mini significantly reduced prediction errors compared to GPT-4o (e.g., 4.0% vs. 10.8% in multimodal text), demonstrating the value of reasoning capabilities.
  • Step Limits: Interestingly, the more advanced o4-mini was more prone to hitting the maximum step limit (12% increase) because it engaged in more self-correction and backtracking, whereas GPT-4o often persisted with errors until termination.

C. Modality Insights

• When Screenshots Help: Multimodal inputs were crucial for tasks involving:
  • Non-textual visual cues (colors, icons, spatial layout).
  • Incomplete text representations (e.g., hidden search bars, reading progress indicators).
  • Complex, multi-step abstract tasks where visual context aids reasoning.
• When Text Suffices: For structured, text-heavy tasks (e.g., settings, lists), screentext performed comparably to multimodal inputs.

5. Significance and Recommendations

A. For App Developers & UI Designers

• Accessibility is Critical: The primary bottleneck is not the LLM's intelligence but the inaccessibility of UI elements. Developers must ensure all interactive elements have unique, descriptive IDs and are exposed via Accessibility APIs.
• Standardization: The paper advocates for standardized UI parsing protocols and schema markup (similar to Model Context Protocol) to make apps "LLM-readable."

B. For Agent Designers

• Action Space Expansion: Current agents lack advanced actions (long-press, swipe, back). Expanding the action space is necessary for robust automation.
• Step Constraints: Reasoning models (like o4-mini) benefit from higher step limits to allow for self-correction loops, which are currently cut off prematurely.
• Context Integration: Reflection modules need better integration with historical context to avoid mismatches between predicted and executed actions.

C. Privacy and Ecosystem

• Privacy-First Approach: Since multimodal inputs offer only marginal performance gains but introduce significant privacy risks (screen recording), the authors recommend prioritizing screentext (Accessibility API) for mobile agents. Screenshots should be a fallback only when text representation is insufficient.

Conclusion

The paper concludes that LLMs do not necessarily need to "see everything" (screenshots) to function effectively in mobile automation. While visual input helps with specific edge cases, the primary barrier to success is UI accessibility and the quality of text-based screen representation. Future progress depends less on more powerful vision models and more on improving the underlying UI standards and accessibility of mobile applications.