Technical Summary: Large Language Models for Mobile GUI Text Input Generation: An Empirical Study

1. Problem Statement

Mobile applications have become ubiquitous, making their quality assurance critical. While Graphical User Interface (GUI) testing is a standard method for ensuring app quality, automated exploration faces a significant bottleneck: text-input components. Unlike simple click or swipe actions, many UI pages require semantically appropriate text inputs (e.g., specific email formats, valid location names, or password constraints) to transition to subsequent screens. Traditional automated tools often fail here, generating random strings (e.g., "hello world") that do not satisfy the app's context, leading to incomplete UI exploration and low coverage.

Although Large Language Models (LLMs) have demonstrated potential in generating context-aware text, it remains unclear how different UI-context representations (textual vs. visual vs. structural), execution feedback, and human intervention affect the effectiveness of LLM-based text-input generation for Android GUI testing. There is a lack of systematic empirical evidence comparing these factors across state-of-the-art models.

2. Methodology

The authors conducted a large-scale empirical study involving nine state-of-the-art (SOTA) LLMs.

2.1 Experimental Setup

• Subjects: 115 Android apps collected from Google Play (providing 115 UI pages with 183 text-input components) and 37 real-world text-input-related bugs collected from F-Droid and GitHub.
• Models: Nine LLMs were selected based on leaderboard rankings (Chatbot Arena+ and LLM Arena), availability, and multimodal support (text and vision). These included proprietary models (e.g., GPT-5.4, Claude-4.6, Gemini-3.1) and open-weight models (e.g., Qwen3.5, Mistral-large-3).
• Tasks:
  1. Page Pass-Through (PPTR): Generating valid text inputs to navigate from the current UI page to the next.
  2. Bug Detection (BDR): Generating invalid text inputs to trigger specific, known text-input-related bugs.

2.2 Prompting Strategies

The study evaluated three distinct UI-context prompting settings:

1. Extracted-Context Prompting: Uses natural language descriptions extracted from the UI hierarchy, including component context (hint text, resource ID), adjacent context (labels nearby), and global context (app name, activity name).
2. UI-Hierarchy-XML Prompting: Feeds the raw UI-hierarchy XML (obtained via UIAutomator) directly to the LLM, exposing the full structural tree.
3. Screenshot-Vision Prompting: Uses the screenshot image of the UI page combined with a textual prompt, leveraging multimodal capabilities.

2.3 Feedback Protocol

A feedback-enhanced generation protocol was tested where failed attempts (parsing errors, validation failures, or execution outcomes) were fed back to the LLM in a new prompt to guide subsequent generation attempts.

2.4 Human Evaluation

Professional testers (20 for assessment, 3 for modification) evaluated the quality of LLM-generated inputs and attempted to reconstruct or modify them to improve effectiveness.

2.5 Integration

The LLM-generated text inputs were integrated into DroidBot, a well-known automated Android testing tool, to evaluate improvements in UI exploration capabilities under Depth-First Search (DFS) and Breadth-First Search (BFS) strategies.

3. Key Contributions

1. Comprehensive Empirical Study: The first study to systematically compare three UI-context prompting settings (extracted text, XML, screenshot) across nine recent LLMs for Android text-input generation.
2. Feedback Protocol Evaluation: An evaluation of a feedback-enhanced generation protocol for both page pass-through and bug detection tasks.
3. Human-in-the-Loop Analysis: A quantitative comparison of LLM-generated inputs against tester-generated and tester-modified inputs.
4. Tool Integration: Integration of LLM capabilities into DroidBot to demonstrate practical improvements in automated testing workflows.
5. Practical Insights: Six specific insights for the Android testing community regarding the trade-offs between cost, effectiveness, and the role of feedback and human refinement.

4. Key Results

4.1 Prompting Effectiveness (RQ1)

• Page Pass-Through: Extracted-context and UI-hierarchy-XML prompting achieved similar Page Pass-Through Rates (PPTR) of 71.4% and 71.0%, respectively. Screenshot-vision prompting was slightly lower at 65.1%.
• Cost Efficiency: Extracted-context prompting was significantly more cost-effective, requiring an average of 1,272.7 tokens per attempt. In contrast, XML-based prompting required 7,896.6 tokens, and screenshot-vision required 5,664.5 tokens.
• Bug Detection: All three settings achieved similar Bug Detection Rates (BDR) at 30 attempts (~51%). However, extracted-context remained the most cost-effective, while XML and vision incurred substantially higher token costs without significant BDR gains.
• Model Differences: Statistical analysis showed that for a specific prompting setting, most LLM pairs exhibited no significant differences in performance. The choice of prompting strategy had a greater impact than the choice of the specific LLM.

4.2 Feedback Protocol (RQ2)

• Page Pass-Through: Feedback improved the average PPTR from 69.18% to 73.78% (a 4.60 pp gain). The improvement was statistically significant but modest.
• Bug Detection: Feedback had a more substantial impact here, increasing the average BDR from 50.95% to 64.46% (a 13.51 pp gain).
• Cost: The feedback protocol significantly increased token usage and latency, particularly for bug detection (latency increased from ~35s to ~506s per attempt on average).

4.3 Human Assessment and Modification (RQ3)

• Assessment: Testers rated valid page-pass-through inputs highly (mean score ~4.1/5), but rated invalid bug-detection inputs lower (mean score ~3.2/5), noting that LLMs struggled to generate the specific, diverse invalid inputs required to trigger bugs.
• Modification: When professional testers modified LLM-generated inputs, the effectiveness improved dramatically.
  • Page Pass-Through: PPTR increased from 59.75% (LLM baseline) to 90.00% (tester-modified).
  • Bug Detection: BDR increased from 36.10% (LLM baseline) to 100.00% (tester-modified).
  • Testers leveraged execution history and specific bug descriptions to refine inputs, a capability the static LLM prompts lacked.

4.4 Practical Significance (RQ4)

• Tool Integration: Integrating LLM-generated text inputs into DroidBot increased UI-page exploration by 36.06% (DFS) and 26.41% (BFS) compared to the original tool.
• Insights: The study concludes that extracted-context is the practical default for cost-effectiveness; feedback is valuable for difficult cases but expensive; and human refinement is highly effective for edge cases.

5. Significance and Claims

The paper claims to advance the knowledge of the Android testing community by providing empirical evidence on how to effectively utilize LLMs for GUI text-input generation.

• Cost-Effectiveness: It establishes that extracting relevant textual context is often superior to feeding raw XML or screenshots due to the massive token cost difference, with negligible loss in effectiveness for page navigation.
• Feedback Utility: It demonstrates that while feedback loops improve bug detection significantly, they come with high latency and token costs, making them suitable only for high-value, difficult testing scenarios.
• Human-AI Collaboration: The study highlights that while LLMs provide a strong baseline, professional testers can significantly bridge the gap to 100% effectiveness by modifying inputs based on execution outcomes and specific bug triggers.
• Tool Enhancement: It validates that augmenting traditional automated tools (like DroidBot) with LLM-generated context-aware inputs can substantially improve UI coverage, addressing a known limitation of random or static input generation.

The authors maintain a modest stance, noting that their findings are based on specific samples (115 apps, 37 bugs) and current LLM versions, and that results may vary with different app architectures or future model iterations. They emphasize that the study offers practical guidelines rather than universal rules for all Android testing scenarios.