🤖 machine learning

MobileDev-Bench: A Comprehensive Benchmark for Evaluating Language Models on Mobile Application Development

MobileDev-Bench is a comprehensive benchmark featuring 384 real-world issue-resolution tasks across Android Native, React Native, and Flutter platforms that reveals significant performance gaps in state-of-the-art LLMs, with end-to-end resolution rates as low as 3.39%–5.21% primarily due to challenges in fault localization across multi-file and multi-artifact changes.

Original authors: Moshood A. Fakorede, Krishna Upadhyay, A. B. Siddique, Umar Farooq

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Moshood A. Fakorede, Krishna Upadhyay, A. B. Siddique, Umar Farooq

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of incredibly smart, super-fast robots (Large Language Models, or LLMs) that can write code. You've tested them on building simple houses (web apps) and fixing broken furniture (standard software libraries), and they've done pretty well. They can follow instructions and fix a leaky faucet or build a bookshelf.

But what happens if you ask these robots to fix a high-speed, self-driving race car that has to navigate a specific track with strict rules, weird sensors, and a complex engine? That's what this paper, MobileDev-Bench, is all about.

Here is the story of the paper, broken down into simple parts:

1. The Problem: The "Race Car" Gap

The authors realized that while AI is great at fixing "house" problems (web apps), nobody had really tested it on "race car" problems (mobile apps like those on your phone).

Mobile apps are tricky. They aren't just one piece of code. They are like a Swiss Army knife where you have to change the blade, the handle, the screw, and the instruction manual all at the same time, or the whole thing breaks.

The Old Benchmarks: Were like testing a robot on a single Lego brick.
The Real World: Is like asking the robot to rebuild a whole Lego city where changing one building affects the traffic, the power grid, and the water supply.

2. The Solution: Building a New Test Track (MobileDev-Bench)

To see if these AI robots could actually fix real mobile apps, the researchers built a new, super-detailed test track called MobileDev-Bench.

The Source: They didn't make up fake problems. They went to 18 real, popular mobile apps (like Thunderbird, WordPress, and Zulip) and found 384 real bugs that real humans had fixed.
The Test: They took the "before" version of the app, gave the AI the bug report, and asked the AI to fix it. Then, they ran the app's own tests to see if the fix actually worked.
The Complexity: These weren't simple fixes. On average, fixing one bug required changing 12.5 different files and 325 lines of code. Some fixes even required changing the source code and the app's "ID card" (manifest files) and the "instruction manual" (build files) simultaneously.

3. The Results: The Robots Got Lost

The researchers tested four of the smartest AI models available (GPT-5.2, Claude, Gemini, and Qwen). The results were surprising and a bit scary for the future of AI coding:

The Score: The AI models only managed to fix 3.4% to 5.2% of the bugs.
The Analogy: Imagine you give a robot a map to find a lost key in a house. On a simple house (web app), it finds the key 80% of the time. On a massive, multi-story mansion with secret passages (mobile app), it finds the key only 5 times out of 100.

4. Why Did They Fail? The "Where" vs. The "How"

The researchers dug deep to find out why the robots failed. They discovered the problem wasn't that the robots couldn't write the code (the "How"). The problem was that they couldn't figure out where to look (the "Where").

The Bottleneck: It's called Fault Localization.
- Imagine a doctor trying to cure a patient. The doctor knows how to perform surgery (write code), but they can't find which organ is sick (locate the bug).
- In mobile apps, a bug in the "display" might actually be caused by a problem in the "database" or the "settings menu." The AI models got confused by these connections.
- When a fix required changing just one file, the AI did okay (about 15% success). But when a fix required changing 6 or more files, the success rate dropped to almost zero.

5. The Big Takeaway

The paper concludes that while AI is getting better at writing code, it is still terrible at understanding the complex web of connections inside a mobile app.

Current AI: Great at writing a single sentence.
Mobile Apps: Require writing a whole paragraph where every sentence depends on the others, and you have to change the font, the paper, and the ink color all at once.

The Future

The authors say we need to teach AI to be a better "detective" before we teach it to be a better "writer." We need to help AI understand how different parts of a mobile app talk to each other (like how the battery, the screen, and the internet connection all affect the app).

Until then, if you want to fix a bug in a mobile app, you still need a human expert to point the AI in the right direction. The AI is a powerful tool, but it's not ready to drive the race car alone yet.

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong capabilities in automated software engineering tasks (e.g., resolving GitHub issues in Python libraries), existing benchmarks largely ignore mobile application development. Mobile development presents unique engineering constraints that differ significantly from general-purpose library development:

Strict Platform Constraints: Mobile apps operate within platform-defined architectures (Android, iOS) with lifecycle-managed execution.
Complex Build Systems: Compilation and testing rely on framework-specific tools (Gradle, npm, pub) rather than generic interpreters.
Multi-Artifact Dependencies: Correctness often requires coordinated changes across heterogeneous artifacts, including source code (Java/Kotlin/Dart/TS), resource files (XML, images), manifests, and build configurations.
Lack of Evaluation: Current benchmarks (e.g., SWE-bench) focus on single-language, library-style repositories, failing to capture the multi-file, multi-artifact complexity inherent in mobile apps.

2. Methodology: MobileDev-Bench Construction

The authors constructed MobileDev-Bench, a new benchmark designed to evaluate LLMs on real-world mobile app issue resolution. The construction followed a rigorous five-phase pipeline:

Phase 1: Repository Selection: Selected 18 production-grade mobile repositories from F-Droid and GitHub, spanning Android Native (Java/Kotlin), React Native (TypeScript), and Flutter (Dart). Criteria included high community adoption (≥400 stars), recent maintenance, and verified buildability.
Phase 2: Issue-PR Mining: Retrieved 2,100+ merged Pull Requests (PRs) linked to GitHub issues. Filtered for PRs that modified test files and contained clear issue descriptions.
Phase 3: Environment Setup: Automatically generated Docker-based build environments for each repository to ensure reproducibility, inferring dependencies from configuration files (e.g., build.gradle, pubspec.yaml).
Phase 4: Execution-Based Validation: Validated candidates by running tests in three states:
1. Base Commit: Tests should fail or be absent.
2. Test Patch: Tests are added/modified but implementation is missing (should fail).
3. Fix Patch: Implementation added (should pass).
- Key Innovation: The benchmark explicitly handles compilation-aware validation, ensuring the project compiles before tests run, which is critical for statically typed mobile languages.
Phase 5: Human Verification: Two authors manually reviewed 560 execution-validated candidates to ensure issue clarity, test alignment, and appropriate difficulty. This resulted in a final set of 384 verified instances.

Benchmark Statistics:

Tasks: 384 instances (124 Easy, 133 Medium, 127 Hard).
Complexity: Average fix modifies 12.5 files and 324.9 lines of code.
Multi-Artifact: 35.7% of tasks require changes across multiple artifact types (e.g., source + manifest + resources).

3. Key Contributions

MobileDev-Bench Dataset: The first comprehensive benchmark for mobile app issue resolution, covering three major frameworks and four languages with 384 manually verified, execution-validated tasks.
Compilation-Aware Evaluation Framework: A novel evaluation harness that supports Java, Kotlin, Dart, and TypeScript, capable of handling complex build systems and multi-artifact dependencies, unlike previous Python-centric tools.
Comprehensive Empirical Analysis: A detailed evaluation of four state-of-the-art LLMs (GPT-5.2, Claude Sonnet 4.5, Gemini Flash 2.5, Qwen3-Coder) revealing systematic failure modes specific to mobile development.

4. Experimental Results

The authors evaluated four frontier LLMs using an adapted Agentless pipeline (Fault Localization $\to$ Patch Generation).

Low Resolution Rates: The models achieved extremely low end-to-end resolution rates, ranging from 3.39% to 5.21%. This is significantly lower than performance on library-based benchmarks (e.g., SWE-bench).
Primary Bottleneck: Fault Localization:
- The main failure point is not generating the code, but identifying which files need to be changed.
- File-level Recall was very low (13.8% – 19.5%).
- Recall degrades sharply with complexity: ~34–53% for single-file tasks, dropping to 2–3% for tasks requiring 11+ file modifications.
Impact of Complexity:
- Multi-File/Multi-Artifact: Models struggle significantly with tasks requiring coordinated changes across source, resources, and build files. Single-artifact tasks had a 1.36x higher resolution rate than multi-artifact tasks.
- Patch Size: Success rates dropped as patch size increased. Tasks with >500 lines of changes had near-zero resolution rates.
Framework Differences:
- Flutter tasks showed slightly better relative performance (overrepresented in resolved instances), likely due to more consistent architectural patterns and single-artifact dominance.
- React Native tasks had 0% resolution, suggesting extreme difficulty in handling the TypeScript/JavaScript co-modification and complex dependency chains.

5. Significance and Implications

Redefining the Challenge: The paper challenges the assumption that patch generation is the primary difficulty in automated repair. For mobile development, fault localization (identifying the correct set of files and artifacts) is the critical bottleneck.
Limitations of Current Benchmarks: Existing benchmarks fail to capture the "coordinated change" nature of mobile apps. A fix in a mobile app often requires updating a manifest, a resource file, and the source code simultaneously; current models fail to reason about these cross-artifact dependencies.
Future Research Directions:
- Development of dependency-aware retrieval mechanisms that can trace cross-file and cross-artifact relationships.
- Incorporation of framework-specific priors (e.g., understanding Android lifecycles or Flutter widget trees) into localization strategies.
- Exploration of multi-modal approaches to utilize screenshots and crash logs often found in mobile bug reports.

In conclusion, MobileDev-Bench exposes a significant gap between current LLM capabilities and the requirements of real-world mobile software engineering, highlighting that while models can write code, they currently lack the architectural understanding necessary to fix complex, multi-component mobile applications.