SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
The paper introduces **SWE-Bench Mobile**, a new benchmark designed to evaluate the ability of LLM agents to handle complex, industry-level iOS development tasks, revealing that current state-of-the-art agents still struggle significantly with real-world mobile software engineering.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you’ve just hired a super-smart, lightning-fast intern to help you build a massive, complex skyscraper. This intern has read every book in the library and can write code faster than you can blink. You’d think they’d be a superstar, right?
Well, a new research paper called "SWE-Bench Mobile" just put these "AI interns" (Large Language Model Agents like Cursor, Claude, and GPT) to the ultimate test. Instead of asking them to solve simple math problems or write tiny snippets of code, the researchers gave them a real-world challenge: "Here are the blueprints, here are the interior design sketches, and here is our massive, existing building. Now, go add a new elevator system."
Here is the breakdown of what happened, explained through a few simple analogies.
1. The Challenge: The "Master Architect" Test
Most AI tests are like asking a student to solve a single equation on a piece of paper. They are easy, isolated, and predictable.
SWE-Bench Mobile is different. It’s like giving the AI a massive, 500,000-line "living" city (a real professional iOS app from a company called XiaoHongshu). To finish a task, the AI can't just write code; it has to:
- Read the Manual (The PRD): Understand the written instructions.
- Look at the Sketches (Figma): Look at visual designs to see where buttons and colors should go.
- Navigate the Labyrinth (The Codebase): Find the right "rooms" in a massive, sprawling building without getting lost.
2. The Result: The "Brilliant but Clumsy" Intern
The results were a reality check. Even the best AI "interns" only successfully finished 12% of the tasks.
Think of it like this: The AI is like a brilliant student who knows all the theory but has never actually held a hammer.
- They might know exactly how a door should look (the code), but they forget to check if the door frame is actually there (the existing codebase).
- They might build a beautiful new room, but they forget to connect the electricity to the rest of the house (the "feature flags" and "data models").
The biggest takeaway? The "tools" the intern uses matter more than their "brain." A smart intern using a high-end toolkit (like the commercial agent Cursor) performed 6 times better than the same smart intern using a basic, DIY toolkit (the open-source OpenCode).
3. Why did they fail? (The "Missing Screw" Problem)
The researchers looked at the "wreckage" to see why the AI failed. The failures usually fell into three categories:
- The "Forgetful Builder" (Missing Feature Flags): In professional software, you don't just turn a new feature on for everyone at once; you use a "hidden switch" (a feature flag) to test it slowly. The AI often forgot to install the switch, which is a huge "no-no" in the real world.
- The "Tunnel Vision" Problem (Incomplete Implementation): The AI would fix one small thing but forget that the change affected five other rooms in the house. It would fix the sink but forget to turn on the water main.
- The "Complexity Wall": If a task was easy (changing one line), the AI was okay. But as soon as the task required changing 7 or 8 different files at once, the AI’s performance plummeted. It’s like a person who can follow a simple recipe but gets completely overwhelmed if they have to cook a five-course Thanksgiving dinner.
4. The Bottom Line
The paper concludes that while AI coding agents are incredibly promising, they aren't ready to be "Lead Architects" yet.
Right now, they are "Super-Powered Copilots." They can help you fly the plane, but you still need a human in the cockpit to make sure they don't accidentally turn off the engines while trying to adjust the air conditioning.
In short: We have built the world's fastest apprentices, but we still need the masters to supervise the construction.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.