Scaling Mobile Chaos Testing with AI-Driven Test Execution
This paper presents an AI-driven mobile chaos testing system that integrates LLM-based test execution with service-level fault injection to automatically validate resilience at scale, successfully identifying critical architectural risks and reducing debugging time across Uber's mobile applications.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the internet as a giant, bustling city where millions of tiny digital messengers (apps) run around delivering messages, food, and rides. Behind the scenes, these messengers rely on a massive, invisible web of services—like power plants, traffic lights, and warehouses—to keep everything running smoothly. Sometimes, one of these invisible parts breaks or gets slow. This is called a "failure." In the world of software engineering, there's a practice called Chaos Engineering, which is basically the art of intentionally breaking things on purpose to see if the system can survive. It's like a fire drill for a building, but instead of fire, you're simulating a power outage to see if the emergency lights work.
However, testing mobile apps (the apps on your phone) is tricky. Unlike a simple website, a phone app has to deal with thousands of different cities, languages, and user habits. If you tried to test every possible way the app could break in every city, you'd need to write millions of test scripts. It's like trying to check every single door in a city the size of New York to see if they lock properly; it's impossible for humans to do manually. This paper tackles that impossible math problem by using Artificial Intelligence (AI) to do the testing instead of people.
The researchers at Uber, a company that connects riders with drivers and people with food, built a system that combines two powerful tools. The first tool is an AI robot named DragonCrawl. Think of DragonCrawl as a super-smart, curious teenager who can look at a phone screen, understand what it sees (like "Oh, that's a button to order a pizza"), and figure out what to click next without needing a human to write a specific instruction for every single button. The second tool is uHavoc, a system that acts like a controlled saboteur. It intentionally breaks parts of the backend (the invisible web of services) to see how the phone app reacts.
Usually, these two tools worked separately. DragonCrawl would test if the app worked when everything was perfect, and uHavoc would test if the servers could handle a crash. But the researchers realized that if they combined them, they could create a "chaos testing" machine that runs automatically. They let DragonCrawl navigate the app while uHavoc simultaneously broke things in the background. The goal was to see if the app could keep working (like ordering a ride) even when the backend services were having a bad day.
The paper explains how they built this system and what happened when they turned it on. They didn't just guess; they ran the system over 180,000 times starting in the first quarter of 2024. They tested 47 critical flows (like booking a ride or ordering food) across Uber's Rider, Driver, and Eats apps. The results were impressive. The AI-driven system found 23 serious problems that humans had missed. Most of these were cases where a small, unimportant service failing caused a big, important feature (like paying for a ride) to crash. In fact, two of the issues found would have caused the app to completely freeze and crash, which is something that only testing on a real phone could catch.
One of the coolest parts of their discovery is how they fixed the "blame game." When a test failed, it used to take senior engineers hours to figure out which broken service caused the phone app to act up. Their new system uses AI to look at the evidence and guess the culprit. It was right about the top cause 88% of the time when looking at the top five guesses. This cut the time it took to fix problems from hours down to minutes.
The paper also highlights that this wasn't just a theory. They proved it works in the real world without hurting real customers. They found that the AI was incredibly reliable, passing 99% of the tests even when things were breaking in the background. This means the system is robust enough to run every night, constantly checking if the app is ready for disaster. The authors argue that this approach solves the "combinatorial explosion" problem—the idea that the number of tests needed grows so fast it becomes impossible to handle. By using AI to adapt to the screen in real-time, they didn't need to write millions of test scripts; the AI just figured it out as it went.
However, the paper is careful to note what this system can't do. It can't test things like internet cable cuts or failures in services Uber doesn't control (like a third-party map provider). It also relies on the company having their services properly labeled and tracked, which takes a lot of work to set up first. But for the problems it does solve, the paper suggests this is a massive leap forward. It turns mobile chaos testing from a rare, expensive, manual event into a routine, automated safety check that happens every single night, ensuring that when you tap "Order" on your phone, the app is ready for anything the internet might throw at it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.