From Feedback to Failure: Automated Android Performance Issue Reproduction

This paper introduces RevPerf, an automated framework that synthesizes ambiguous user app reviews through semantic retrieval and prompt engineering to generate and execute reproduction steps, successfully identifying Android performance issues with a 72.73% success rate.

The Gist

Imagine you own a bustling restaurant. One day, a customer leaves a note on the table: "The soup was great, but when I tried to eat the big bowl of chili, it took forever to cool down, and my spoon got stuck!"

As the chef (the app developer), you read this. You know the soup is fine. But you have no idea why the chili is slow or why the spoon got stuck. Did they use a specific bowl? Was the kitchen too hot? Did they stir it too fast? Without more details, you can't recreate the problem in your kitchen to fix it.

This is exactly the problem mobile app developers face with Android apps. Users leave reviews saying things like "It's so slow!" or "It freezes!" but they don't give the technical details needed to fix it.

This paper introduces RevPerf, a smart robot assistant designed to solve this mystery. Here is how it works, broken down into simple steps:

1. The Detective Work (Gathering Clues)

The Problem: A single user review is like a blurry photo. It's often vague, emotional, or missing key details.
The Solution: RevPerf acts like a super-detective. When it reads a complaint (e.g., "The app is laggy"), it doesn't just look at that one note. It goes into the "library" of thousands of other reviews for that same app version and finds similar notes from other customers.

• Analogy: If one person says "The chili is slow," RevPerf finds three other people who said, "I tried the chili with a metal spoon," and "I added extra hot sauce." It stitches these clues together to build a complete picture of what went wrong.

2. The Rehearsal (Acting it Out)

The Problem: Even with clues, you can't fix the soup until you actually try to make it and see what happens.
The Solution: RevPerf has a "hands" (an execution agent) that can control a virtual Android phone (an emulator). It reads the combined clues and starts acting them out.

• The Twist: It's not just clicking buttons blindly. If the review says "It happens when you rotate the screen," RevPerf rotates the screen. If it says "It happens when you type a long text," RevPerf types a massive paragraph.
• The Safety Net: If the robot tries to click a button and nothing happens, it doesn't give up. It thinks, "Hmm, maybe I need to unlock the screen first," or "Maybe I need to switch apps," and tries a different path. It keeps adjusting its actions until it successfully recreates the user's problem.

3. The Medical Checkup (Finding the Glitch)

The Problem: Sometimes the app looks fine on the screen, but it's actually struggling underneath (like a car engine making a weird noise even if the car is moving).
The Solution: While the robot is acting out the problem, it has three "doctors" watching closely:

• The Log Doctor: Reads the app's secret diary (system logs) to see if it screamed "I'm tired!" or "I'm stuck!"
• The Resource Doctor: Checks the app's vital signs (battery, memory, CPU). Is the app eating up all the memory like a greedy kid eating candy?
• The Eye Doctor: Watches the screen to see if the app froze or if the buttons stopped responding.

If any of these doctors find a problem, RevPerf rings the bell and says, "Found it! Here is exactly what happened, and here is the proof."

Why is this a big deal?

Before this, developers had to guess what was wrong or wait for a user to send a perfect, step-by-step video of the bug. That's rare.

• The Result: RevPerf successfully recreated 72% of the performance problems it was tested on.
• The Comparison: Other tools that tried to do this (but focused on crashes, not slowness) only got about 45% right. RevPerf is like a master chef who can finally recreate a dish just by reading a vague food critic's review.

The Bottom Line

RevPerf turns vague, messy complaints like "My phone is hot and slow" into a clear, reproducible recipe that developers can use to fix the code. It bridges the gap between what users feel and what developers need to know to make apps run smoothly.

Technical Summary

Here is a detailed technical summary of the paper "From Feedback to Failure: Automated Android Performance Issue Reproduction" (RevPerf).

1. Problem Statement

Mobile application performance issues (e.g., lag, ANR, battery drain, memory leaks) are critical to user experience but notoriously difficult to detect and reproduce in development environments.

• The Gap: While app stores contain vast amounts of user feedback regarding performance, these reviews are unstructured, ambiguous, and often lack the technical context (e.g., specific UI screens, device configurations, or workload conditions) required for reproduction.
• Limitations of Existing Work: Current automated bug reproduction frameworks primarily focus on crashes and functional bugs. They fail to address performance issues because:
  1. Performance issues lack clear binary failure signals (unlike crashes).
  2. They depend on complex, hidden environmental factors (e.g., rapid typing, screen rotation, specific system settings).
  3. Existing datasets lack sufficient examples of reproducible performance issues.
• Core Challenge: How to automatically translate vague, subjective user reviews into a structured, executable reproduction plan that can trigger and detect performance degradation in an Android emulator.

2. Methodology: The RevPerf Framework

The authors propose RevPerf, the first framework to automatically reproduce and detect Android performance issues using user reviews. It operates in three main stages (see Fig. 5 in the paper):

A. Review Augmentation (Context Enrichment)

To address the lack of structured context in user reviews:

• Semantic Retrieval: The system encodes the original review and retrieves the top-$M$ most similar reviews from the same app version using vector embeddings (cosine similarity).
• LLM Filtering: A Large Language Model (LLM) filters these candidates to ensure they share the same issue topic and provide new, relevant information.
• Synthesis: The original review, selected relevant reviews, and essential app context (e.g., login credentials, app description) are synthesized into an Augmented Review. This enriched prompt provides the execution agent with a clearer understanding of the issue, potential triggers, and necessary settings.

B. Feedback-Driven Execution

An execution agent simulates user interactions within an Android emulator using an iterative Observation–Thought–Action loop:

• Observation: The agent monitors the GUI hierarchy (XML), system logs, and resource usage.
• Thought: The LLM interprets the augmented review and current state to plan the next action. Crucially, it checks for environmental constraints mentioned in reviews (e.g., "disable animations" in system settings, "dark mode") and applies them before executing the trigger.
• Action: The agent executes commands via uiautomator2 and ADB.
  • Extended Action Set: Unlike standard frameworks, RevPerf supports complex interactions like input_long_text (to stress test text processing), lock/unlock_screen, and open_app_drawer (for cross-app switching).
  • Adaptability: If an action fails, the agent backtracks or tries alternative UI elements.

C. Multi-Faceted Issue Detection

Since performance issues lack a single failure signal, RevPerf employs three complementary analyzers to detect anomalies during execution:

1. LogAnalyzer: Uses an LLM to scan Android logs (Logcat) for specific patterns like frame drops, rendering delays, ANR warnings, or resource closure failures.
2. ResourceDiagnoser: Monitors runtime metrics (CPU, Memory, Heap, Active Activities) over time. It uses an LLM to detect sustained growth trends indicative of memory leaks or resource exhaustion.
3. UIInspector: Analyzes the evolution of the UI state to detect visual symptoms like UI freezing, unresponsiveness, or abnormal state transitions.

The system aggregates evidence from all three analyzers. If a performance issue is confirmed, it generates a structured report; otherwise, it provides a failure analysis.

3. Key Contributions

1. Empirical Study & Dataset: The authors conducted a manual analysis of 95,067 app reviews, identifying 616 developer-attended reviews. They manually verified 55 high-quality, reproducible performance issues across 29 open-source apps, creating a benchmark dataset for future research.
2. Novel Framework (RevPerf): The first automated system to reproduce performance issues from unstructured reviews. It introduces:
   • A semantic retrieval mechanism to enrich context.
   • An execution agent capable of handling complex environmental dependencies (e.g., system settings, rapid inputs).
   • A multi-modal detection strategy (Logs + Resources + UI) to handle the subjectivity of performance complaints.
3. Comprehensive Evaluation: Extensive experiments demonstrating the framework's superiority over state-of-the-art baselines.

4. Experimental Results

The framework was evaluated on the 55 manually verified performance issues:

• Success Rate: RevPerf successfully reproduced 40 out of 55 issues, achieving a 72.73% success rate.
• Comparison:
  • Outperformed the best baseline (ReBL) by 27.28% (ReBL achieved 45.45%).
  • Outperformed AdbGPT significantly (9.1% success rate).
• Efficiency: The average reproduction time was 146.66 seconds per issue.
• Ablation Study: Removing any single component (ReviewAggregator, LogAnalyzer, ResourceDiagnoser, or UIInspector) resulted in a significant drop in success rate (ranging from 7.5% to 30% relative decrease), proving the necessity of the multi-component design.
  • Example: Removing the ReviewAggregator dropped success to 61.82%.
  • Example: Removing the LogAnalyzer dropped success to 50.91%.

Failure Analysis: The 15 failures were attributed to:

• Need for rapid, consecutive human-like typing (hard to simulate via standard ADB).
• Issues requiring comparison between different app versions.
• Dynamic UI rendering (e.g., Canvas-based galleries) that accessibility services cannot detect.

5. Significance

• Bridging the Gap: RevPerf effectively bridges the gap between unstructured user feedback and actionable engineering data, allowing developers to reproduce performance bugs that were previously impossible to trigger in a lab setting.
• Beyond Crashes: It shifts the focus of automated bug reproduction from deterministic crashes to the more elusive and subjective realm of performance degradation.
• Practical Impact: By automating the detection of performance regressions, memory leaks, and ANRs, the tool significantly reduces the time and expertise required for mobile app maintenance, directly improving user experience.
• Future Direction: The paper highlights the potential of using LLMs not just for code generation, but for interpreting subjective human feedback and orchestrating complex, multi-step system interactions.