💻 computer science

Automated Functional Testing for Malleable Mobile Application Driven from User Intent

This paper proposes \tool, an automated framework that leverages LLMs to generate GUI tests and oracles for verifying user-specified functionalities in malleable mobile applications, thereby enabling a shift from product-manager-driven to end-user-driven app development.

Original authors: Yuying Wang, Kaifeng Huang, Hao Deng, Zhiyuan Sun, Jinxuan Zhou, Shengjie Zhao

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuying Wang, Kaifeng Huang, Hao Deng, Zhiyuan Sun, Jinxuan Zhou, Shengjie Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a smartphone app, like a note-taking app or a weather app. Right now, if you want a new feature—say, "I wish I could undo deleting a note by mistake"—you have to wait. You send a complaint to the company, they add it to a long list, and maybe, if enough people ask for it, they might add it in a future update. But until then, you're stuck with what they gave you.

This paper proposes a radical new idea: What if you could just ask the app to change itself, and it would do it instantly?

The authors call this "Malleable Software." Think of it like clay. Right now, most apps are like hard plastic statues; you can't reshape them once they're made. Malleable software is like clay; you can mold it to fit your specific needs.

The Problem: The "Magic Box" Dilemma

The authors imagine a future where you tell an AI, "I want a dark mode for this news app," or "I want a button to share my location." The AI writes the code, updates the app, and gives you your custom version.

But here's the catch: How do we know the AI didn't break the app?
If you ask a human developer to add a feature, they write code and then a tester checks it. But if an AI does it for every single user instantly, we can't hire a human tester for everyone. We need a robot tester that can instantly check:

Did the feature actually appear?
Does it work correctly?
Did it break anything else?

The Solution: Meet "Aladdin"

The authors built a tool called Aladdin (named after the magic lamp, because it grants your wishes). Aladdin is an automated robot that acts as a super-intelligent tester.

Here is how Aladdin works, using a simple analogy:

Imagine you are a detective trying to find a hidden treasure (the new feature) inside a giant, confusing maze (the mobile app).

Phase 1: The Map Maker (Navigation)

Aladdin doesn't just wander randomly. It reads your request ("I want to change the language to Bengali") and looks at the app's screens.

The Analogy: Imagine you are in a mall looking for a specific store. Aladdin is like a GPS that doesn't just say "Go left," but says, "Go left because that sign says 'Electronics,' and your store is likely there." It uses AI to guess which button to click next to get closer to your goal, ignoring dead ends.

Phase 2: The Key Turner (Triggering)

Once Aladdin finds the right screen, it tries to turn the "key" (click the button) to open the feature.

The Analogy: You found the door. Aladdin tries to open it. It clicks the "Language" button. Then, it checks: "Did a list of languages appear?" If the list is empty, it knows the AI failed. If the list appears, it moves to the next step.

Phase 3: The Truth Teller (Oracle Generation)

This is the most important part. Aladdin needs to verify the result.

The Analogy: You clicked "Bengali." Now, Aladdin looks at the app again. It asks: "Is the text now in Bengali? Is the header in Bengali?" It compares the "Before" picture and the "After" picture. If the app looks exactly the same, Aladdin says, "Fail, the feature didn't work." If the app changed correctly, it says, "Pass!"

Why is this a Big Deal?

Currently, software is built by companies for the "average" person. If you have a weird, specific need, you are out of luck.

Old Way: You wait for the company to update the app for everyone.
New Way (with Aladdin): You ask for your custom feature. The AI builds it. Aladdin instantly tests it to make sure it's safe and works. Then, you get your own personal version of the app.

The Results

The researchers tested Aladdin on six real apps (like a notes app, a news reader, and a browser) and 34 different user requests.

Success Rate: Aladdin successfully found the right screens and verified the features 81% of the time.
Detecting Bad Code: When they gave Aladdin apps with "broken" AI-generated features, Aladdin caught the errors 90% of the time. It was very good at saying, "No, this doesn't work," when it shouldn't.

The Bottom Line

This paper envisions a future where software isn't a rigid product you buy, but a flexible tool you shape. Aladdin is the safety guard that makes this possible. It ensures that when you ask your phone to change itself, it doesn't accidentally delete your photos or crash your system. It turns the dream of "vibe coding" (coding just by talking) into a safe, reliable reality for everyone.

1. Problem Statement

The paper addresses the emerging paradigm of per-user malleable mobile applications, where end-users can specify custom requirements via natural language, and these requirements are automatically implemented by Large Language Models (LLMs) without the user needing access to the source code.

Key Challenges Identified:

The "Black Box" Constraint: Most mobile apps are proprietary. Users cannot modify source code directly. Instead, they submit requests to vendors, who use LLMs to generate code in a vendor environment.
The Validation Gap: While LLMs can generate code, ensuring the correctness and presence of user-specified functionalities before releasing a customized binary is a critical bottleneck.
Limitations of Existing Testing:
- Existing LLM-based test generation focuses on unit tests (UI-free scenarios).
- Existing oracle generation relies on static syntax or dynamic execution feedback, failing to validate high-level, semantically meaningful UI behaviors driven by natural language intent.
- Current approaches lack generalizability across different apps and specific user intents.

The core problem is designing an automated functional testing framework that can navigate a mobile GUI, trigger a user-requested feature, and verify its correctness without access to the source code, solely based on the user's natural language requirement.

2. Methodology: The Aladdin Framework

The authors propose Aladdin, a user-requirement-driven GUI test generation framework. It operates in three sequential phases, guided by an LLM, with a Selector Refinement mechanism integrated throughout to fix UI element identification errors.

Phase 1: Functional Triggering State Navigation

Goal: Navigate from the app's home screen to the specific UI state where the user's feature can be triggered.
Mechanism:
- Uses a heuristic-driven scoring mechanism rather than random or fixed exploration (like BFS).
- An LLM analyzes the UI DOM tree and the user requirement to assign a relevance score ( $\gamma$ ) to candidate UI elements.
- A priority queue manages the exploration path. The relevance score is calculated using a geometric mean of atomic scores along the path to penalize low-confidence steps.
- State Equivalence Check: Prevents infinite loops by detecting if a state has already been visited via a different path.
Output: The entry state ( $s'$ ) and the operation sequence to reach it.

Phase 2: Functional Presence Check & Execution

Goal: Verify that the functionality exists and can be successfully triggered.
Mechanism:
- Generates and executes a triggering script (using uiautomator2) based on the entry state.
- Performs Assertion Generation: The LLM creates assertions to verify that the resulting UI contains the expected elements (e.g., "Does the 'Bengali' language option exist in the list?").
- If assertions fail, the Selector Refinement module attempts to repair the UI element selectors (e.g., switching from text to content-desc or using fuzzy matching) to retry the action.
Output: Confirmation that the feature is present and executable.

Phase 3: Functional Correctness Oracle Generation

Goal: Validate that the behavior of the feature matches the user's intent.
Mechanism:
- Captures the Pre-Execution State and Post-Execution State.
- The LLM compares these states to generate a set of sub-oracles (assertions) that define the correct behavioral change.
- Example: If the user requested a "confirmation window," the oracle verifies that the window appears only after a delete action and that the app state changes correctly based on the user's choice (Yes/No).
Output: A final pass/fail decision on whether the implementation satisfies the requirement.

3. Key Contributions

Paradigm Shift Proposal: The paper envisions and validates a shift from "Product-Manager-Driven" to "End-User-Driven" mobile app development, where individual users can request and receive customized app versions.
Aladdin Framework: A novel, three-phase GUI test generation system specifically designed for validating LLM-generated, user-customized mobile features. It uniquely combines navigation, execution, and oracle generation driven by natural language intent.
Comprehensive Benchmark:
- Constructed a dataset of 34 natural language requirements collected from Google Play reviews.
- Mapped these to 6 popular open-source Android apps.
- Generated 144 application versions: 80 correct implementations and 64 faulty implementations (simulating real-world LLM generation errors).
Selector Refinement: A novel mechanism to automatically detect and repair ambiguous or incorrect UI selectors during test execution, significantly improving robustness.

4. Experimental Results

The framework was evaluated on the constructed benchmark using GPT-5 as the underlying LLM.

Success Rate (RQ1):
- Achieved an End-to-End success rate of 81.2% (65/80 correct cases).
- Phase 1 (Navigation) achieved 96.3% success.
- Phase 2 (Presence/Execution) achieved 92.2% success.
- Phase 3 (Oracle) achieved 91.5% success.
Discrimination Capability (RQ2):
- When distinguishing between correct and faulty versions, Aladdin achieved:
  - Precision: 90.3%
  - Recall: 81.2%
  - Specificity: 89.1%
- This indicates the system is highly effective at rejecting faulty implementations (low False Positives), which is crucial for quality assurance.
Comparison with Baselines (RQ3):
- Navigation: Outperformed a prompt-only baseline (74.3% vs. 96.3% success) and required fewer interaction steps (1.54 avg vs. 1.98).
- Oracle Generation: Significantly outperformed AugmenTest (a state-of-the-art LLM oracle generator) in Precision, Recall, and Specificity across all phases.
Overhead: The average time per instance was 246 seconds, with Phase 1 and 2 being the most time-consuming due to iterative navigation.

5. Significance and Impact

Feasibility of Malleable Software: The paper demonstrates that "vibe coding" (coding via natural language) for mobile apps is technically feasible if paired with robust, automated functional testing.
Quality Assurance for AI-Generated Code: It provides a critical solution for the "trust gap" in AI-generated software. Before releasing a customized app to a user, the vendor can automatically verify that the LLM's code actually does what the user asked.
Scalability: The approach supports a "massive delivery" model where unique, per-user app versions can be generated, tested, and released without human intervention, addressing the "long tail" of user demands that traditional development ignores.
Future Direction: The work lays the groundwork for a future where software is not static but dynamically adapted to individual user needs, shifting the software lifecycle from a fixed release model to a continuous, user-centric adaptation model.

Limitations Noted: The current implementation is Android-specific, relies on clear user requirements (vague requests remain a challenge), and the dataset size, while representative, is limited compared to the entire app ecosystem.