Technical Summary: AppWorld-UL

Problem Statement

Current benchmarks for evaluating LLM-based tool-use agents largely focus on fully specified, autonomous tasks where goals are clear, complete, and fixed from the outset. However, real-world digital task solving is inherently "user-in-the-loop": users often provide vague, underspecified, or even infeasible instructions, requiring agents to iteratively refine intent through interaction. Existing interactive benchmarks fail to capture the diversity and complexity of these interactions. They are limited by:

1. Narrow Interaction Types: Most restrict interactions to simple clarification questions, missing critical patterns like handling infeasibility or seeking approval for high-cost actions.
2. Unbalanced User Simulation: Prior work oscillates between rigid, rule-based users (lacking flexibility) and unconstrained LLM simulators (introducing instability and making failure attribution difficult).
3. Small Environments: Many operate in limited environments with few, often non-state-changing APIs, failing to test long-horizon planning in complex, stateful systems.

Consequently, there is a lack of a principled testbed to evaluate agents that must effectively combine tool use with diverse, adaptive user communication.

Methodology

The authors introduce AppWorld-UL, a benchmark built upon the existing AppWorld framework (which simulates 9 popular apps like Amazon and Spotify with ~475 APIs). The core innovation is a perturbation-based methodology that systematically transforms well-defined autonomous tasks into user-in-the-loop variants.

1. Perturbation-Based Task Transformation

Instead of constructing interactive scenarios from scratch, the authors modify existing autonomous tasks by introducing specific, known gaps in the instruction ($I$) or initial state ($S_0$) that can only be resolved through user interaction. This creates a task tuple $T = (I, S_0, E, K)$, where $K$ is the User Knowledge Set—information held exclusively by the user.

The perturbations target three specific interaction types:

• Underspecification: Removing critical details (e.g., "return the shirt I bought last week" when multiple shirts exist) to force the agent to query the user for clarification.
• Infeasibility: Modifying the state or constraints so the original goal is unattainable (e.g., the requested item is out of stock), requiring the agent to communicate the failure and propose alternatives.
• Confirmation-Seeking: Introducing high-cost consequences (e.g., a price spike) to a previously low-risk action, requiring the agent to seek explicit approval before proceeding.

These perturbations are also composed to create tasks requiring multiple interaction types simultaneously.

2. Constrained User Simulation

To address the instability of prior user simulators, AppWorld-UL employs a constrained LLM-based user.

• Knowledge Boundaries: The simulated user is prompted with a specific set of question-answer pairs ($K$) derived from the perturbation process.
• Controlled Response: The user LLM is instructed to answer only if the agent's question maps to a pair in $K$. If the question is outside $K$, the user deflects or redirects (e.g., "Please decide it yourself").
• Reliability: This approach ensures reproducibility (by limiting the user's knowledge) while maintaining natural conversational flow, avoiding the rigidity of rule-based systems and the unpredictability of unconstrained LLMs.

3. Programmatic Evaluation

The benchmark introduces Interactive Task Goal Completion (I-TGC) and Interactive Scenario Goal Completion (I-SGC). Unlike standard metrics, these include a unit test verifying that the agent successfully elicited all necessary information from the user (i.e., asked all questions in $K$). This allows for the evaluation of interaction quality (recall of required information) alongside task success.

Key Contributions

1. Perturbation Framework: A systematic method for generating diverse user-in-the-loop tasks by injecting specific information gaps into autonomous tasks, enabling precise control over interaction requirements.
2. AppWorld-UL Benchmark: A dataset of 516 challenging tasks across 9 applications, comprising:
   • 306 tasks requiring exactly one interaction type (underspecification, infeasibility, or confirmation).
   • 210 compositional tasks requiring combinations of two or three interaction types.
3. Constrained User Simulator: A robust simulation mechanism that balances flexibility and evaluation reliability by strictly defining user knowledge boundaries.
4. Empirical Evidence: Demonstration that correct user interaction is a primary driver of task difficulty and success, distinct from environmental complexity.

Experimental Results

The authors evaluated state-of-the-art agents (Code Agents and Function Calling Agents) using various frontier LLMs (including Claude Opus 4.7, GPT-5.5, Qwen 3.7 Max, and others).

• Overall Performance: Even the best-performing system (Code Agent with Claude Opus 4.7) achieved only 48.6% I-TGC and 30.2% I-SGC on the overall benchmark.
• Compositional Difficulty: Performance dropped significantly on compositional tasks, with the best system achieving only 35.7% I-TGC and 21.3% I-SGC.
• Interaction Correlation: There is a strong correlation between interaction quality and task success. Successful agents achieved ~100% recall and 80-91% precision in asking required questions, while failed agents showed drastically lower metrics (37-40% recall).
• Ablation Study:
  • Hidden Knowledge: When the simulated user refused to provide any information, performance plummeted (TGC: 12.8%), confirming that agents cannot solve these tasks through environmental reasoning alone.
  • Oracle Knowledge: When all user knowledge ($K$) was provided upfront in the prompt, performance improved significantly (TGC: 78.1%), indicating that the interaction requirement itself accounts for the majority of the difficulty. However, performance remained lower than on original autonomous tasks, suggesting additional challenges in integrating split information.

Significance

AppWorld-UL provides a principled testbed for advancing research on user-in-the-loop tool-use agents. The paper claims that current models struggle substantially with the nuances of real-world interaction, such as recognizing underspecification, handling infeasibility, and seeking approval. By isolating interaction requirements from environmental complexity, the benchmark reveals that the ability to effectively communicate with users is a critical bottleneck for agent deployment. The work aims to support the development of agents that are not only capable of tool use but also behave as collaborative partners who can query, confirm, and adapt to user intent in realistic, stateful digital environments.