Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants

1. Problem Statement

Current research on Large Language Model (LLM) agents has largely focused on reactive paradigms, where agents execute tasks only after receiving explicit user instructions. However, proactive agents that anticipate user needs and autonomously execute tasks hold greater promise for digital assistants.

The primary bottleneck in developing and evaluating proactive agents is the lack of realistic user simulation frameworks. Existing benchmarks and environments suffer from three critical limitations:

1. Flat API Abstraction: Most frameworks model applications as flat tool-calling APIs. This fails to capture the stateful and sequential nature of real-world mobile interactions (e.g., navigating screens, filling forms).
2. Passive User Simulation: Existing user simulators are often passive or lack constraints. Real users are restricted by the current UI state (they cannot call an API directly without navigating to the correct screen), whereas assistants often have unrestricted backend access. This asymmetry is missing in current evaluations.
3. Infeasible Evaluation: Without an active user who interacts with the environment, it is impossible to evaluate whether a proactive agent correctly infers goals, times its interventions appropriately, or respects user autonomy.

2. Methodology: The Pare Framework

The authors introduce Pare (Proactive Agent Research Environment), a simulation framework designed to bridge the gap between theoretical proactive agents and realistic deployment.

A. Asymmetric Stateful Environment

Pare models the interaction between a User and an Assistant with distinct constraints:

• User Simulator (Active): The user agent interacts with applications via Finite State Machines (FSMs). To perform an action (e.g., sending a message), the user must navigate through specific screens (Open App $\to$ Search $\to$ Open Conversation $\to$ Send). The user's action space is strictly state-dependent.
• Proactive Assistant: The assistant has access to a flat API across all applications, allowing it to gather information and execute tasks directly without navigating UI screens. This mirrors real-world deployment where assistants have backend access while users are limited to the UI.
• Observation Asymmetry:
  • User: Receives truncated notifications (e.g., sender and subject only) and only sees their own history.
  • Assistant: Receives full serialized content of events and observes all user actions and environment states.

B. Interaction Protocol (Stackelberg POMDP)

The interaction is formalized as a Stackelberg Partially Observable Markov Decision Process (POMDP):

1. User (Leader): Acts first, taking actions based on their goals and observations.
2. Assistant (Follower): Observes user actions and environment events, infers latent goals, and proposes a plan ($p$).
3. Decision: The user accepts or rejects the proposal. If accepted, the assistant executes the plan; if rejected, the assistant returns to observation mode.
4. Objective: Maximize Goal Success Rate ($R_{Succeed}$) and Plan Acceptance Rate ($R_{Accept}$).

C. Agent Architecture: Observe-Execute

The proactive assistant is implemented using a two-phase architecture to preserve user autonomy:

1. Observe Mode: The agent monitors user actions and environment events using read-only tools. It decides whether to wait or propose a task.
2. Execute Mode: Triggered only after user acceptance. The agent uses full API access to autonomously complete the multi-step task across different apps.

3. Key Contributions

1. Pare Framework: The first asymmetric simulation environment where users navigate state-dependent FSMs while agents access flat APIs. This creates an ecologically valid testbed for proactive assistance.
2. Pare-Bench: A benchmark comprising 143 diverse scenarios across four domains: Communication, Productivity, Scheduling, and Lifestyle. It tests four core capabilities:
   • Context Observation
   • Goal Inference
   • Intervention Timing
   • Multi-app Orchestration
3. Observe-Execute Architecture: A novel agent design that separates continuous monitoring from task execution, ensuring the agent only acts upon explicit user consent.
4. Comprehensive Evaluation: Evaluation of seven LLMs (including frontier models like GPT-5, Claude 4.5, and smaller open-weights models) demonstrating the current state of proactive AI.

4. Experimental Results

The authors evaluated seven models on Pare-Bench. Key findings include:

• Performance Ceiling: Even the best frontier models (Claude 4.5 Sonnet, Gemini 3 Flash) achieved only a ~42% success rate. This indicates that proactive assistance is a significantly harder task than reactive instruction following.
• Consistency Gap: There is a massive drop in consistency for smaller models. For example, Llama 3.2 3B dropped from 23.8% success in a single run to 1.4% when requiring success across all runs ($Success^4$), highlighting a lack of reliability.
• Proposal Quality vs. Quantity:
  • Claude 4.5 Sonnet achieved the highest acceptance rate (78.2%) with the lowest proposal rate (12.8%), indicating high confidence and precision.
  • GPT-5 had the highest proposal rate (28.1%) but moderate acceptance, suggesting overconfidence in goal inference.
  • Gemma 3 4B struggled significantly (3.0% success rate) and was often passive or inaccurate.
• Bottlenecks:
  • Information Gathering: Top-performing models performed ~20% more read actions than smaller models, suggesting sufficient observation is a prerequisite for accurate inference.
  • Execution: For smaller models (e.g., Qwen), execution was the primary bottleneck; they could infer goals well enough to get accepted but failed to execute the tasks correctly.
• Robustness: Models varied significantly in handling tool failures and environment noise (spurious notifications). Claude maintained stable success rates under noise, while others degraded.

5. Significance and Impact

• Realistic Evaluation: Pare moves beyond static datasets and flat APIs, providing the first framework to evaluate the full proactive loop (observation $\to$ inference $\to$ proposal $\to$ execution) in a stateful environment.
• Privacy and Autonomy: The framework emphasizes on-device deployment (using smaller models for observation) and human-in-the-loop control (Observe-Execute architecture), addressing privacy concerns and the loss of user autonomy often associated with autonomous agents.
• Future Direction: The results suggest that while frontier models are capable, the gap between inference and execution remains wide. The authors propose future work in asymmetric configurations (small on-device observer + large cloud executor) and incorporating multimodal (visual) inputs to further bridge the gap to real-world deployment.

In summary, Pare establishes a new standard for evaluating proactive agents by simulating the complex, state-dependent reality of mobile user interactions, revealing that current AI agents still struggle significantly with the timing, precision, and reliability required for true proactive assistance.