Technical Summary: VISTA - A Versatile Interactive User Simulation Toolkit for Agent Evaluation

Problem Statement

Evaluation remains a critical bottleneck in the development of interactive agents deployed in real-world settings such as finance, education, and e-commerce. Current evaluation methodologies rely heavily on static benchmarks that measure performance on isolated tasks. These static approaches fail to capture the dynamic, multi-step nature of agentic behavior and struggle to expose meaningful failure modes, such as poor action selection or tool-use breakdowns that occur during open-ended interactions.

While user-simulation-based evaluation offers a scalable alternative to human testing, existing simulation frameworks suffer from two primary limitations:

1. Lack of Robust Evaluation Metrics: Existing frameworks provide limited mechanisms to evaluate the quality and comprehensiveness of the simulated interactions themselves. It is difficult to determine if a simulator sufficiently explores an agent's capabilities and failure modes, as prior work often focuses solely on interaction fidelity or human-likeness without assessing coverage.
2. Restricted Interaction Paradigms: Most frameworks are restricted to either UI-only actions or API-only actions. UI-only simulators are often error-prone and computationally expensive due to the need to process noisy, long-context web pages. Conversely, API-only simulators are constrained by a limited set of tools, preventing them from modeling the full range of realistic user behaviors required to comprehensively evaluate modern agents.

Methodology

To address these limitations, the authors propose VISTA (Versatile Interactive user Simulation Toolkit for Agent evaluation). The toolkit consists of two core components: a suite of six evaluation metrics and a hybrid user simulator.

1. Hybrid User Simulator

VISTA introduces a simulator capable of performing both UI-based and API-based actions alongside natural language interaction. The simulation process follows an iterative observation-planning-action loop:

• Scenario Generation: Scenarios are constructed using a user profile (demographics, preferences), a user goal (task objective), and relevant domain knowledge (FAQs, catalogs).
• Tool Integration: The simulator utilizes two categories of tools:
  • General Tools: Shared across domains (e.g., web search, knowledge base retrieval).
  • Task-Specific Tools: Mocked APIs tailored to the product agent (e.g., retrieving order info, rescheduling exams). These are designed with structured responses and realistic error messages to decouple the simulator from production systems while preserving interaction dynamics.
• Action Orchestration: An LLM-based router decides whether to invoke an API tool to gather information or perform a UI action (e.g., clicking, typing) based on the current observation (DOM tree) and user goal. This hybrid approach leverages the reliability of APIs for data retrieval and the flexibility of UI interactions for navigation.

2. Evaluation Metrics

VISTA introduces a suite of six metrics to measure simulation quality across four dimensions:

A. Coverage (Measuring Exploration of Agent Capabilities)

• Tool-calling Transition Entropy (TE): Measures the normalized entropy of tool-calling sequences to assess the diversity of routing paths.
• Tool-calling Distribution Entropy (TDE): Measures the global distribution of tool usage to ensure the simulator does not rely on a narrow set of tools.
• Trajectory Distance (TD): Calculates the average pairwise normalized Levenshtein distance between tool-call sequences within the same user goal to detect repetitive trajectories.

B. Realism (Measuring Behavioral Consistency)

• Uses reference-free LLM-as-a-Judge scores to evaluate the consistency of generated user utterances against three components: the user profile, the task goal, and domain knowledge.

C. Cost (Measuring Efficiency)

• Token Cost: Average number of prompt and completion tokens per turn.
• Action Cost: Average number of actions (API calls + UI operations) per turn, serving as a proxy for time cost.

D. Agent Failure Identification

• Assesses the simulator's ability to trigger and identify agent failures. An LLM judge evaluates assistant responses on helpfulness, coherence, verbosity, relevance, and faithfulness, consolidating justifications into a count of unique agent failures.

Key Contributions

1. VISTA Toolkit: A flexible evaluation toolkit based on hybrid user simulation that combines UI and API actions to enable more realistic and comprehensive agent evaluation.
2. Evaluation Metrics Suite: The introduction of six metrics covering realism, capability coverage, cost, and failure identification, providing interpretable signals for simulator quality.
3. Hybrid Simulator Architecture: A novel framework that integrates UI-based navigation with API-based data retrieval, overcoming the limitations of single-paradigm simulators.
4. Empirical Validation: Demonstration of the toolkit's effectiveness in two representative domains: e-commerce shopping and education customer service.

Experimental Results

The authors evaluated VISTA using commercial (gpt-5.4) and open-source (Qwen3.5-27B) models in e-commerce and education settings, comparing the hybrid simulator against a UI-only baseline.

• Coverage and Failure Discovery: The hybrid simulator consistently outperformed the UI-only setup, achieving up to 10% higher coverage on metrics (Transition Entropy, Distribution Entropy, Trajectory Distance) and identifying up to 42% more unique agent failures.
• Realism: The hybrid simulator achieved high realism scores, including 100% goal consistency in the education agent case. Human evaluations confirmed that the hybrid simulator received higher ratings for human-likeness, coherence, and goal consistency, and was 6% more likely to be mistaken for a human than the UI-only baseline.
• Metric Correlation: The proposed automatic metrics showed strong alignment with human judgments (Spearman correlation $\rho = 0.60$ for realism) and downstream evaluation effectiveness, suggesting they serve as efficient proxies for simulator quality.
• Case Studies: Qualitative analysis revealed that the UI-only simulator often failed to detect agent errors because it accumulated excessive context through UI navigation, leading to unrealistic oversharing of information. In contrast, the hybrid simulator used API tools to retrieve precise information, triggering realistic failure modes (e.g., missing buttons, incorrect order handling) that the UI-only simulator missed.

Significance and Claims

The paper claims that VISTA addresses the "Sim2Real" gap in agent evaluation by providing a more flexible and comprehensive framework than existing methods. By combining UI and API interactions, the toolkit enables the discovery of a broader range of agent failure modes and more effectively exercises agent capabilities across diverse workflows.

The authors assert that their proposed metrics provide meaningful, interpretable signals regarding simulator quality without requiring expensive human annotation or complex LLM-judge analysis for every metric. They conclude that flexible hybrid simulation is essential for the rigorous evaluation of modern interactive agents, particularly as these systems are deployed in complex, multi-step real-world environments.

Limitations

The authors acknowledge several limitations:

• Behavioral Bias: Like other LLM-based simulators, VISTA may not perfectly model real human behavior and could exhibit biases (e.g., being overly cooperative) inherited from the underlying LLMs.
• Domain Scope: The evaluation is currently limited to e-commerce and education. Generalizability to other domains like healthcare or enterprise productivity requires further validation.
• Deployment Correlation: The proposed metrics measure coverage and interaction quality but do not directly estimate real-world deployment success.
• System Complexity: The hybrid approach introduces additional implementation overhead due to the need for routing logic and environment-specific integrations between UI and API actions.