SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation

The paper proposes SAGE, a novel user simulation framework for evaluating multi-turn agents that integrates top-down business logic and bottom-up infrastructure knowledge to generate realistic, diverse interactions capable of identifying significantly more agent errors than existing methods.

The Gist

Imagine you are building a new, high-tech robot barista. Before you open your shop to the real world, you need to test it. You want to make sure it can handle rude customers, complex orders, and weird questions without crashing or spilling coffee everywhere.

In the past, developers had two main ways to test their agents (AI chatbots):

1. Hire real humans: Expensive, slow, and hard to scale.
2. Use a generic computer program: Fast, but these programs act like "robots pretending to be robots." They ask boring, repetitive questions and don't behave like real people with real needs.

Enter SAGE.

The paper introduces SAGE (a Top-Down Bottom-Up Knowledge-Grounded User Simulator). Think of SAGE not as a generic robot, but as a Master Method Actor for testing AI. It doesn't just "act" like a human; it acts like a specific human with a specific job, a specific budget, and a specific reason for being there.

Here is how SAGE works, broken down into simple metaphors:

1. The "Top-Down" Approach: The Casting Director

Imagine you are casting a play. You don't just say, "We need a customer." You say, "We need a 35-year-old project manager named Sarah, who is stressed about her budget, loves efficiency, and is looking for a cleaning robot for her office."

• The Metaphor: This is the Top-Down part. SAGE uses "Ideal Customer Profiles" (ICPs). It builds a detailed character sheet for the simulator based on real business logic.
• Why it matters: A generic simulator might ask, "How much does this cost?" A SAGE simulator, playing "Sarah the Project Manager," asks, "Does this robot fit within my $10,000 Q3 budget, and can it handle the high-traffic hallway in my office?" This forces the AI to answer specific, realistic questions.

2. The "Bottom-Up" Approach: The Scriptwriter

Now, imagine Sarah the Project Manager walks into the store. She has already read the company's website, looked at the product manual, and knows about the "50% off" sale mentioned in the FAQ.

• The Metaphor: This is the Bottom-Up part. SAGE feeds the simulator the actual "script" of the business: the product catalogs, the FAQs, the technical specs, and the knowledge base.
• Why it matters: If the AI says, "Our robot can fly," but the product manual says "It stays on the ground," a generic simulator might just nod along. SAGE, having read the manual, will immediately say, "Wait, your website says it's ground-only. Why are you telling me it flies?" This catches hallucinations (lies) and errors.

3. The "Method Acting" Session

When SAGE talks to the AI agent, it's not a robot talking to a robot. It's a fully realized character interacting with the AI.

• It might be grumpy because it's late.
• It might be confused because it read two different webpages.
• It might ask a follow-up question based on a specific detail it "remembered" from the product catalog.

The Result: Finding the Bugs

The paper tested SAGE against other simulators and found that:

• It's more realistic: The conversations sounded like real humans, not robots.
• It's a better bug hunter: SAGE found 33% more errors than other methods.

Why? Because SAGE is the only one asking the hard, specific questions that real customers ask.

• Generic Simulator: "Is this good?" -> AI: "Yes." -> Pass.
• SAGE Simulator: "I need this for a farm in a dusty environment. Your manual says it's for indoor cafes. Does it have a dust seal?" -> AI: "Uh, yes, it's great for farms." -> FAIL. (The AI is lying; SAGE caught it).

The Bottom Line

SAGE is like a stress-test simulator for AI. Instead of just checking if the AI can say "Hello," it puts the AI in a complex, realistic scenario where it has to juggle a specific customer's personality, a specific budget, and a specific set of facts.

By combining who the customer is (Top-Down) with what the customer knows (Bottom-Up), SAGE creates a "digital twin" of a real customer. This helps companies find and fix their AI's mistakes before they ever annoy a real human, saving time, money, and reputation.

Technical Summary

Here is a detailed technical summary of the paper "SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation."

1. Problem Statement

Evaluating multi-turn interactive agents (e.g., customer support bots, sales agents) is a critical bottleneck in their development.

• Limitations of Human Evaluation: Traditional reliance on human assessors is costly, difficult to scale, and slow for iterative development cycles.
• Limitations of Existing Simulators: Current automated user simulators often rely on:
  • Static Benchmarks: Narrow, task-specific datasets that fail to capture the open-ended, dynamic nature of real-world interactions.
  • Generic Personas: Models that treat users as domain-general interlocutors, ignoring specific business contexts, user roles, and information needs.
  • Data Scarcity: Many approaches require large volumes of proprietary human interaction data, which is often unavailable or expensive to obtain.

Consequently, existing simulators produce interactions that lack realism and fail to rigorously stress-test agents against specific business logic or knowledge bases, leading to missed bugs and poor evaluation fidelity.

2. Methodology: The SAGE Framework

The authors propose SAGE (Simulation framework for multi-turn AGent Evaluation), a knowledge-grounded user simulator that integrates two distinct types of knowledge to generate realistic, diverse, and context-aware interactions.

A. Knowledge Integration Strategy

SAGE combines Top-Down and Bottom-Up knowledge sources:

1. Top-Down Knowledge (Business Logic):
   • Ideal Customer Profiles (ICPs): Defines user attributes based on business goals (e.g., firmographics, decision-making authority, budget, deal stage).
   • Individual Differences: Incorporates psychological traits (Big Five personality framework) and demographic factors to simulate varied human behaviors.
   • Purpose: Ensures the simulated user behaves like a specific type of customer relevant to the business (e.g., a B2B procurement officer vs. a B2C bargain hunter).
2. Bottom-Up Knowledge (Agent Infrastructure):
   • Source Material: Structured and unstructured data from the company's agent infrastructure, including product catalogs, FAQs, knowledge bases, and technical documentation.
   • Purpose: Grounds the user's information needs in the actual content the agent is expected to retrieve. This allows the simulator to ask specific, fact-based questions that test the agent's retrieval capabilities and faithfulness.

B. The SAGE Pipeline

The framework operates through four sequential processes:

1. Configure Knowledge: The system ingests user attributes (ICPs + demographics), agent capabilities (functionalities to test), and infrastructure knowledge (product data/FAQs).
2. Build Task Scenarios:
   • Goal Selection: Iterates through agent capabilities to define user goals (e.g., "compare Product A and B").
   • Knowledge Selection: Uses an LLM and clustering algorithms to select relevant infrastructure knowledge pieces aligned with the goal.
   • Profile Construction: Generates a coherent user profile by combining the selected infrastructure context with the top-down attributes (e.g., ensuring a user's budget aligns with the products they are inquiring about).
3. Generate Interactions:
   • The user simulator (an LLM) interacts with the target agent in a shared environment.
   • The simulator is explicitly grounded in the infrastructure knowledge via its system prompt, while the agent accesses it via its internal framework.
   • Interactions continue until a stop token is emitted or a turn budget is reached.
4. Evaluate Agent and Identify Bugs:
   • Scoring: An LLM-as-a-Judge evaluates each agent turn on five dimensions: Helpfulness, Coherence, Verbosity, Relevance, and Faithfulness (0–4 scale).
   • Bug Extraction: Turns scoring below a threshold (e.g., <2) are flagged. A two-stage LLM process summarizes these low-scoring turns into high-level failure categories and then into fine-grained, unique bug statements (e.g., "Failed to provide weight capacity for Product A").

3. Key Contributions

• Novel Framework: Introduction of SAGE, the first user simulator to systematically integrate both top-down business logic (ICPs) and bottom-up agent infrastructure knowledge.
• Improved Realism & Diversity: Demonstrated that combining these knowledge sources produces interactions that are significantly more human-like, diverse, and contextually grounded than existing methods.
• Enhanced Bug Discovery: Proved that SAGE identifies 25–33% more agent errors compared to baseline simulators and human interactions, making it a superior tool for iterative agent improvement.
• Scalability: Offers a scalable alternative to human evaluation that does not require historical interaction data, making it suitable for "cold-start" agent evaluation.

4. Experimental Results

The authors evaluated SAGE on two use cases: a RAG-based Sales Agent (robotics company) and a Tool-Augmented Shopping Agent (Shopify platform). They compared SAGE against two baselines:

• w/o ICP: Uses bottom-up knowledge but lacks top-down business logic (generic personas).
• w/o Agent-Infra: Uses top-down logic but lacks specific product/infrastructure knowledge.

Key Findings:

• Lexical Diversity: SAGE achieved the highest scores across metrics (MTLD, HD-D, Vocab Size), producing interactions closest to real human diversity.
• Human Fidelity: In human evaluations (rated on Likert scales), SAGE interactions scored significantly higher in Human-likeness and Coherence compared to baselines.
• Error Identification:
  • SAGE identified the highest number of unique bugs (e.g., 38.6 vs. 29.0 for w/o ICP in the RAG case).
  • Removing bottom-up knowledge caused the sharpest drop in bug detection, highlighting the importance of grounding users in actual product data.
  • SAGE outperformed even real human interactions in finding bugs, suggesting simulated users can be more effective at "stress-testing" agents than typical human users who may not probe deep enough.
• Case Studies: Analysis showed that without ICPs, simulators produced generic/repetitive queries. Without infrastructure knowledge, simulators asked irrelevant questions (e.g., asking about farming capabilities for a restaurant robot). SAGE successfully combined both to trigger specific faithfulness errors (e.g., incorrect charging times or usage environments).

5. Significance and Impact

• Practical Utility: SAGE provides a robust, automated solution for the "evaluation bottleneck" in agent development, allowing companies to iterate faster and deploy more reliable agents.
• Domain Adaptability: By decoupling the simulation logic from specific training data and relying on configurable business knowledge, SAGE can be easily adapted to various industries (B2B, B2C, enterprise) without needing massive datasets.
• Quality Assurance: The framework shifts the paradigm from "can the agent chat?" to "can the agent accurately retrieve and apply business-specific knowledge under realistic user pressure?"
• Limitations & Future Work: The authors acknowledge potential biases in LLM-generated personas and the current limitation to single-goal, single-knowledge scenarios. Future work aims to incorporate multi-turn topic switching and real-world log validation once data becomes available.

In conclusion, SAGE represents a significant advancement in agent evaluation by moving beyond generic simulation to knowledge-grounded, business-logic-driven user modeling, resulting in more rigorous testing and higher-quality agent deployment.