Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce
This paper introduces a modular two-agent simulation framework for evaluating e-commerce search architectures, demonstrating through 2,011 conversations that rolling-window memory outperforms intent-extraction methods, systematic failure analysis significantly reduces error rates, and different LLM backbones or judges yield distinct performance and evaluation outcomes.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are the manager of a massive, high-tech department store. You want to build a new, super-smart digital shopping assistant that can chat with customers, understand what they want, and help them find the perfect product. But before you hire this assistant and put it in front of real customers, you need to test it.
The problem is, testing on real people is slow, expensive, and risky. If the assistant is bad, real customers get frustrated. If you try to test it by having a computer pretend to be both the customer and the assistant, the test is fake: the "customer" knows exactly what the "assistant" will say, so the conversation feels robotic and unnatural.
This paper presents a solution: A Two-Agent Simulation Lab.
Think of this lab as a realistic "flight simulator" for shopping assistants. It uses two separate AI characters that don't know what the other is thinking:
- The Buyer Agent: A digital customer with a specific personality (e.g., "an impatient tech geek" or "a patient bargain hunter"). This agent has a mission (like "find a specific watch band") and a patience level. It reacts only to what the assistant actually shows it.
- The Responder Agent: The shopping assistant you are testing. It talks to a real, live search engine (like eBay's actual database) to find products and answer questions.
Because the "Buyer" and the "Responder" are separate, you can swap out the "Responder" (try a new design) while keeping the "Buyer" exactly the same. This lets you see if your new design is actually better, without the noise of changing customers.
The researchers ran 2,011 conversations with 14 different types of "Buyers" to test four main ideas. Here is what they found, explained simply:
1. Less is More (The Memory Lesson)
They compared two ways the assistant could remember the conversation:
- The "Summarizer" (Sys-A): After every sentence, the assistant stops to ask a smart AI, "What is the customer really trying to say?" and writes a summary.
- The "Scroll-Back" (Sys-B): The assistant just keeps a rolling list of the last few sentences the customer said, like a chat history you can scroll up.
The Result: The "Scroll-Back" method won. It was 35% faster and actually did a better job.
The Analogy: Imagine a waiter who stops after every order to write a formal report on what you might want, rather than just listening to your order. The waiter who just listens and remembers the last few things you said was faster and made fewer mistakes. The "Summarizer" sometimes got the meaning wrong and threw away important details.
2. The "Fix-It" Speed Run
After running the tests, the team looked at the conversations where the assistant failed. They found a specific pattern: the assistant struggled with very picky, impatient customers who wanted exact matches.
- The Fix: They made three small, targeted changes to the assistant's code to handle these specific failures.
- The Result: In just two days, they reduced the number of "near-failures" by 62%.
The Analogy: It's like a mechanic noticing that a car's engine sputters only when it's cold. Instead of rebuilding the whole engine, they just tightened one specific bolt. The car ran perfectly after that.
3. The Brain Matters (The Model Lesson)
They kept the "Scroll-Back" design the same but swapped the "brain" (the underlying AI model) powering the assistant.
- Brain A: A top-tier model called Gemini.
- Brain B: A slightly different top-tier model called Llama.
The Result: Even though the design was identical, the assistant with the Gemini brain was consistently better at being helpful and understanding the customer. The Llama brain was 13% faster, but it gave more generic, robotic answers (like a brochure) instead of specific, useful advice (like a knowledgeable shop clerk).
The Takeaway: You can have a perfect car chassis, but if you put a weak engine in it, the car won't perform well.
4. The "Judge" Problem (The Most Surprising Finding)
To grade the shopping assistants, the researchers used two other super-smart AIs as "Judges" (one from Google, one from Anthropic). They gave both judges the exact same conversation to grade.
The Result: The judges disagreed on 30% of the conversations, sometimes by a huge margin.
- Judge A (Gemini) loved assistants that were polite, explained things well, and had a nice conversation flow.
- Judge B (Claude) only gave high scores if the customer actually bought something or added an item to their cart.
The Analogy: Imagine two movie critics watching the same film. One gives it 5 stars because the acting was beautiful and the story was emotional. The other gives it 1 star because the movie didn't make the audience laugh. Both are "right" based on their own rules, but they are judging different things.
The Conclusion: Choosing which AI you use to grade your system is just as important as the system itself. If you pick the wrong judge, you might think your assistant is great when it's actually failing to sell products.
Summary
This paper built a realistic "flight simulator" for shopping bots. They learned that:
- Simple memory (just remembering the chat) works better than complex summarizing.
- You can fix bad performance very quickly if you look closely at where it fails.
- The "brain" (the AI model) matters just as much as the design.
- Who you ask to grade your work changes the results. If you want a helpful chat, pick one judge; if you want sales, pick another.
The goal of this research isn't to sell a specific product, but to give companies a reliable, cheap, and fast way to test their shopping assistants before they ever talk to a real human.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.