Designing Reliable LLM-as-a-judge Measurement Systems for Multi-turn Business Agents
This paper presents a practical, maintainable framework for evaluating multi-turn business agents that integrates modular LLM judges, intent-preserving user simulation, and human-in-the-loop governance to ensure reliable, actionable measurement of agent performance and business knowledge gaps.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital economy, businesses increasingly rely on artificial intelligence to handle complex conversations with customers. These are not simple, one-off exchanges where a bot answers a single question and disappears. Instead, they are multi-turn dialogues where a customer might ask about a product, request a price check, verify an address, and then complain about a shipping delay, all in a single thread. To ensure these digital assistants work correctly, companies need a way to grade their performance. For years, researchers have used a method called "LLM-as-a-judge," where one artificial intelligence model evaluates the answers given by another. This approach has become standard for testing simple tasks, but it struggles when the conversation is long, the rules change from one company to another, and the correct answer depends on specific facts that the AI might not know. The core challenge is that a conversation can fail for two very different reasons: the AI might be too confused to understand the customer, or the business simply failed to provide the AI with the necessary information to do its job. Distinguishing between these two failures is critical, because fixing a confused AI requires a different solution than asking a business to update its data.
A team of researchers at Meta has developed a new, comprehensive system designed to solve this specific problem. Rather than treating the evaluation of these agents as a simple prompt-writing exercise, they built a full measurement system that mimics the complexity of the real world. Their approach combines four distinct parts: a clear definition of what success looks like for a specific business, a set of specialized AI judges that check different aspects of the conversation, a simulated user that can interact with the agent to test its limits, and a human review process that constantly checks the accuracy of the whole system. The researchers found that by connecting these parts into a single, self-correcting loop, they could reliably measure whether an agent failed because of its own limitations or because of missing business data. Over time, this system allowed both the human reviewers and the automated judges to improve together, creating a more accurate picture of performance than either could achieve alone.
The researchers began by acknowledging that there is no single "correct" answer for every conversation. In a standard test, a model might be graded against a fixed list of right and wrong answers. But in a real business setting, a statement like "we ship to that address in three days" is only correct if the company actually has that shipping policy. If the company's data is missing, the AI cannot know the answer, and calling it a failure would be unfair. The team realized that to measure these agents fairly, they first had to define the goal of the conversation clearly. They created a specification that breaks every interaction down into specific end states. Instead of just saying "good" or "bad," the system categorizes the outcome into seven distinct possibilities. These range from a full resolution where the customer's problem is solved, to partial resolutions, or cases where the conversation was invalid but handled well. Crucially, the system also separates failures caused by the AI's inability to understand (a model gap) from failures caused by missing business facts (a knowledge gap). This distinction is vital because it tells the business exactly who needs to fix the problem: the engineers building the AI or the business team updating their data.
To test these agents without needing thousands of real customers to try them out, the researchers built a simulated user. This is an AI program designed to act like a human customer, but with strict rules to ensure it stays true to the original task. The simulator has two modes. In one mode, it replays historical conversations, taking the exact intents and facts from a real customer's chat and feeding them to the new agent to see how it reacts. In the other mode, it follows a set of instructions to create new scenarios, such as a customer who is confused or a spammer trying to trick the system. The key to this simulator is that it is audited to ensure it does not accidentally change the customer's original goal. If the simulator invents a new request that the real customer never made, it would give a false reading of the agent's performance. The researchers verified that this simulated user could preserve the original task while adapting to the agent's changing responses, making it a reliable tool for pre-launch testing.
Once the conversations were generated, they were evaluated by a team of specialized AI judges. Instead of relying on one giant AI to give a single score, the system uses many small, focused judges. One judge checks if the AI made up facts, another checks if it followed the company's specific procedures, and a third checks if it handled the customer's intent correctly. These individual checks are then combined using a clear set of rules to form a final verdict. This structure allows the system to be transparent; if a conversation is marked as a failure, the system can point to the specific piece of evidence that caused it, such as a missing price list or a misunderstood instruction. The researchers also built in a layer of human oversight. A human reviewer looks at a sample of the conversations independently, without seeing what the AI judges decided. If the human and the AI disagree, a senior expert steps in to make the final call. This process does not just produce a label; it creates a feedback loop. The disagreements are analyzed to see if the human made a mistake, if the guidelines were unclear, or if the AI judge needs to be retrained.
The results of this system, tested over two and a half years on real production agents, showed a clear pattern of improvement. Initially, the automated judges were not much better than random chance at identifying specific types of failures. However, as the system ran through repeated cycles of auditing and adjustment, the accuracy of the automated judges rose significantly. In one dimension, the system's ability to detect missing business knowledge improved from a low score to a high level of accuracy, eventually surpassing the performance of the human reviewers on that specific task. Conversely, on other dimensions, the human reviewers remained slightly more accurate, but the gap between the two narrowed over time. The most important finding was that the combined workflow—where human and machine worked together in a shared feedback loop—produced the most reliable results. The system did not replace human judgment; instead, it created a partnership where the machine handled the heavy lifting of checking facts and the humans focused on the complex nuances, with both sides learning from the other's errors.
The researchers were careful to note that these findings are based on observational data from their own production environment, not a controlled experiment against a different method. They did not claim that their system is universally perfect or that it will work exactly the same way for every business. The human reviewers themselves were not infallible, and the system's performance eventually plateaued, meaning it stopped improving after a certain point. However, the study demonstrated that treating AI evaluation as a maintained measurement system, rather than a one-time test, allows for continuous refinement. By defining clear goals, using specialized judges, simulating users carefully, and keeping humans in the loop to correct errors, companies can build a reliable way to measure their digital agents. This approach ensures that when a conversation fails, the business knows exactly why, and can take the right action to fix it, whether that means updating a database or retraining a model.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.