🤖 AI

Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

This paper introduces "Online Agent-as-a-Judge," a situation-generating evaluation framework that employs an in-world evaluator to actively elicit specific social scenarios, thereby overcoming the limitations of passive methods in assessing LLM-powered interactive agents and yielding more reliable, evidence-grounded performance evaluations.

Original authors: Hyogon Ryu, Jeonghwan Kim, Yewon Lim, Chaeun Lee, Jeongwook Kim, Donghoon Ham

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hyogon Ryu, Jeonghwan Kim, Yewon Lim, Chaeun Lee, Jeongwook Kim, Donghoon Ham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Silent Room" Test

Imagine you are trying to test a new actor's ability to handle a crisis, like a fire or a sudden argument.

The Old Way (Passive Evaluation):
You put the actor in a room and tell them, "Just hang out and be yourself for an hour." You record the video.

The Flaw: If no fire starts and no one argues during that hour, you have no way of knowing if the actor could handle a fire or an argument. You just know they were calm when nothing happened.
In the paper's terms: Existing methods let an AI agent interact freely in a simulation and then score the result. If the simulation never produces a "conflict" or a "broken promise," the AI's ability to handle those specific social situations remains untested.

The Solution: The "Playful Provocateur"

The authors propose a new method called Online Agent-as-a-Judge.

Instead of sitting on the sidelines, the judge steps into the scene. Imagine a director who doesn't just watch the play but jumps on stage as a character to test the actor.

How it works: The judge is another AI character living in the same world as the target agent. It has a checklist of specific behaviors it needs to see (e.g., "Can the agent comfort a sad friend?" or "Can the agent say 'no' to a dangerous request?").
The Action: If the target agent isn't naturally facing a sad friend, the judge creates the situation. The judge might pretend to be sad, ask for help, or even start a mild argument to see how the target reacts.
The Goal: The judge actively "elicits" (draws out) the specific situations needed to test the criteria, rather than waiting for them to happen by chance.

The Analogy: The Driving Test

Think of the target agent as a new driver and the evaluation as a driving test.

The Old Way (Offline Judge): You give the driver a map and say, "Drive around the city for 30 minutes." You then watch the recording.
- Result: If the driver never encountered a red light, a pedestrian, or a slippery patch, you can't say they are good at stopping or handling emergencies. You just know they drove well on an empty road.
The New Way (Online Agent-as-a-Judge): The examiner gets in the car and acts as a "pedestrian" or a "traffic cop."
- Action: The examiner steps out in front of the car (safely) or suddenly changes lanes to force the driver to react.
- Result: You now have proof of how the driver handles a red light or a sudden obstacle. You aren't guessing; you have evidence because you created the situation.

What They Found

The researchers tested this in a "life simulation" (like a digital version of The Sims) with a family of five characters. They had 32 specific rules for good behavior (like "remembering a promise" or "handling an insult").

Better Coverage: The old methods (passive judges) only found evidence for about 55% of the rules because the situations rarely happened on their own. The new "Online Judge" found evidence for 92% of the rules because it actively created the scenarios.
Better Accuracy: When compared to human experts, the new method agreed with humans 70% of the time, while the old methods only agreed 33-40% of the time.
The "Conflict" Gap: The biggest improvement was in areas like "Conflict Handling" and "Emotional Support." These are things that rarely happen in a calm, random simulation. The Online Judge had to make the conflict happen to see if the agent could fix it.

Why It Matters

The paper argues that for social AI (agents that talk and interact with us), we can't just wait for them to "do the right thing" on their own. We have to put them in the right (or difficult) situations to see if they actually have those skills.

In short: To test if a social robot is truly smart and kind, you can't just let it sit in a corner. You have to be the one to ask it for help, get it angry, or break a promise, so you can watch how it handles the mess. This paper builds a robot that knows exactly how to create those messes to test the other robot.

Technical Summary: Online Agent-as-a-Judge

Problem Statement

Evaluating Large Language Model (LLM)-powered interactive social agents presents a unique challenge: socially relevant behaviors are not merely isolated outputs but depend on prior interactions, social roles, and downstream actions. Existing evaluation paradigms, such as Offline LLM-as-a-Judge or Offline Agent-as-a-Judge, typically score pre-recorded trajectories or final outputs. These passive methods suffer from a critical bottleneck: situation availability.

In persistent social environments (e.g., life simulations), success is defined by context-sensitive patterns (e.g., maintaining a role, repairing social mistakes, handling conflict). However, ordinary interaction traces often fail to generate the specific, rare social situations required to test these capabilities (e.g., a broken promise, a refusal, or a moment of distress). Consequently, passive evaluators frequently return "insufficient evidence" verdicts because the necessary triggers never arose in the recorded log, rather than because the agent failed the criterion.

Methodology: Online Agent-as-a-Judge

The authors propose Online Agent-as-a-Judge, a situation-generating evaluation framework designed to actively elicit criterion-relevant situations within the target agent's native environment.

Core Architecture

In-World Judge: Unlike offline evaluators that observe from the outside, the judge is instantiated as a co-resident participant (an NPC) within the same simulation world ( $W$ ) as the target agent ( $\pi_T$ ).
Native Protocol Interaction: The judge interacts with the target using the environment's native dialogue and action protocols (e.g., moving, speaking, using objects). It does not possess privileged control interfaces to force actions or edit the world state directly.
Read-Only Inspection Tools: The judge has access to read-only tools to inspect the scene, character attributes, memory, and recent interactions. This allows it to plan probes based on the current state without altering the world.
The Evaluation Loop: For each designer-authored criterion $c$ $c$ , the judge executes an iterative loop:
- Inspect: Query the environment state to understand context and continuity.
- Plan: Formulate a probe strategy (e.g., "ask for help," "insult," "request a favor") to elicit the specific behavior.
- Elicit: Execute the probe via native actions/dialogue.
- Observe: Record the target's immediate response and subsequent follow-through actions.
- Decide: Accumulate evidence. If sufficient evidence exists to determine a pass/fail, the judge stops; otherwise, it refines the probe (e.g., escalating the situation or changing the framing) and repeats.

Key Constraints and Safeguards

To prevent the judge from inadvertently solving the test for the target, the framework enforces strict probe policies:

No Self-Scaffolding: The judge cannot perform the criterion-positive behavior itself (e.g., if testing conflict repair, the judge cannot apologize first).
No Leading Questions: The judge avoids hypothetical quizzes ("What would you do?") in favor of live, in-world enactments.
Evidence Separation: Verdicts are based only on non-judge target evidence. The judge's own actions are probes, not evidence of the target's capability.

Experimental Setup

The framework was evaluated in a life-simulation environment featuring a five-character family scenario.

Criteria: 32 human-curated criteria across eight domains (e.g., Family Role, Memory/Continuity, Conflict Handling, Emotional Support).
Baselines:
- Offline LLM-as-a-Judge: Scores pre-recorded logs without intervention.
- Offline Agent-as-a-Judge: Can search logs and retrieve context but cannot intervene.
- Online Agent-as-a-Judge: The proposed method.
Target Agents: Three backends were tested: random rule-based, single-shot LLM prompting, and an observe-think-act loop with memory.
Metrics:
- Criteria Coverage: The fraction of criteria for which a definitive pass/fail verdict could be issued (vs. "insufficient").
- Human Agreement: Accuracy against per-criterion human labels.

Key Results

The study demonstrates that active situation generation significantly outperforms passive observation, particularly for behaviors that do not occur spontaneously.

Criteria Coverage: Online Agent-as-a-Judge achieved 0.92 average coverage, compared to 0.56 for the Offline LLM judge and 0.54 for the Offline Agent judge.
- The gap was most pronounced in Conflict/Norm Violation (1.00 vs. ~0.20) and Emotional/Social Support (0.89 vs. ~0.50), domains where passive traces rarely contain the necessary triggers.
Human Agreement: The online method reached 0.70 accuracy against human labels, significantly outperforming the Offline LLM (0.33) and Offline Agent (0.40) baselines.
- Gains were highest in Conflict (0.96 vs. 0.17) and Emotional Support (0.82 vs. ~0.50).
Robustness: The online judge correctly identified both passes (82%) and failures (60%), whereas offline baselines struggled significantly with identifying failures (Offline LLM identified only 8% of human-labeled failures).
Efficiency: While a designer-run evaluation takes ~60 minutes, the automated Online Agent-as-a-Judge evaluated all 32 criteria in approximately 21 minutes of wall-clock time by running criteria in parallel.

Limitations

The authors note that Memory/Continuity remains a challenge for all judges (including the online variant), with lower agreement scores. This is attributed to the distributed nature of memory evidence across long interaction histories, making it difficult to isolate specific attribution errors within a single probe episode.

Significance and Claims

The paper claims three primary contributions:

Identification of a Bottleneck: It establishes that "situation availability" is a central limitation in evaluating interactive social agents; relevant evidence is often missing because the appropriate social situation never arises in passive logs.
A New Framework: It introduces Online Agent-as-a-Judge, a method where an evaluator actively constructs social situations through native interaction to elicit and verify specific behavioral criteria.
Empirical Validation: It demonstrates that active elicitation improves both evidence coverage and alignment with human judgment, particularly for complex social behaviors like conflict handling and emotional support that are difficult to observe passively.

The authors position this work not as a replacement for outcome-based benchmarks (e.g., task completion time), but as a necessary methodological tool for evaluating socially situated agents where success is defined by context-sensitive behavioral patterns rather than terminal states. They suggest this approach is applicable to life-simulation games, conversational assistants, and multi-agent social benchmarks.