Technical Summary: When Is an Agent Evaluation Over?

Problem Statement

Current agent evaluation frameworks typically score models based on the state visible at the moment a run is stopped (the "endpoint"). This approach assumes that the endpoint simultaneously establishes two critical conditions: outcome finality (the result is settled and cannot change) and cross-unit separation (the current run is independent of previous or future runs).

The paper argues that these two conditions are independent and often unmet at the endpoint.

1. Outcome Finality: A run may stop while asynchronous operations (e.g., delayed writes, background processes) are still pending. Scoring the state before these operations complete can lead to incorrect labels (e.g., marking a task as failed when a delayed write would have succeeded, or vice versa).
2. Cross-Unit Separation: If the environment retains state between runs (e.g., shared databases, persistent accounts, or uncleaned artifacts), a previous run can alter the starting conditions or outcome of a subsequent run. This violates the assumption that trials are independent and identically distributed (i.i.d.), rendering aggregated metrics (like $pass@k$) invalid.

Existing audits often check for benchmark flaws or reset mechanisms but fail to distinguish between a scored outcome that is merely "unfinished" versus one where the connection between runs is unaccounted for.

Methodology

The author employs a three-pronged approach to investigate these issues:

1. Theoretical Framework (The Completion Argument):
   The paper develops a logical framework distinguishing between the endpoint (when interaction stops) and completion (when the outcome is settled and boundaries are secure). It defines the evidence required to justify a final label versus treating runs as separate trials.

2. Controlled Replay Experiment:
   Using AgentDojo 0.1.35, the author constructed a system to isolate boundary choices.

   • Setup: A standalone runner replayed fixed tool calls and operation schedules. A local HTTP service simulated asynchronous delays (0, 25, 100, 250 ms) and state persistence.
   • Variables: The study varied scoring timing (snapshot at endpoint vs. waiting for terminal state) and state management (shared state vs. namespaced state vs. verified reset).
   • Metrics: The experiment measured the disagreement between endpoint labels and terminal labels (finality) and the frequency of cross-run exposure (separation).

3. Documentation Review:
   The author reviewed the public documentation and papers for ten prominent agent benchmarks: WebArena, WorkArena, OSWorld, SWE-bench, tau-bench, ToolSandbox, TheAgentCompany, RE-Bench, Cybench, and AgentCanary.

   • Criteria: They coded for explicit statements regarding run definitions, stop rules, state persistence, reset mechanisms, and evidence for treating runs as separate trials.
   • Limitations: The review focused on what was explicitly reported, not on inferring unstated properties.

Key Contributions

1. Conceptual Distinction: Outcome Finality vs. Cross-Unit Separation

The paper establishes that these are distinct requirements requiring different evidence:

• Outcome Finality: Requires that every relevant operation or event that could change the outcome is resolved, bounded, or confirmed as cancelled.
• Cross-Unit Separation: Requires that no relevant route (shared state, credentials, artifacts) allows one run to influence another's conditions or outcome.
• Implication: One can achieve finality without separation (e.g., waiting for a write to finish, but leaving the file accessible to the next run) and separation without finality (e.g., isolating runs, but scoring before a delayed operation completes).

2. The Completion Argument

The author proposes a decision framework for evaluators:

• For Final Labels: A success/failure label is only justified if all routes that could change the outcome are blocked, followed to completion, or tightly bounded. Otherwise, the outcome must be reported as unresolved.
• For Separate Trials: Runs can only be counted as separate analysis units if all routes between them are blocked or proven unable to affect the outcome. If a connection remains, the runs must be modeled as a connected unit or grouped.

3. The Open-Effects Record

The paper proposes a new reporting standard: an open-effects record. This record should list operations or resources that remain relevant after the endpoint, their current status, and whether they could change the scored outcome or affect another run.

Experimental Results

Controlled Replay Findings

• Finality: At non-zero delays, endpoint labels disagreed with terminal labels in 100% of cases (150/150 trials). Snapshot scoring recorded 50 successes, whereas reconciliation (waiting for completion) recorded 200 successes. Verified cancellation correctly identified pending writes as failures.
• Separation: Under shared state, 75% of pairs (150/200) showed exposure where Run A changed Run B's outcome. This exposure was eliminated (0/200) under namespaced state, verified reset, or when Run B ran before Run A.
• Conclusion: The endpoint alone cannot justify the final label or the analysis unit. The scoring time and state management policy directly determine the validity of the result.

Documentation Review Findings

• Reset/Retention: Explicitly documented in 8/10 protocols; partial in 2/10.
• Unfinished Operations: Reported much less consistently. 6/10 protocols exposed shells, browsers, or services without stating whether descendant processes or delayed effects are finished, cancelled, or checked before scoring.
• Evidence for Separation: Only 3/10 protocols provided explicit evidence for treating runs as separate observations. Seven described reset procedures but failed to fully state the scope of resources covered or how successful restoration was verified.
• Gap: No protocol consistently documented the time periods over which relevant effects could change the outcome.

Significance and Claims

The paper claims that current evaluation practices often conflate the end of interaction with the end of the task's causal chain. Its significance lies in:

1. Correcting Metric Validity: It demonstrates that without verifying finality and separation, aggregated metrics (like pass rates) may measure a mixture of task performance and environmental artifacts.
2. Refining Evaluation Boundaries: It argues that the "evaluation boundary" is not a single moment but a set of decisions regarding when to stop, when to score, and how to separate runs.
3. Proposing a Reporting Standard: By introducing the "open-effects record," the paper provides a concrete mechanism for evaluators to transparently report unresolved states and persistent resources, allowing readers to assess the validity of the claimed results.

The author maintains a modest stance, noting that their documentation review is limited to ten protocols and that their experimental counts reflect constructed conditions rather than the frequency of these issues in the broader landscape of published benchmarks. The core argument is that a final label is justified only when anything that could still change the claimed outcome is resolved, bounded, or retained as uncertainty.