Technical Summary: SentinelBench

Problem Statement

Current AI agent benchmarks predominantly evaluate agents in reactive environments where state changes occur only in response to the agent's direct actions (e.g., tool calls, navigation). However, many real-world tasks require sustained attention rather than continuous action. In these "monitoring" scenarios, progress depends on external events occurring independently of the agent. The default agent behavior of continuous polling or forced action is inefficient for such tasks, leading to wasted resources (tokens, compute) and potentially missed opportunities if the agent fails to recognize when to wait. Existing benchmarks lack the infrastructure to systematically evaluate agents on time-evolving tasks where the environment changes autonomously, creating a gap in measuring the trade-off between responsiveness and resource cost.

Methodology

The SentinelBench Benchmark

SentinelBench is an open-source benchmark designed to evaluate agents on time-evolving monitoring tasks. It consists of 100 tasks distributed across 10 synthetic web environments (e.g., email, calendars, finance, professional networking, entertainment).

• Environment Architecture: Each environment is a lightweight, modular React application served by a Vite server with a FastAPI backend and SQLite database. They feature realistic UIs (e.g., inboxes, feeds, dashboards) and expose a live web interface.
• Synthetic Data & Consistency: The benchmark utilizes a synthetic data generation pipeline to populate environments with 100 user personas and 201 entities (companies, bands, etc.). Content (emails, posts, stock prices) is generated to ensure in-world consistency across all 10 environments.
• Event Simulation: Tasks involve scripted sequences of events played back over time. The simulation lifecycle includes pre-initialization, a "Ready" state, a "Running" state where events occur at scheduled times, and a "Completed" state triggered by agent action or timeout.
• Task Dimensions:
  • Action Requirement: Tasks are categorized as Passive (watch for a condition, minimal action), Active (periodically check and act), or No-Operation (conditions never occur; agent must not falsely conclude).
  • Criterion Type: Tasks are Absolute (e.g., "reach 2,000 stars") or Relative (e.g., "gain 200 more stars"), requiring memory of initial states.
• Evaluation Protocol: The benchmark is framework-agnostic. Agents are invoked via a shell command template. The evaluation harness manages the simulation lifecycle, injecting events and recording metrics. Agents signal completion by submitting a web form.

Baseline Evaluation Setup

The authors evaluated three models paired with two agent tool configurations:

1. Models: GPT-5.4 (low reasoning), Qwen 3.5:9b (local), and GPT-4o (older frontier).
2. Tool Configurations:
   • sleep(time): A standard tool that unconditionally blocks execution for a fixed duration.
   • wait_for(condition, timeout): A purpose-built tool that heuristically detects page changes (via text diffs) and only engages the LLM when changes occur, blocking until a condition is met or a timeout expires.

Metrics

SentinelBench measures:

• Success Rate: Task completion based on database state and agent actions.
• Reaction Time: The delay between the target condition occurring and the agent completing the task.
• Resource Utilization: Input/output token counts and monetary cost.

Key Results

Performance and Cost Trade-offs

• Model Performance: GPT-5.4 achieved the highest success rates (75% with wait_for, 68% with sleep), significantly outperforming GPT-4o and Qwen 3.5:9b.
• Tool Efficiency: The wait_for tool consistently reduced costs compared to sleep. For GPT-5.4, the median cost per task was 5.1x lower with wait_for ($0.23 vs. $1.17).
• Task Duration Impact: When task durations were extended to 40 minutes (via speed_factor), the cost disparity widened. GPT-5.4 agents using sleep cost 9.7x more ($4.65 vs. $0.48) and completed 13 fewer tasks (56% vs. 69%) compared to those using wait_for.
• Reaction Time: Results were mixed. GPT-5.4 using wait_for had slightly slower median reaction times (51.7s vs. 42.3s) than sleep, but for GPT-4o and Qwen, wait_for was significantly faster.
• Failure Modes: Agents using sleep frequently failed "No-Operation" tasks by concluding monitoring too early, whereas wait_for agents were more robust in distinguishing between "waiting" and "giving up."

Key Contributions

1. SentinelBench Benchmark: The first open-source benchmark specifically designed for time-evolving monitoring tasks, featuring 10 synthetic environments and 100 diverse scenarios.
2. Methodological Framework: A flexible evaluation protocol that decouples the agent from the browser framework, allowing for the measurement of responsiveness, cost, and success in non-reactive environments.
3. Empirical Evidence: Demonstration that agent design choices (specifically the choice of waiting strategy) have a dramatic impact on viability. The paper establishes that continuous polling is often suboptimal for monitoring tasks, and that specialized tools like wait_for can drastically reduce costs while maintaining or improving success rates.
4. Baseline Data: Establishment of performance baselines for three distinct models, highlighting the gap between current frontier models and the requirements of efficient long-running monitoring.

Significance and Claims

The paper claims that SentinelBench fills a critical gap in agent evaluation by shifting focus from "how well an agent acts" to "when an agent acts." The authors argue that as agents are deployed for longer-horizon tasks, the ability to wait, monitor, and respond becomes as important as the ability to execute tools.

The results suggest that the "continuous action" paradigm is a liability for monitoring tasks, leading to unnecessary token consumption and potential task failure. By providing a controlled environment to measure the tension between responsiveness and cost, SentinelBench enables the systematic investigation of agent strategies. The paper concludes that future agent development must prioritize efficiency and patience, moving beyond simple reactive loops to architectures capable of handling asynchronous, time-evolving environments. The authors release all code, environments, and data-generation pipelines to facilitate reproducible research in this domain.