Technical Summary: STT-Arena

Problem Statement

Large Language Model (LLM) agents are increasingly deployed in real-world commercial applications such as airline reservations and clinical consultations. While existing dynamic benchmarks (e.g., GAIA-2, Real-Time Reasoning Gym) focus on measuring an agent's ability to detect temporal changes and respond rapidly, they largely overlook the complementary challenge of adaptive replanning. In realistic scenarios, mid-task disruptions—driven by spatio-temporal dynamics—can invalidate prior decisions, requiring the agent to abandon a failed plan and reconstruct an alternative multi-step strategy. Current benchmarks fail to systematically evaluate this capability, leaving a gap in understanding how LLMs handle environments where state changes are abrupt, coupled across time and space, and capable of rendering ongoing plans obsolete.

Methodology

STT-Arena Benchmark Construction

The authors introduce STT-Arena (Spatio-Temporal Tool-Use Arena), a benchmark comprising 227 high-quality interactive tasks. The construction follows a three-stage pipeline:

1. Environment Curation: Static, executable environments are synthesized from real-world user queries (filtered for statefulness and spatio-temporal sensitivity) using LLMs to infer entity attributes and tool specifications. These are implemented as Python classes and validated for functional correctness.
2. Spatio-Temporal Dynamic Injection: Controlled conflicts are injected into static environments based on a taxonomy of nine conflict types categorized into Temporal (T1–T3), Spatial (S1–S3), and Spatio-Temporal (ST1–ST3) dynamics. A "blueprint" defines the user goal, nominal tool sequence, trigger conditions, state mutations, and recovery paths. This stage creates tasks with varying solvability levels: Easy, Medium, Hard, and Impossible.
3. Dual-Agent Assessment: A planning agent generates a conflict-free trajectory, while a checking agent verifies that the injected triggers fire correctly, mutate the state as intended, and successfully disrupt the original plan. Human-in-the-loop review ensures consistency and realism.

Evaluation Protocol

Tasks are evaluated by interacting the LLM with a passive user simulator. Success is measured via Pass@1, calculated as a weighted average across difficulty levels. For solvable tasks, a checklist of criteria verifies the final state. For impossible tasks, the model must correctly recognize infeasibility and communicate it to the user.

STT-Agent Training

To address identified failure modes, the authors propose STT-Agent, a 4B-parameter model trained using:

• Iterative Trajectory Refinement: A post-processing technique that reorders, deletes, or modifies tool-call blocks in training data to eliminate inefficient patterns (e.g., blind retries, missing verification).
• Online Reinforcement Learning (RL): Fine-tuning on verifiable dynamic tasks using REINFORCE++ to optimize for successful adaptation.

Key Contributions

1. A Fine-Grained Spatio-Temporal Taxonomy: The benchmark categorizes environmental dynamics into nine subtypes (e.g., Window Expiry, Site Mismatch, Failure Cascade) that explicitly capture temporal evolution, spatial dependency, and their coupled interactions.
2. Scalable Interactive Infrastructure: A reproducible framework for generating dynamic environments where triggers can abruptly invalidate plans, forcing agents to detect state shifts and replan.
3. Comprehensive Benchmarking and Training Paradigm: Extensive evaluation of frontier LLMs reveals significant performance gaps. The authors introduce an iterative trajectory refinement technique combined with online RL to produce STT-Agent-4B, which outperforms many larger open-source models despite its compact size.

Results

• Fundamental Difficulty: Even the strongest proprietary models, including Claude-4.6-Opus, achieve less than 40% overall accuracy (35.39%) on STT-Arena. Performance degrades significantly as task difficulty increases from Easy to Hard.
• Model Scale vs. Training: Open-source models generally trail closed-source leaders. Efficient models (e.g., Llama-3.1-8B) perform substantially worse, highlighting the critical role of model scale. However, STT-Agent-4B (27.17% Pass@1) outperforms several larger frontier models (e.g., GLM-5.1 at 27.31%, Deepseek-V3.2 at 32.16% in some metrics but lower overall), suggesting that targeted training on spatio-temporal failure patterns is highly effective.
• Failure Modes: Systematic analysis identifies three recurring error patterns in existing models:
  1. Stale-State Execution: Continuing to act on outdated world states after changes occur.
  2. Misdiagnosis of Dynamic Triggers: Interpreting regulatory or structural blocks as parameter errors or transient glitches.
  3. Missing Post-Adaptation Verification: Failing to verify that the final global state satisfies all constraints after an adaptation step.
• Test-Time Scaling: Increasing the number of attempts (Pass@k) improves performance but yields diminishing returns, indicating that sampling alone cannot overcome the fundamental reasoning challenges.

Significance

The paper argues that the ability to replan and reconsider under mid-task environmental shifts is essential for the trustworthy deployment of LLM agents in real-world settings. STT-Arena exposes fundamental deficiencies in current models' ability to handle spatio-temporal dynamics, demonstrating that high performance on static benchmarks does not guarantee robustness under realistic, evolving conditions. By identifying specific failure modes and proposing a training paradigm (trajectory refinement + RL) that effectively mitigates them, the work provides a foundation for building agents that are genuinely robust in dynamic environments. The authors position STT-Arena as a necessary stress test for assessing deployment readiness, moving beyond simple responsiveness to evaluate true adaptive reasoning.