Technical Summary: ClawForge

Problem Statement

Interactive agent benchmarks currently face a fundamental tension between scalable construction and realistic workflow evaluation. Existing approaches rely heavily on hand-authored tasks, which are expensive to extend and revise, or static prompt evaluations that miss failures occurring only when agents operate over persistent state. Crucially, most current interactive benchmarks initialize tasks from a "clean state," failing to systematically test how agents handle pre-existing partial, stale, or conflicting artifacts. This gap leaves a central aspect of workflow competence unmeasured: correctness that depends on the final state of a live environment rather than merely on a plausible textual answer or command sequence.

Methodology

The authors present ClawForge, a generator-backed benchmark framework designed for executable command-line workflows under state conflict. Unlike static prompt collections, ClawForge treats each task as an executable specification comprising five coupled components:

1. Instruction ($x$): A natural language command.
2. Initialized State ($S_0$): A seeded environment state containing pre-existing artifacts (which may be valid, stale, or conflicting).
3. Reference Trajectory ($C^\star$): A ground-truth command sequence derived from executing the intended workflow against the initialized state.
4. Executable Checks ($E$): Validators that verify required state transitions and observable side effects.
5. Metadata ($m$): Structured information regarding scenario families and ability categories.

Generation and Execution Pipeline

The framework compiles scenario templates and grounded slots (e.g., city, timezone, recipient) into reproducible task specifications. The execution environment exposes a CLI-style interface over workflow surfaces including tasks, calendars, email, messaging, files, runtime configuration, and weather.

• Stateful Rollout: Agents interact step-by-step in a stateful loop. The environment executes commands, records state-changing effects, and constructs the next observation based on updated state.
• Result-First Evaluation: Evaluation is performed on a normalized end state ($\hat{S}$) and observable side effects rather than exact trajectory matching. This allows multiple valid command sequences to pass if they produce the required state transitions.
• Scoring: The framework reports two metrics:
  • Strict Accuracy: The fraction of tasks where every required check passes.
  • Partial-Credit Score: A weighted average measuring how much of the workflow was completed correctly, distinguishing between early breakdowns and "near-miss" closures.

Key Contributions

1. ClawForge Framework: A system for automated task construction, stateful execution, and result-first evaluation that maintains reproducibility and extensibility in interactive benchmarks.
2. ClawForge-Bench: A concrete instantiation of the framework containing 17 scenario families organized into 6 ability categories:
   • Duplicate Avoidance
   • Gap Completion
   • Information Transfer
   • Multi-Source Reasoning
   • State Repair
   • Workflow Completion
3. Failure Mode Analysis: The benchmark specifically targets realistic failure modes such as duplicate-aware completion, stale-state repair, wrong-state replacement, multi-source branch resolution, and full workflow closure.

Experimental Results

The authors evaluated seven frontier models (including Kimi-K2.5, GPT-5 variants, and Claude 4.6 variants) on ClawForge-Bench. Key findings include:

• Low Saturation: The benchmark remains far from saturated. The best-performing model (Claude Opus 4.6) achieved only 45.3% strict accuracy.
• State Conflict Difficulty: "Wrong-State Replacement" proved to be the most challenging shared slice, with all models scoring below 17% (the best reaching 17.4%). This indicates a significant inability to identify stale state, preserve valid parts, and replace invalid parts simultaneously.
• Model Separation: The widest performance gap (17% to 90%) was observed in the "Interrupted Workflow Resume" scenario. This separation is driven primarily by whether agents inspect existing state before acting; models that skip inspection tend to recreate existing objects, triggering duplicate guards.
• Failure Styles: Partial-credit and step-efficiency analyses revealed that many failures are "near-miss" closures rather than early breakdowns. Models often identify the correct repair direction but miss a final side effect. Furthermore, models exhibit qualitatively different failure styles: some are step-efficient but miss final operations, while others consume excessive steps without improving closure.

Significance

The paper claims that ClawForge addresses a critical blind spot in agent evaluation: the ability to manage state conflicts in persistent workflows. By moving beyond clean-state initialization, the benchmark exposes failure modes—such as incorrect state replacement and failure to recognize existing valid artifacts—that static or clean-state benchmarks cannot observe. The results suggest that current frontier models struggle significantly with the judgment calls required in state-conflict scenarios, specifically distinguishing between what needs to be repaired versus what needs to be preserved. The framework provides a mechanism for systematically measuring these capabilities, offering a more diagnostic view of agent reliability in real-world operational contexts.