ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

1. Problem Statement

Current benchmarks for Language Model (LM) agents primarily evaluate "one-shot" task solvers within static, single-session environments. However, the emerging use case for LM agents is that of persistent coworkers who work alongside humans over multiple days. This setting presents three critical challenges that existing benchmarks fail to address:

• Temporal Evolution: The environment changes independently of the agent between turns (e.g., new emails arrive, calendar entries shift, files are updated). Agents must adapt to these exogenous state changes rather than assuming a static world.
• Dynamic State: Unlike static logs or cached snapshots, real workflows involve stateful services where data read in Step 1 may differ in Step 2 due to external factors.
• Multimodal Evidence: Professional workflows rely on raw, untranscribed evidence across diverse modalities (images, scanned PDFs, audio, video, spreadsheets), not just text.

Existing benchmarks often rely on "LLM-as-a-judge" for scoring, which introduces subjectivity and non-determinism, and they rarely simulate the "living world" where the environment mutates between agent turns.

2. Methodology: The ClawMark Benchmark

ClawMark is a benchmark designed to evaluate coworker agents in a stateful, dynamic, and multimodal environment.

A. Environment & Architecture

• Stateful Sandboxed Services: The benchmark runs against five isolated, stateful services within a Docker environment:
  1. Filesystem
  2. Email (GreenMail SMTP/IMAP)
  3. Calendar (Radicale CalDAV)
  4. Knowledge Base (Notion-compatible)
  5. Spreadsheet (Google Sheets-compatible)
• Multi-Turn, Multi-Day Workflow: Each task spans 2–6 turns, where one turn equals one in-universe working day.
• Exogenous Mutations: Between turns, the environment mutates independently of the agent via two mechanisms:
  • Loud Events: Explicitly announced in the "wake-up" prompt (e.g., "A new email arrived").
  • Silent Mutations: Unannounced changes injected into the services (e.g., a spreadsheet row is overwritten, a file is modified) without notification. The agent must detect these by refreshing the state.
• Multimodal Inputs: Evidence is delivered as raw artifacts (photos, audio, video, PDFs) without pre-transcription, requiring agents to use tools (e.g., whisper, ffmpeg, PyMuPDF) to process them.

B. Scoring Protocol (No LLM-as-Judge)

ClawMark employs a fully deterministic, rule-based scoring system:

• Checkers: Each task includes 6–29 weighted Python checkers that inspect the post-execution state of the sandboxed services.
• Metrics:
  • Weighted Score: A continuous metric (0–100) rewarding partial progress based on the pass rate of weighted checkers.
  • Task Success: A strict binary metric (0–100%) requiring all checkers to pass.
• Red-Line Constraints: 55 specific checkers act as "red lines" (hard constraints). Violating these (e.g., approving a claim before a final report, data exfiltration) results in severe penalties or failure, regardless of other progress.
• Determinism Guarantee: Tasks are only released if two independent re-runs produce bit-identical checker verdicts, ensuring reproducibility without LLM judges.

C. Construction Pipeline

The benchmark was built using a task-first pipeline:

1. Task Authoring: Designers define the workflow, rubric, and required state changes.
2. Evidence Sourcing: 1,072 raw multimodal artifacts are sourced via web collection, original recording, or targeted AI synthesis to match the task design.
3. Review Loop: A 3–5 round process involving human review, AI audits (checking for "checker hacking" or brittle string matches), and trajectory reviews by reference models.
4. Release Gate: Tasks are only released if they pass all audits and demonstrate deterministic checker consistency.

3. Key Contributions

1. First "Living-World" Benchmark: Introduces a benchmark combining multi-day timelines, exogenous environment mutations, and raw multimodal evidence.
2. Deterministic Evaluation Protocol: Establishes a 1,537-checker, rule-based scoring system that eliminates LLM-as-judge bias and ensures bit-identical reproducibility.
3. Comprehensive Dataset: Releases 100 tasks across 13 professional scenarios (e.g., Clinical Assistant, Insurance, EDA, Legal) involving 87 distinct roles.
4. Open Infrastructure: Releases the benchmark, evaluation harness, and construction pipeline to support reproducible research.

4. Experimental Results

The authors benchmarked seven frontier agent systems (5 proprietary, 2 open-source) using the OpenClaw framework.

Overall Performance

• Weighted Score: The top performer, Claude Sonnet 4.6, achieved 75.8. Other top models included Claude Opus 4.6 (74.6) and GPT-5.4 (72.0).
• Strict Task Success: The gap between partial progress and full completion is massive. The best Task Success rate was only 20.0% (Claude Opus 4.6). Most models solved fewer than 15% of tasks end-to-end.
• Scenario Variance: Performance varied significantly by domain. For example, models excelled in "EDA" (Electronic Design Automation) and "Real Estate" but struggled significantly in "Project Management" (mean score ~35.1).

Turn-Level Analysis (Adaptation)

• Day 2 Drop: Analysis of 73 three-turn tasks revealed that six out of seven models dropped in performance on Day 2, the point where the first exogenous mutation occurs.
• Recovery: By Day 3, most models only partially recovered, remaining below their Day 1 baselines. This highlights that adapting to changing state is a primary failure point.
• Efficiency: There was no monotone relationship between token/tool usage and score. Interestingly, the top-scoring models (Sonnet 4.6, Opus 4.6) were also the most tool-efficient.

Failure Taxonomy

Analysis of 10,759 checker evaluations revealed two dominant failure modes:

1. Silent-Change Detection (56.5% failure rate): Agents frequently failed to notice unannounced changes in the environment.
2. Backend Writeback (53.6% failure rate): Agents often reasoned correctly but failed to commit the final action to the correct service (e.g., saving a file or updating a row).

• Red-Line Violations: While rare overall (7.1%), they were concentrated in specific tasks. Notably, Qwen 3.6 Plus had a significantly higher red-line failure rate (14.5%) compared to top-tier models (~3.6%).

5. Significance

ClawMark represents a paradigm shift in agent evaluation:

• From Static to Dynamic: It moves beyond "solve this puzzle" to "manage this workflow," testing an agent's ability to maintain context over time and adapt to external noise.
• Reliability over Fluency: The strict Task Success metric and red-line constraints emphasize that in professional settings, compliance and correctness are more critical than partial progress or fluent conversation.
• Real-World Readiness: The results indicate that while frontier models can handle complex reasoning, they currently lack the robustness required for persistent, autonomous coworker roles, particularly in detecting silent state changes and executing reliable backend updates.

The release of ClawMark provides the necessary infrastructure to drive progress toward truly reliable, persistent AI coworkers.