💬 NLP

RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions

RealClawBench introduces a live benchmark framework derived from real OpenClaw developer-agent sessions that utilizes reconstructed execution environments and deterministic scorers to transform 281 challenging, real-world tasks into reproducible evaluations, revealing significant performance gaps in current models.

Original authors: Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to test how good a new robot chef is.

The Old Way (Conventional Benchmarks):
Traditionally, researchers would write a list of perfect, textbook recipes like "Make a grilled cheese sandwich" or "Bake a chocolate cake." They would give these recipes to the robot and see if it follows the steps. The problem? Real life isn't a textbook. In the real world, a customer might say, "My bread is slightly stale, the cheese is in the back of the fridge, and I only have a toaster oven, not a grill. Can you make something edible?" The old tests didn't check if the robot could handle those messy, real-world details.

The New Way (REALCLAWBENCH):
The authors of this paper realized that to truly test a "developer agent" (a robot that helps programmers), they needed to stop writing fake tests and start watching real people use the robots in the wild.

They built a system called REALCLAWBENCH. Here is how it works, using simple analogies:

1. Catching Real Moments (The Source)

Instead of inventing tasks, the team watched 76,155 real sessions where actual developers used a tool called "OpenClaw" to get help with their code. They saw real people asking for help with messy, complicated, and sometimes vague requests.

Analogy: Imagine a security camera recording thousands of real customers ordering food at a busy restaurant, rather than a chef writing a menu in a quiet kitchen.

2. The "Magic Clean-Up" (The Pipeline)

You can't just take a raw recording of a real customer and turn it into a test. The recording might contain private passwords, secret company files, or references to computers that don't exist anymore.
The paper describes a pipeline (a step-by-step factory line) that cleans these real moments:

Privacy Screen: They scrub out all secrets and private info, like blurring faces in a video.
Environment Reconstruction: If a developer asked the robot to fix a file on their specific computer, the team builds a fake, safe version of that computer environment so the test can be run again later.
Rewriting the Request: They turn a vague, conversational request ("Hey, can you look at that thing I mentioned earlier?") into a clear, standalone instruction ("Please fix the bug in the login file").
The Automatic Judge: Instead of a human grading the work, they write a computer program that checks the result. Did the file get fixed? Yes/No. No guessing allowed.

3. The Result: A "Live" Test

The final product is a benchmark with 281 real-world tasks.

Why it's special: It's "live" and "anchored." This means the test isn't a static list of questions that gets old. Because they built it from a continuous stream of real user data, they can update the test later with new real-world examples to keep it fresh.
The "Realism Gap": The paper argues that previous tests had a "gap" between what they tested and what actually happens. REALCLAWBENCH closes that gap. It's like switching from testing a driver on a perfect, empty racetrack to testing them in actual rush-hour traffic with potholes and confused pedestrians.

What Did They Find?

They tested 14 of the smartest AI models currently available on this new, messy, real-world test.

The Score: Even the best AI model (Claude Opus 4.7) only solved about 66% of the tasks.
The Takeaway: This means there is still a lot of room for improvement. The current "super-intelligent" robots are still struggling with the kind of messy, real-life problems that actual developers face every day.

Summary

In short, the paper says: "Stop testing robots with fake, perfect scenarios. Let's test them with the messy, real-life problems they actually face, but clean them up just enough so we can grade them fairly."

They built a system to do exactly that, and they found that even the best robots today are still only about two-thirds of the way to being truly reliable in the real world.

Technical Summary: REALCLAWBENCH

Problem Statement

Current agent benchmarks often fail to reflect the reality of deployed developer-agent sessions. While existing benchmarks (e.g., SWE-bench, WebArena, AgentBench) have improved executability and tool-awareness, they primarily rely on authored tasks, public artifacts, or curated environments. This creates a "realism gap" where benchmarks may be executable and difficult but lack the local state, implicit user intent, and naturally occurring success criteria found in actual production workflows.

Real-world developer-agent sessions (specifically within the OpenClaw ecosystem) present unique challenges for benchmarking:

Context Dependency: Tasks often rely on local execution environments and specific project states.
Implicit Intent: User requests frequently contain underspecified goals or conversational ellipsis.
Verification Difficulty: Success criteria are often non-deterministic or require subjective judgment, making automated scoring difficult.

Consequently, existing evaluations may not accurately measure an agent's capability to solve tasks drawn from the actual distribution of deployed usage.

Methodology

The paper introduces REALCLAWBENCH, a live benchmark framework constructed directly from real OpenClaw user sessions. The core methodology involves a pipeline that transforms raw, sensitive deployment logs into reproducible, privacy-screened, and automatically scored tasks.

1. Source Distribution Anchoring

The framework uses 76,155 deployed tool-use sessions from OpenClaw as the source distribution. Unlike previous approaches that sample from final curated pools, REALCLAWBENCH anchors its sampling to the cleaned tool-use stream before final quality filtering. This ensures the benchmark preserves the empirical distribution of real user demands, including rare categories and specific task mixes.

2. Session-to-Task Pipeline

Each released instance is constructed through a four-stage transformation process (illustrated in Figure 2):

Environment Reconstruction ( $E_0$ ): The initial workspace is materialized from session evidence, including files, dependencies, command contexts, and schemas. Tasks relying on private services are either fixture-backed or rejected.
Request Rewriting ( $x$ ): Original user messages are converted into standalone instructions. This process resolves local references and conversational ellipsis while removing private identifiers. Crucially, rewrites preserve the original task category without adding solution hints.
Verifier Construction ( $V$ ): A deterministic program is built to verify the final workspace, stdout, and bounded subprocesses. Verifiers check structural outcomes (e.g., file existence, schema validity). Tasks requiring subjective or aesthetic judgment are excluded to ensure automatic, deterministic scoring.
Fidelity Review: Draft instances undergo probing to detect bugs, ambiguous wording, or "leaky" instructions. Only instances passing reconstructability and verifier-validity checks are released.

3. Live Release Mechanism

REALCLAWBENCH supports versioned live releases. The same construction map ( $T$ ) can be applied to non-overlapping temporal windows of future logs. This allows the benchmark to track temporal shifts in user demand while maintaining a controlled evaluation protocol.

Key Contributions

Identification of the Realism Gap: The paper formally defines the discrepancy between executable agent benchmarks and the actual distribution of deployed developer-agent use, highlighting the lack of local state and implicit intent in current datasets.
Session-to-Task Pipeline: A novel pipeline that converts raw, privacy-sensitive deployment logs into reproducible benchmark tasks using environment reconstruction, intent-preserving rewriting, and deterministic verifiers.
Live Benchmark Release: The release of REALCLAWBENCH, a distribution-anchored benchmark containing 281 executable tasks sampled from real sessions. It demonstrates that the construction process can be rerun on future logs to produce versioned updates without redesigning filters.

Results

Distributional Fidelity

The construction pipeline successfully preserves the shape of the real user distribution.

Jensen-Shannon Divergence (JSD): The final 281-task release maintains a maximum JSD of 0.0448 against the source distribution (76,155 sessions) across four axes: task type, user turns, tools defined, and tool calls.
Filtering Impact: Aggressive filtering (from raw calls to the final set) did not introduce significant distribution drift, with all adjacent-stage shifts remaining below 0.05 bits.

Model Performance (Leaderboard)

The benchmark was used to evaluate 14 contemporary models (including Claude Opus/Sonnet, GPT-5.5, DeepSeek, Gemini, and others) under a shared OpenClaw harness with a 600-second timeout.

Top Performance: The best-performing model, Claude Opus 4.7, achieved a 65.8% sample-average success rate.
Headroom: The fact that the top model solves only ~66% of tasks reveals substantial headroom for current agents on realistic developer workloads.
Cost-Efficiency: Performance and cost are not tightly aligned. For instance, MiMo V2.5 Pro offered a favorable tradeoff at a significantly lower cost ($11.87) compared to higher-cost models like GPT-5.5 ($61.87) which ranked lower.
Long-Tail Behavior: Macro-averaged scores (accounting for the imbalanced distribution of task types) revealed different rankings compared to simple sample averages, highlighting the importance of evaluating performance across diverse, real-world task categories.

Live Benchmarking Validation

The pipeline was tested on a later temporal window ( $W_{live}$ ) without modifying filters or rules.

Yield: The pipeline successfully produced 34 final cases from the live window, with retention ratios comparable to the base window.
Drift: The live window remained distributionally close to the base window (max JSD of 0.0411), confirming that the protocol can track demand shifts while maintaining comparability.

Significance and Claims

The paper claims that REALCLAWBENCH provides a practical path toward benchmarks that better measure agent capability in actual use. By turning real deployed sessions into controlled evaluation instances, the framework addresses the limitations of authored benchmarks.

Key claims include:

Realism: The benchmark captures the distribution, diversity, and real-world difficulty of deployed agent use, which is often missed by curated datasets.
Reproducibility: It demonstrates that real sessions can be converted into reproducible, automatically scored tasks without losing the "measured shape" of the source distribution.
Dynamic Evaluation: The framework supports live, versioned releases that reduce staleness and data contamination, allowing the benchmark to evolve alongside deployed agent usage.

The authors emphasize that the benchmark is specific to the OpenClaw ecosystem and developer-oriented user population, and that programmatic scorers favor outcomes checkable in a final workspace, potentially missing qualities like code maintainability or partial progress. However, the framework establishes a new standard for grounding agent evaluation in real-world deployment data.