🤖 AI

Embodied-BenchClaw: An Autonomous Multi-Agent System for Embodied Spatial Intelligence Benchmark Construction

The paper introduces Embodied-BenchClaw, an autonomous multi-agent system that automates the construction of verifiable, maintainable, and diverse embodied spatial intelligence benchmarks through a five-stage pipeline, thereby addressing the labor-intensive and static limitations of existing evaluation frameworks.

Original authors: Baoyang Jiang, Fengchun Zhang, Leyuan Wang, Haotian Li, Yida Wang, Zhe Ji, Jinshan Lai, Xi Ren, Jianwei Hu, Qiang Ma

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Baoyang Jiang, Fengchun Zhang, Leyuan Wang, Haotian Li, Yida Wang, Zhe Ji, Jinshan Lai, Xi Ren, Jianwei Hu, Qiang Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a house, drive a car, or fly a drone. To do this, you need a "test" (a benchmark) to see if the robot is smart enough. But right now, making these tests is like building a custom house for every single new robot: it takes months of manual labor, the blueprints are messy, and by the time you finish, the robot has already learned how to pass the test, making the test useless.

Embodied-BenchClaw is a new system that acts like an automated construction crew for these robot tests. Instead of humans building every test from scratch, this system uses a team of AI "agents" to design, build, and quality-check the tests automatically.

Here is how it works, using simple analogies:

1. The Three-Worker Team

The system doesn't just have one AI doing everything. It has three specialized workers (agents) that pass the project along like an assembly line:

The Architect (Planning Agent): You tell this worker, "I need a test for a robot that walks on four legs over rocky terrain." The Architect listens, figures out exactly what skills need testing, and draws up a blueprint.
The Builder (Construction Agent): This worker takes the blueprint and starts gathering materials. It pulls in real photos, video from simulators, or old test data. It then assembles the actual test questions, making sure they are grounded in real visual evidence (like a photo of a rock), not just made-up stories.
The Inspector (Evaluation Agent): This worker is the strict quality control manager. As the Builder makes the test, the Inspector checks every step. Did the Builder use the right photo? Is the answer actually derivable from that photo? If something is wrong, the Inspector doesn't throw the whole project away; they send it back to the Builder to fix just that specific part.

2. The "Lego" Library (Skill Library)

One of the paper's big innovations is the Skill Library. Imagine building a house. Instead of inventing how to lay a brick or install a window every time, you have a box of pre-made, pre-tested Lego blocks.

In Embodied-BenchClaw, these "blocks" are Skills. A skill might be "find the distance between two objects in an image" or "check if a path is clear."
Because these skills are pre-verified and reusable, the system can build complex tests quickly without reinventing the wheel every time. If a new type of robot is introduced, the team just grabs the right "Lego blocks" and snaps them together.

3. The "Self-Healing" Process

Usually, if a human makes a mistake building a test, they might have to start over or spend hours fixing it. Embodied-BenchClaw has a Local Repair mechanism.

Think of it like a GPS that realizes you took a wrong turn. Instead of telling you to go back to your house and start the drive over, it just reroutes you from your current location.
If the system finds a bad test question, it traces exactly where the mistake happened (provenance tracing) and fixes only that specific step, keeping the rest of the work intact.

4. What Did They Build?

The paper shows that this system successfully built six different types of "robot driving tests" (benchmarks) covering:

Indoor Navigation: Robots moving through rooms.
Driving: Cars navigating streets.
Robot Arms: Robots picking up and moving objects.
Four-Legged Robots: Dogs or quadrupeds walking over rough terrain.
Drones: Aerial views and flying.
Upgrading Old Tests: Taking old, "saturated" tests and making them harder and more specific.

5. The Results

The authors tested this system by having it build a test for drones (UAVs).

The "Blind" Test: When they showed the test to AI models without letting them see the images (just the text), the models scored very low (around 30%). This proves the test actually requires looking at the picture, not just guessing based on language patterns.
The "Vision" Test: When the models could see the images, their scores jumped to around 65%.
The Verdict: This proves the system created a fair, difficult, and useful test that can actually tell the difference between a smart robot and a dumb one.

Summary

Embodied-BenchClaw is a system that automates the creation of tests for robots. It uses a team of AI agents, a library of reusable "building blocks," and a self-correcting process to build high-quality, visual tests quickly. This solves the problem of tests being too slow to make and too easy to cheat, ensuring we can always find new ways to measure how smart our robots are becoming.

Technical Summary: Embodied-BenchClaw

Problem Statement

The construction of benchmarks for embodied spatial intelligence is currently a bottleneck in the field. While Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) are advancing rapidly, benchmark creation remains labor-intensive, difficult to reuse, and hard to maintain. Existing benchmarks are often released as static datasets that quickly become saturated or overfitted as model capabilities improve, reducing their ability to distinguish next-generation models. Furthermore, current benchmarks often exhibit representational redundancy, covering similar regions of the model's internal feature space, and lack the flexibility to update or refine evaluation protocols without substantial manual effort. There is a critical need for a system that can automatically construct verifiable, executable, and continually updatable benchmark packages tailored to specific user intents.

Methodology

The paper proposes Embodied-BenchClaw, an autonomous multi-agent system designed to automate the construction of embodied spatial intelligence benchmarks. The system transforms high-level user requests into continually updatable benchmark packages through a five-stage pipeline coordinated by three specialized agents: a Planning Agent, a Construction Agent, and an Evaluation Agent (which also handles process quality control).

Core Components

Five-Stage Automated Workflow:
- Intent Blueprinting: Converts user requests into structured blueprints defining target capabilities, task scope, and resource plans.
- Data Collection: Aggregates data from heterogeneous sources, including simulators, real-world images/videos, user-provided data, and existing benchmarks.
- Structuring and Cleaning: Normalizes raw data into structured evidence records, removing invalid samples and organizing metadata.
- Benchmark Synthesis: Instantiates benchmark items by binding expert templates with verified evidence, generating questions, reference answers, and executable scoring protocols.
- Evaluation Reporting: Executes model evaluations, parses responses, computes scores, and generates diagnostic reports.
Hierarchical Skill-Guided Execution:
- The system utilizes a Skill Library where skills are defined as contract-based execution units ( $Sk = \langle I_k, O_k, E_k, V_k, R_k, \tau_k \rangle$ ). These units have explicit input/output schemas, executable components (e.g., Python scripts, LLM calls, simulator adapters), validation rules, and optional repair actions.
- Construction proceeds via Stage-wise Skill DAGs (Directed Acyclic Graphs). Instead of free-form generation, agents execute predefined DAGs where nodes represent skills and edges represent artifact dependencies. This ensures the workflow is controllable, reproducible, and traceable.
Process Quality Control and Local Repair:
- Unlike post-processing, quality control is embedded within the workflow. A Process Quality-Control Agent performs contract-guided verification at every stage.
- Verification checks include schema validity, evidence completeness, answer derivability, scoring executability, and traceability.
- If a verification fails, the system uses provenance tracing to locate the specific failed node and triggers local repair (e.g., recollecting data, regenerating items) rather than restarting the entire workflow.
Reusable Knowledge Resources:
- Expert Templates: Define task patterns (question forms, evidence requirements, scoring constraints) for specific embodied scenarios.
- Capability Cards: Describe available resources (simulators, datasets, tools), their invocation methods, and constraints, ensuring that benchmark items are grounded in feasible evidence.

Key Contributions

Embodied-BenchClaw Framework: A fully automated multi-agent system that converts user evaluation intents into complete, updatable benchmark packages, covering planning, construction, and evaluation.
Hierarchical Skill-Guided Execution: A mechanism using stage-wise DAGs and contract-defined skills to ensure modular, traceable, and repairable benchmark construction.
Process Quality Control: A novel mechanism combining executable verification, provenance tracing, and local repair to constrain agent behavior and ensure the reliability of generated artifacts.
Benchmark Instantiation: The system was used to construct six representative benchmarks: Indoor-Bench, NavLand-Bench, Drive-Bench, Aerial-Bench, Quad-Bench, and Manip-Bench, covering diverse embodied carriers (indoor robots, autonomous vehicles, UAVs, quadruped robots) and spatial capabilities.

Results

The authors conducted extensive experiments including human evaluation, judge-based quality assessment, consistency checks, and model evaluation on the generated benchmarks.

Benchmark Utility (UAV/Aerial-View Case Study):
- A benchmark of 5,000 questions was generated for UAV/aerial-view understanding.
- Evidence Dependency: Vision-language models achieved an average score of 65.39%, while blind evaluation (text-only) dropped to 29.82%, significantly below the random baseline (33.30%). This gap confirms that the benchmark requires genuine visual-spatial reasoning rather than language priors.
- Model Separability: The benchmark successfully distinguished between models. For instance, GPT-5.5 achieved 74.13%, while open-source models like Qwen3.6-27B scored 71.27%. Different models showed distinct strengths across question types (e.g., binary comparison vs. interval selection).
- Difficulty Hierarchy: The benchmark revealed varying difficulty levels, with interval selection being the most challenging (32.07% average) and binary comparison the easiest (84.36%).
Construction Reliability:
- The system demonstrated high success rates in constructing valid packages across diverse scenarios.
- Ablation studies confirmed that removing components like the Skill Library, Formal Verification, or Local Repair significantly degraded the valid yield rate and quality of the generated benchmarks.

Significance and Claims

The paper claims that Embodied-BenchClaw addresses the critical mismatch between rapid model development and slow benchmark construction. By making benchmark construction reusable, verifiable, and maintainable, the system supports:

Timely Evaluation: The ability to rapidly generate benchmarks for new model capabilities or application scenarios.
Lifecycle-Oriented Maintenance: The capacity to continually refresh and update benchmarks to prevent saturation and overfitting.
Diagnostic Value: The generation of fine-grained, evidence-grounded benchmarks that can expose specific failure modes in embodied spatial intelligence (e.g., depth reasoning, occlusion handling).

The authors position Embodied-BenchClaw not merely as a question generator, but as a comprehensive framework for the lifecycle management of embodied spatial benchmarks, enabling scalable and continually refreshable evaluation of VLMs and MLLMs.