💻 computer science

Enabling Extensible Embodied Capabilities with Tools

This paper proposes the Embodied Tool Protocol (ETP) and a corresponding tool base to decouple heterogeneous embodied capabilities into independently optimized tools, demonstrating through the EmbodiedToolBench that while this externalization significantly boosts perception and cognition performance, it faces challenges in execution and tool invocation management.

Original authors: Xueyang Zhou, Zijia Wang, Qianjiang Li, Yibo Hu, Guiyao Tie, Li Wan, Yidan Liu, Pan Zhou, Lichao Sun, Yongchao Chen

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xueyang Zhou, Zijia Wang, Qianjiang Li, Yibo Hu, Guiyao Tie, Li Wan, Yidan Liu, Pan Zhou, Lichao Sun, Yongchao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to do chores around the house, like cleaning up a messy room or making a sandwich.

The Old Way: The "Super-Brain" Problem
Traditionally, scientists tried to build one giant, all-knowing robot brain. They wanted this single brain to do everything: see what's on the table, figure out the best plan, remember where the keys are, and actually move the robot's arm to pick them up.

The paper argues this is like trying to teach a single student to be a world-class surgeon, a professional chef, a master carpenter, and a GPS navigator all at the same time. It's too much! The brain gets confused, the different skills get tangled up, and when the robot tries to learn one thing (like how to hold a cup), it might accidentally forget how to walk.

The New Way: The "Toolbox" Approach
The authors propose a smarter idea: Don't build one giant brain; build a smart manager with a toolbox.

Instead of forcing the robot to learn every skill itself, they give it a set of specialized "tools" (software programs) that it can call upon when needed.

The Manager (The Robot's Brain): Its only job is to look at the situation and decide which tool to use. It doesn't need to know how to calculate depth or track objects; it just needs to know, "Hey, I need to know how far that cup is. Let me ask the Depth Tool."
The Tools: These are pre-made, specialized experts. One tool is great at finding objects (like a magnifying glass). Another is great at planning a path (like a GPS). Another is great at remembering where things were (like a notepad).

The "Embodied Tool Protocol" (ETP): The Universal Adapter
The paper introduces a standard rulebook called the Embodied Tool Protocol (ETP). Think of this as a universal power strip or a standard USB port.

Before, every tool spoke a different language. The "Vision Tool" might speak French, and the "Planning Tool" might speak Spanish. The robot couldn't use them together.
ETP forces all tools to speak the same language. Now, the robot's manager can plug in any tool, ask it a question, and get a clear answer, no matter who made the tool.

The "Toolbox" Collection
The researchers didn't just talk about this; they actually built it. They curated a massive library of over 100 validated tools.

Some tools help the robot see (finding a specific red cup in a messy kitchen).
Some help it think (figuring out that the cup is on top of a book).
Some help it plan (deciding to move the book before grabbing the cup).
Some help it act (calculating the exact angle to grab the cup without dropping it).

The Test: "EmbodiedToolBench"
To see if this actually works, they created a giant exam called EmbodiedToolBench. It's like a driving test for robots, but instead of just driving, the robot has to decide:

Do I need help? (Should I call a tool, or can I do this myself?)
Which tool? (Do I need the GPS or the Map?)
Can I use it? (Did I ask the tool the right question?)
Can I combine them? (Can I use the GPS, then the Map, then the Compass in the right order?)

What They Found

The Good News: Giving robots this toolbox made them much better at tasks that require thinking and seeing. For example, on tasks involving navigation and following complex instructions, the robots got about 30-36% better just by using tools. It was like giving a student a calculator and a dictionary; they solved problems much faster and more accurately.
The "Open-Source" Surprise: Smaller, cheaper robot brains (open-source models) benefited the most. With tools, a small robot brain could actually outperform a giant, expensive brain that didn't have tools.
The Limitation: The toolbox wasn't a magic wand for everything. When it came to the very fine, physical details of moving a robot arm (like picking up a delicate egg without crushing it), the tools didn't help as much. The robot still struggled with the physical "muscle" part of the job.
The Real Challenge: The biggest problem wasn't the tools themselves; it was the robot's ability to know when to use them. Many robots still forgot to call for help when they needed it, or they picked the wrong tool (like using a hammer to fix a watch).

In Summary
This paper says: Stop trying to make one robot brain do everything. Instead, give the robot a smart manager and a standardized toolbox of specialized helpers. This makes robots smarter, more flexible, and better at complex tasks, as long as we can teach them when to reach for the right tool.

Technical Summary: Enabling Extensible Embodied Capabilities with Tools

Problem Statement

Current embodied intelligence methods predominantly rely on unified, parameterized policies (e.g., vision-language-action models) to handle perception, reasoning, planning, and control simultaneously. The authors argue that this monolithic architecture faces inherent structural limitations because embodied decision-making is fundamentally hierarchical and heterogeneous. High-level planning and low-level control differ significantly in their functional roles and optimization objectives. Encoding these diverse capabilities within a single parameter set leads to three critical issues:

Opaque Coupling: Capabilities are jointly encoded without clear attribution, preventing selective invocation or diagnosis of failures.
Capability Isolation: Competencies learned by one model remain siloed and cannot be transferred or reused across different systems.
Entangled Optimization: Improving one capability risks degrading others due to conflicting gradient signals over shared parameters.

While recent advances have improved spatial grounding and reasoning, these monolithic models still underperform on long-horizon, compositional, and safety-critical tasks. The paper posits that the gap is structural rather than a lack of capacity, motivating a shift toward tool-based embodied intelligence where specific capabilities are externalized as independently optimizable tools.

Methodology

1. Embodied Tool Protocol (ETP)

To address the lack of a unifying framework, the authors introduce the Embodied Tool Protocol (ETP). This standardized protocol decouples capability representation from implementation, enabling registration, discovery, invocation, and execution of tools.

Tool as a Capability Unit: Each tool $z_m$ is defined by its input-output spaces $(X_m, Y_m)$ , a realized capability subset, and an executable mapping $f_m$ . The implementation can be a learned model, a classical algorithm, or a hybrid, but the interface remains consistent.
Registry-Based Discovery: Tools are maintained in a capability registry $R$ with descriptions ( $\rho_m$ ) and schema constraints ( $\kappa_m$ ). Agents query the registry based on capability needs rather than specific calling conventions.
Isolated Execution: Tools run in isolated sessions with runtime feedback, supporting concurrent calls and remote execution without coupling the tool implementation to the agent's policy.

2. Tool Base Construction

Grounded in ETP, the authors curated a tool base of 100+ validated tools spanning four core capability dimensions:

Perception and Grounding: Object detection, segmentation, spatial localization, and scene understanding.
Cognition and State Modeling: State estimation, relational modeling, affordance understanding, and memory retrieval.
Reasoning and Planning: Goal decomposition, causal reasoning, and long-horizon planning.
Execution and Control: Trajectory generation, grasp planning, and low-level actuation.

Each tool was selected based on being lightweight, easy to deploy, and environmentally robust, then encapsulated with a structured API and tool card.

3. EmbodiedToolBench

The authors constructed EmbodiedToolBench to systematically evaluate the tool-use competence of current models. The benchmark consists of four tasks:

Tool-Need Recognition: Determining whether external tools are necessary for a given decision state.
Tool Selection: Identifying the most appropriate tool from a candidate set (including semantically similar distractors).
Tool Execution: Constructing valid tool queries (Stage 1) and correctly acting upon tool outputs (Stage 2).
Tool-Chain Composition: Selecting the minimal required tools and arranging them in the correct dependency order.

4. Evaluation Framework

Experiments were conducted on both simulation platforms (EB-ALFRED, EB-Habitat, EB-Navigation, EB-Manipulation) and real-world robotic platforms. The study evaluated eight representative open-source and closed-source models, comparing performance with and without tool augmentation.

Key Results

Performance Improvements

Consistent Gains: Tool augmentation consistently improved embodied task performance across all benchmarks.
- EB-ALFRED: Average gain of 31%. Open-source models like Qwen3-8B saw dramatic improvements (0.20 $\to$ 0.72).
- EB-Navigation: Average gain of 36%, with particularly strong gains in Visual (+0.40) and Long-horizon (+0.54) subtasks.
- Real-World: In real-world robotic tasks (desktop cleaning, balance scale, building blocks), models without tools achieved 0/10 success, while tool-augmented agents achieved substantial success rates (e.g., 8/10 to 9/10).
Narrowing the Gap: Tool augmentation significantly improved the competitiveness of open-source models, allowing them to outperform stronger closed-source models that lacked tool access (e.g., Qwen3-8B with tools outperformed GPT-4o without tools on EB-ALFRED).

Capability Boundaries

The study revealed a clear boundary regarding which capabilities benefit from externalization:

High Benefit: Perception and Cognition tasks showed the largest gains (up to +0.34 for open-source models). Tools effectively support visual parsing and knowledge-driven tasks.
Moderate Benefit: Reasoning tasks showed consistent improvements.
Limited Benefit: Execution capabilities showed smaller and more variable gains (+0.03 to +0.19). The authors note that tasks requiring precise spatial grounding, contact-rich interaction, and low-level physical control are less effectively supported by current tool interfaces.

Competence Bottlenecks

Despite performance gains, the analysis identified persistent bottlenecks in how models use tools:

Tool-Need Recognition: Models frequently fail to recognize when tools are necessary, with recall dropping as low as 0.31 for some open-source models.
Tool Selection: Selecting the correct tool among semantically similar candidates remains a challenge (best CSR ~0.74).
Tool Execution: While input construction is generally high, result utilization is a major bottleneck. Models often struggle to interpret and act upon returned tool outputs effectively.
Tool-Chain Composition: Models tend to "over-compose," including superfluous tools rather than selecting the minimal set, indicating difficulty in precise multi-tool coordination.

Significance and Claims

The paper claims that capability externalization is a viable and effective approach to overcoming the structural limitations of monolithic embodied policies. By decoupling heterogeneous capabilities into independently optimized tools, the framework allows for:

Modular Reuse: Specialized capabilities can be registered and reused across different agents and tasks.
Targeted Optimization: Tools can be improved independently without interfering with other capabilities.
Scalability: The system can scale to large and heterogeneous tool collections without retraining the core agent policy.

However, the authors modestly conclude that tool competence—specifically the ability to know when, which, and how to invoke tools—remains the central bottleneck for truly capable embodied agents. The research highlights that while tool availability improves performance, the autonomous orchestration of these tools by current models is not yet fully reliable, particularly in execution-heavy scenarios. This identifies embodied tool competence as a critical direction for future research.