🤖 AI

Towards the Harness of Embodied Agents

This paper introduces Thea, a harness for embodied agents that adapts the coding agent paradigm to the physical world by integrating a persistent symbolic scene graph for state awareness and an evaluation mechanism for action feedback, thereby enabling the successful execution of long-horizon tasks through a closed agentic loop.

Original authors: Qi Wang, Tianyi Wang, Chengyang Li, Shikun Ban, Yurun Chen, Yizhong Ge, Jason Qin, Chengtai Li, Wentao Zhu

Published 2026-08-13

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Wang, Tianyi Wang, Chengyang Li, Shikun Ban, Yurun Chen, Yizhong Ge, Jason Qin, Chengtai Li, Wentao Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to do your chores. For a long time, scientists thought the secret to a smart robot was just building a "brain" that was smart enough to figure everything out on its own. But there's a catch: the real world is messy, unpredictable, and doesn't have a "save button" like a video game. If a robot drops a cup, it's broken. If it walks into a wall, it's stuck. This is the realm of embodied AI—artificial intelligence that lives inside a physical body and interacts with the real world.

To understand the challenge, think about how you use a computer. When you write code, if you make a mistake, the computer instantly tells you "Error: Line 5." It gives you a clear signal that something went wrong, and you can fix it. This is called a closed loop: you act, you get feedback, and you adjust. In the digital world, this feedback is free and instant. But in the physical world, the robot doesn't get a little pop-up saying "Grasp Failed." It just closes its hand, and maybe the cup falls, or maybe it didn't grab it at all. The robot has to guess what happened. This paper asks a big question: Can we build a robot that learns from its mistakes just like a computer programmer does, even when the world doesn't give it easy answers?

The Robot's "Harness": Turning a Brain into a Body

Meet Thea. The authors of this paper aren't just building a smarter robot brain; they are building a "harness." Think of a harness like the saddle and reins on a horse. The horse (the robot's body) is strong, and the rider (the AI brain) is smart, but without the harness, the rider can't steer the horse effectively. Thea is that harness. It takes a powerful AI model and connects it to a robot body in a way that lets the robot learn, recover from mistakes, and finish long, complicated tasks.

The paper suggests that the secret isn't just making the AI "smarter" in a vacuum. Instead, it's about building the right infrastructure around it. The authors show that by wrapping the robot's abilities (like moving, grabbing, or looking) into simple "tools" that the AI can call, and by creating a system that constantly checks if those tools worked, the robot can handle tasks that were previously impossible.

The Two Big Problems: Reading the World and Knowing When You Win

The paper identifies two huge gaps between the digital world (where coding agents live) and the physical world (where robots live).

1. The World is Hard to Read (The "Scene Graph" Solution)
In a computer program, the computer knows exactly where every file is. It can read the whole "codebase" at once. But a robot only sees a tiny slice of the world through its cameras, like looking through a straw. It sees a cup for a second, then turns and sees a table. It has no memory of where the cup was unless it builds one.

The Fix: Thea builds a Scene Graph. Imagine this as a persistent, magical notebook that the robot carries. Every time the robot sees something, it writes it down in the notebook: "Cup is on the table, near the lamp." Even if the robot turns its head, the notebook remembers. This turns the chaotic, blurry world into a clear, readable map that the AI can understand, just like a programmer reads code.

2. The World Doesn't Give "Exit Codes" (The "Evaluator" Solution)
When a computer program runs, it finishes and says "Success" or "Error." If a robot tries to pick up a bottle, it doesn't get a signal saying "Success." It just keeps moving. How does it know if it actually grabbed the bottle or if it just squeezed the air?

The Fix: Thea introduces an Evaluator. This is like a strict referee standing right next to the robot. After the robot tries to grab something, the Evaluator looks at the result and says, "You missed," or "You got it, but it slipped," or "Great job!" It acts like the "exit code" that the physical world is missing. If the robot fails, the Evaluator doesn't just say "No"; it explains why (e.g., "You were too far away"), so the robot can try again with a better plan.

How It Works: The Loop of Learning

The magic happens in a loop. The AI looks at its "notebook" (the Scene Graph), decides what to do, and picks a tool (like "grab cup"). The robot tries it. Then, the Evaluator checks the result.

If it works: The robot updates its notebook and moves to the next step.
If it fails: The Evaluator gives a reason. The AI reads that reason, updates its plan, and tries again.

This might sound simple, but it changes everything. The paper shows that without this loop, if a robot has to do 10 steps to finish a task, and it has an 85% chance of getting each step right, the chance of finishing the whole task is tiny (less than 20%). But with the harness, if the robot fails a step, it retries until it gets it right. Suddenly, the chance of finishing the whole task jumps to over 90%.

What They Found (and What They Didn't)

The researchers tested Thea on three different robots: a wheeled humanoid (Astribot S1), a mobile arm (AgileX Cobot Magic), and a smaller robot with dexterous hands (Unitree G1). They gave them tasks ranging from picking up one object to navigating across a room, finding a specific item in a messy cabinet, and delivering it to another table.

The Results: Thea was the most successful system in every test. When the tasks got harder and longer, the other systems (which didn't have this "harness" with an evaluator) started to fail and couldn't recover. Thea, however, kept trying, adjusting its position, and asking for help when it was stuck.
The "Active Perception": In one demo, the robot was asked to find a power bank. It wasn't on the table. Instead of giving up, the robot used its "notebook" to realize it hadn't checked the cabinet. It went to the cabinet, opened the drawers, and found the item. It didn't just follow a script; it figured out it needed to look somewhere new.
The "Exit Code" Accuracy: The Evaluator was very good at judging success or failure, getting it right about 93% of the time. The paper notes that this high accuracy is crucial; if the judge is wrong too often, the robot gets confused.

Why This Matters

The paper suggests that we don't need to wait for a "super-intelligent" robot brain that never makes mistakes. Instead, we can build a system that makes mistakes, learns from them, and recovers. The "harness" is the key. It allows different robot bodies to use the same smart brain, and it lets the same brain work on different robots.

The authors are careful to say this isn't a perfect, finished product. The "notebook" (Scene Graph) isn't perfect yet; sometimes it gets confused about where things are. The "referee" (Evaluator) isn't 100% accurate. And the robot is still a bit slow because it has to think through a language model for every move. But the direction is clear: by building the right infrastructure around the AI, we can turn clumsy robots into reliable helpers who can navigate our messy, real-world homes. The future of robotics isn't just about better brains; it's about better harnesses.

Technical Summary: Towards the Harness of Embodied Agents (Thea)

1. Problem Statement

The success of coding agents in the digital realm has established a paradigm where an agent's capability depends less on the underlying model alone and more on the "harness"—the surrounding infrastructure that orchestrates tools, manages context, and closes the loop between intent and outcome through evaluation and recovery. However, transferring this paradigm to embodied agents in the physical world faces two fundamental gaps that software environments do not:

Readability of the World: Unlike software codebases, which are structured, symbolic, and persistent, the physical world is unstructured, high-dimensional, and ephemeral. An agent perceives only local, momentary sensor streams (RGB-D frames) rather than a complete, readable state.
Verifiability of Outcomes: Software environments provide explicit termination signals and exit codes (e.g., PASS/FAIL, stack traces). The physical world offers no such signals; a robot action (e.g., a grasp) has no built-in "done" signal, and success or failure must be inferred without a natural "stderr" or exit code.

Without addressing these gaps, long-horizon tasks in the physical world fail because errors compound geometrically (open-loop execution), and the agent cannot recover from failures it cannot detect.

2. Methodology: Thea Harness

Thea is a harness designed to bridge these gaps by adapting the architecture of coding agents to the physical world. It decouples orchestration from execution, allowing diverse foundation models, policies, and robot bodies to be plugged into a unified system.

Core Architecture

The system operates via an Agentic Loop where a Vision-Language Model (VLM) selects one tool per turn, executes it, observes the result, and decides the next action.

Tool Protocol: Every robot capability (navigation, manipulation) is wrapped as a callable tool with a standardized interface (Model Context Protocol). Tools include preconditions, descriptions, and post-conditions.
Context Engineering: The context is managed across three lifetimes:
- Resident: System prompts, memory, and embodiment profiles.
- Refreshed: Current sensor data and a Scene Graph Brief (see below).
- Accumulated: Task history, notes, and tool results.
Memory: A lifecycle system that maintains Task Notes for the current session, MEMORY.md for cross-session preferences, and tool_experience files that record specific failure modes and recovery strategies for individual tools.
Safety: Deterministic safety checks (e.g., clearance measurements) are enforced via pre-execution hooks, independent of the model's discretion.

Bridging the Gaps (Key Innovations)

Thea introduces two specific mechanisms to replace the missing infrastructure of the physical world:

A. Scene Graph as Context (Restoring Readability)

To make the world readable, Thea maintains a persistent, object-centric Scene Graph.

Structure: Nodes represent objects (with IDs, coarse 3D bounding boxes, confidence, and freshness) and containers. Edges encode spatial relations (on, inside, near).
Update Mechanism: The graph is updated by two streams: perception (visual/depth evidence) and execution (evaluator-confirmed action outcomes). Only confirmed outcomes (e.g., a successful grasp) modify the graph state.
Access: The model receives a compact Scene Graph Brief (listing objects, positions, and robot state) in every turn. For specific details, the model can issue ref-keyed queries (e.g., get_image, get_object_relations) to fetch omitted evidence on demand. This transforms fleeting sensor data into a stable, addressable symbolic representation.

B. Evaluation as Exit Codes (Restoring Verifiability)

To make outcomes verifiable, Thea implements an explicit Evaluator module that acts as a structural post-hook after every manipulation tool.

Independence: The evaluator is a separate component that does not see the model's reasoning or self-reports. It only observes the current camera view and the tool's specific post-condition.
Output: It returns a three-state verdict (Success, Failure, In Progress) along with a structured failure reason (e.g., "gripper closed without lifting," "robot too far").
Function: This verdict acts as the physical world's "exit code." If failure is detected, the agent uses the reason to adapt its next step (e.g., repositioning) rather than blindly retrying.

C. Embodiment Profile

To ensure portability, Thea uses an Embodiment Profile, a document describing the specific robot's physical limits (base footprint, reach, sensor modalities, camera positions). This allows the same harness logic to control different robot bodies (e.g., wheeled humanoids vs. dual-arm cobots) without architectural changes.

3. Key Contributions

Paradigm Transfer: Demonstrates that the "harness" paradigm, successful in coding agents, can be adapted for embodied agents by explicitly reconstructing world readability and outcome verifiability.
Scene Graph as Context: Introduces a persistent, symbolic world state that allows agents to reason about the physical environment as if it were a codebase, enabling long-horizon planning and recovery.
Evaluation as Exit Codes: Proposes a structural evaluation module that generates explicit success/failure signals and diagnostic reasons, closing the perception-action loop in environments that lack native termination signals.
Portability and Extensibility: Shows that the harness can orchestrate diverse policies and robot bodies (Astribot S1, AgileX Cobot Magic, Unitree G1) by treating capabilities as pluggable tools and bodies as configurable profiles.

4. Experimental Results

The authors evaluated Thea on real robots in real-world environments across three task levels of increasing complexity:

L1 (Short-horizon): Single pick-and-place.
L2 (Long-horizon): Multiple pick-and-place cycles on a single table.
L3 (Navigation + Manipulation): Navigating between rooms to retrieve and deliver objects.

Performance:

Success Rates: Thea achieved the highest success rates across all three levels compared to baselines including end-to-end VLA policies (ACT, $\pi0.5$ , LingBot-VLA-V2), Coding-as-Policy (CaP-X), and hierarchical planning (SayCan).
Scaling: While end-to-end policies and other baselines saw performance degrade significantly as task length increased (due to error compounding), Thea maintained high success rates. This is attributed to its ability to detect failures via the evaluator and recover via tool composition.
Evaluator Accuracy: The evaluator achieved 93.3% accuracy in classifying task states (Success, Failure, In Progress) across three different robot embodiments. The analysis showed that with an evaluator accuracy of $\alpha \approx 0.93$ and up to 5 retries per step, the theoretical success probability for a 5-step task jumps from 0.33 (open-loop) to 0.91 (harness).
Emergent Capabilities: The system demonstrated emergent behaviors such as active perception (searching cabinets when objects are missing from the graph), failure recovery (repositioning based on evaluator feedback), and user interaction (querying users for ambiguous targets).

5. Significance and Claims

The paper claims that embodied AI is undergoing a paradigm shift similar to software engineering, where the harness (infrastructure) is becoming as critical as the model itself.

Reliability as a Systems Problem: The authors argue that reliability in robotics can be engineered at the harness level (via evaluation and retry loops) without requiring perfect single-step policies or retraining model weights.
Additive Capability: New policies and new robot bodies can be integrated as tools and profiles without retraining the core orchestration logic, allowing policy builders and system builders to advance in parallel.
The Loop as a Training Target: The authors suggest that the behaviors of reading a scene graph, selecting tools, and recovering from failure are learnable behaviors, potentially becoming objectives for the next generation of embodied foundation models.

The paper concludes modestly, acknowledging limitations: the scene graph relies on the quality of underlying perception (positions are coarse), the evaluator is an approximation with false positives/negatives, and the system currently operates slower than real-time due to language model latency. However, it posits that these are engineering challenges solvable through better representations, compact local evaluators, and edge acceleration.