🤖 AI

DeepInsight II: One Trace from Benchmark to Robot

DeepInsight II extends the unified evaluation framework of its predecessor to the embodied AI layer by quantifying navigation, manipulation, and whole-body control performance across simulation and real-robot trials, thereby establishing a continuous, repair-oriented diagnostic pipeline that bridges the sim-to-real gap through shared trace identities.

Original authors: Siyi Li, Yuchen Kang, Wuliang Wang, Zhengjie Zhang, Jiangpin Liu, Jianhao Yao, Jie Chen

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siyi Li, Yuchen Kang, Wuliang Wang, Zhengjie Zhang, Jiangpin Liu, Jianhao Yao, Jie Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The field of physical artificial intelligence seeks to build machines that can move through the real world, not just think inside a computer. To do this, engineers stack different layers of intelligence on top of one another. At the very top sits a reasoning layer, like a brain that understands language and plans complex tasks. Below that are skill layers that handle specific actions, such as walking to a door or picking up a cup. At the very bottom is a control layer that manages the robot's muscles and joints to keep it balanced. For years, scientists have been able to test the top reasoning layer with great precision using standard tests and shared data. However, testing the lower layers that actually touch the physical world has been much harder. Each lab often builds its own unique testing ground, using different robots and different rules, making it difficult to compare results or know if a robot will truly work when it leaves the lab.

A team at XPENG Robotics has addressed this fragmentation by creating a unified way to test these lower layers, bridging the gap between computer simulations and real-world hardware. Their new report, DeepInsight II, introduces a system that allows researchers to run the exact same test on a virtual robot and a physical robot, treating both as part of a single, continuous experiment. Instead of building a new type of robot or a new algorithm, they built a better measuring stick. This system connects the reasoning brain, the skill layers, and the physical controller into one cohesive unit, allowing them to trace a failure from the moment a robot trips on the floor all the way back to the specific decision that caused the stumble.

The researchers began by testing how well their system could replicate existing benchmarks for navigation and manipulation. They took released versions of robot brains and tested them against standard tasks, such as following instructions to walk through a house or picking up objects in a simulated kitchen. The results showed that their unified system could reproduce the scores of these standard tests with high accuracy, proving that their new measuring stick did not distort the data. This was a crucial first step, confirming that the new infrastructure could handle the complex, messy reality of different robot designs and software versions without breaking the rules of the original tests.

Next, the team turned their attention to the robot's whole-body control, the layer responsible for keeping the machine upright and moving smoothly. They gathered four different control systems from various research groups and ran them through a standardized set of motion tasks in a high-fidelity simulation. They found that while all the systems could perform the tasks, some were significantly better at avoiding falls and maintaining precise movement than others. One system, in particular, stood out as the most reliable, successfully completing nearly all the motions with high precision. This simulation phase allowed them to screen out weaker candidates quickly and safely before ever risking a physical machine.

The most significant leap occurred when they moved these qualified systems from the computer screen to a real robot. They selected two versions of the same control system and ran them through matched trials, where the virtual robot and the physical robot attempted the exact same motions at the same time. By linking these two runs under a single identity, they could compare the results directly. They discovered that while the physical robot made slightly more errors than the virtual one, the ranking of the systems remained the same. The system that performed best in the simulation also performed best on the real floor. This confirmed that the simulation was a trustworthy predictor of real-world performance, provided the testing conditions were tightly controlled.

Beyond just measuring success, the researchers developed a method to diagnose exactly why a robot failed. When a robot stumbles or drops an object, it is often unclear whether the mistake came from a bad plan, a clumsy movement, or a misunderstanding of the environment. The team created a diagnostic tool that breaks down a failure into specific categories. They found that some failures happened because the robot's different layers were not speaking the same language; for example, the navigation layer might stop when it reached a spot, but the greeting layer required the robot to be facing a specific direction, a detail the first layer had ignored. Other failures happened because the robot simply could not physically move into the required position, no matter how perfect the plan was.

By applying this diagnostic tool to both simulated and real-world runs, the team showed that they could pinpoint the exact cause of a failure. In one real-world example, a robot failed to greet a person because its internal map drifted slightly, causing it to stop just out of reach. The diagnostic system identified this as a boundary error, where the robot's definition of "arrived" did not match the physical reality of the situation. This level of detail allows engineers to fix the specific interface or rule that broke, rather than guessing which part of the robot needs to be rebuilt.

The work demonstrates that the gap between simulation and reality can be narrowed not by making better simulations, but by creating a shared language for evaluation. By treating the virtual and physical worlds as two branches of the same experiment, the researchers proved that they could carry the same test from a computer to a real robot without losing the ability to compare results. This approach does not solve every problem in robotics, but it provides a clear, reliable way to measure progress and understand exactly where and why a robot fails. It turns the complex, opaque process of robot testing into a transparent, traceable journey, giving engineers the confidence to move their creations from the lab into the real world.

Technical Summary: DeepInsight II: One Trace from Benchmark to Robot

Problem Statement

The evaluation of Physical AI systems suffers from a structural fragmentation that inversely correlates with deployment risk. While foundation models (System 2) benefit from mature, standardized evaluation harnesses, the embodied layers critical for actual deployment—navigation/manipulation (System 1) and whole-body control (System 0)—remain siloed. These layers are evaluated across disparate simulators, embodiments, and interfaces, where changing a single variable (task, architecture, or execution substrate) typically alters the evaluation harness and result representation.

This heterogeneity prevents direct comparison and obscures the "sim-to-real" gap. Current infrastructure efforts often federate benchmark servers but leave individual episode loops and result representations intact, joining real-robot evaluation only post hoc if at all. Consequently, aggregate success metrics often hide local failures and simulation-reality discrepancies that govern deployment. The core challenge is not to erase this fragmentation with a universal benchmark, but to make the heterogeneity explicit while preserving comparable evaluation semantics across the entire stack.

Methodology

DeepInsight II extends the substrate established in DeepInsight v1, which unified evaluation behind three abstractions: Task (reset/step driver), Resource (acquire/release protocol), and Result (identity-addressed trace). This paper does not introduce new abstractions but applies the existing substrate to embodied layers and physical hardware.

1. Unified Runtime and Backend Architecture

The system treats simulators, policies, and real robots as interchangeable backends attached to a single control plane:

Simulators: Third-party stacks (Habitat, MuJoCo/robosuite, SAPIEN/ManiSkill) are sandboxed as container images behind a handle, maintaining dependency isolation while sharing a unified lease lifecycle and parallelism.
Policies: Models enter as inference backends (observation-in, action-out) registered in a service catalog, allowing configuration-based substitution without integration overhead.
Real Robots: The physical robot is treated as an asymmetric backend. It is leased per episode, bound to the episode identity, and its events (including human-gated resets and emergency stops) are recorded as first-class trace events.
Trace Identity: A matched pair of rollouts (simulated and physical) shares a single parent trace identity. They hang as sibling branches, retaining execution-domain-specific records (e.g., execution_domain = simulation vs. real_robot) while sharing the same task and metric contract. This allows cross-domain reduction to be a "join" on the parent identity rather than a reconciliation across toolchains.

2. System 1: Native-Protocol Fidelity

The paper evaluates released checkpoints across six benchmarks under their native protocols (task splits, reset distributions, success predicates, and interfaces) using the shared DeepInsight driver:

Navigation: R2R-CE and RxR-CE in Habitat-Sim.
Manipulation: LIBERO, LIBERO-Plus, SimplerEnv (WidowX/Bridge), and RoboCasa365.
Goal: To verify that the shared runtime preserves the readings of original evaluation stacks without introducing systematic shifts.

3. System 0: MotionBench and Sim-to-Real Correspondence

To standardize Whole-Body Control (WBC) evaluation, the authors introduce MotionBench:

Frozen Contract: A curated, versioned workload (Core1000, 1,000+ motion clips) and a fixed metric contract (Success Rate, MPJPE-l, progress-soft) are applied across heterogeneous WBC models (HoloMotion, SONIC).
Hardware Qualification: Models are admitted to real-robot trials only after achieving 100% simulation success rate and passing trace-derived checks.
Matched Trials: Qualified models undergo matched trials on a Unitree G1 robot. Simulation and physical rollouts share the same motion, reference, checkpoint, and body set, allowing for a direct comparison of kinematic and dynamic profiles (e.g., joint position RMSE, acceleration, jerk) under a single trace identity.

4. Full-System Diagnosis: Handoff Attribution

The paper extends trace localization to a composed System 2–1–0 episode, introducing an automatic attribution mechanism for failures at system handoffs:

Attribution Logic: Failures are classified into five labels mapped to concrete repair actions: Termination Error (TE), Handoff-Condition Violation (HCV), Contract Gap (CG), Skill-Internal Execution (SIE), and Execution-Time Invariant Lapse (EIL).
Repairability Criterion: A quantitative measure distinguishes between Boundary-definition problems (missing interface conventions, repairable by adding conditions) and Executability problems (feasible entry regions narrower than the minimum executable step, requiring action space changes). This is determined by measuring the ratio of the feasible entry region width to the minimum executable step.

Key Results

System 1: Benchmark Fidelity

Navigation: DeepInsight II results closely match published references. On R2R-CE, differences in bounded metrics were within 0.55–3.81 percentage points; on RxR-CE, differences were within 1.27 points. Residual gaps were attributed to stochasticity and implementation-level differences (e.g., image preprocessing) rather than the runtime itself.
Manipulation: Across LIBERO, SimplerEnv, and RoboCasa365, the unified execution tracked published references without systematic upward or downward shifts.
- Exception: A discrepancy was noted for the $\pi_0$ model under Sensor Noise in LIBERO-Plus (DI: 18.7% vs. Ref: 79.0%). Independent evaluations confirmed the 16–20% range, suggesting the original reference may be an unresolved benchmark issue.

System 0: WBC Evaluation and Sim-to-Real

Simulation Ranking: HoloMotion 1.3.2 led the benchmark with 96.7% SR and 4.53 cm MPJPE-l. SONIC variants showed lower progress-soft scores, revealing larger separation when accounting for partial completion and drift.
Sim-to-Real Gap: In matched trials on the Unitree G1, hardware execution increased tracking errors and shifted joint dynamics.
- MPJPE-l: Increased by +10.83% (SONIC release) and +18.65% (SONIC low-latency).
- Joint Dynamics: Hardware reduced acceleration/jerk RMS (smoother motion) but increased upper-tail peaks (P95), indicating higher transient stress.
- Consistency: The numerical ordering of models was preserved across simulation and hardware, validating the standardized contract.

Full-System: Attribution and Repairability

Attribution Accuracy: In simulation, the attribution mechanism correctly identified failure causes, distinguishing between boundary errors (HCV) and contract gaps (CG).
Repairability: The study confirmed that H2 (navigation to greeting) was a boundary-definition problem (missing orientation criterion) repairable by adding a condition. H5 (navigation to presentation) was an executability problem where the entry region was too narrow for the step size, making it unrepairable via interface changes alone.
Physical Validation: On the real robot, the diagnosis reproduced simulation failure classes. In one case, odometry drift consumed the "slack" budget, causing a failure that the simulation criterion had predicted as repairable, demonstrating the criterion's ability to quantify hardware limitations.

Significance and Claims

The paper explicitly claims not to offer a new evaluation architecture, new policies, or a universal benchmark. Instead, its contribution is empirical continuity:

Unified Substrate: It demonstrates that embodied heterogeneity can be managed without a fourth abstraction, using the existing Task/Resource/Result substrate to bind simulators and physical robots.
Native Fidelity: It proves that a shared runtime can reproduce native benchmark results across diverse simulators and embodiments.
Matched Evidence: It establishes a method where simulated and physical rollouts share a parent trace identity, making the sim-to-real gap a "native reduction" rather than a post-hoc reconciliation.
Actionable Diagnosis: It provides a mechanism to convert implicit skill-boundary assumptions into inspectable contracts, distinguishing between errors fixable by interface adjustments and those requiring fundamental changes to the action space.

The work positions itself as a bridge between benchmark execution and matched robot evidence, enabling repair-oriented diagnosis grounded in hardware-observable states.