Technical Summary: DeepInsight – A Unified Evaluation Infrastructure Across the Physical AI Stack

1. Problem Statement

Evaluating a "Physical AI" stack—defined here as an embodied humanoid system comprising semantic reasoning (System 2), visuomotor policy execution (System 1), and whole-body control (System 0)—presents a fundamental engineering challenge due to extreme heterogeneity. The operators within this stack differ by more than three orders of magnitude in episode length (from a single foundation-model decoding step to thousands of physics ticks), observation modalities (text, vision, audio, continuous physics state), reward semantics (exact match, model-based judgment, trajectory-analytic termination), and resource profiles (GPU-bound inference, I/O-bound sandboxing, compute-bound simulation).

Current evaluation practices rely on stitching together separate harnesses for each segment. While these individual harnesses preserve local validity, they lack a shared runtime, scoring identity, or trace continuity. This fragmentation destroys the ability to diagnose cross-layer regressions: a failure originating in a semantic planner that manifests as a control instability cannot be localized because the layers are evaluated in disjoint environments. No existing framework spans this full spectrum, as their abstractions are typically "regime-local," optimized for specific operator assumptions (e.g., short text episodes or long physics rollouts) and incompatible with others.

2. Methodology: The DeepInsight Architecture

DeepInsight is an evaluation infrastructure designed to serve the full Physical AI spectrum on a single runtime. Rather than forcing heterogeneity into a single homogeneous regime, it preserves the distinct characteristics of each layer behind three narrow, shared abstractions: Task, Resource, and Result.

2.1 The Task Abstraction

This abstraction absorbs heterogeneity in episode shape, modality, and reward semantics.

• Stateless Environments: Unlike conventional pipelines where state resides in the environment instance, DeepInsight makes environments stateless. All transient state (dataset samples, conversation history, in-flight results) lives on a per-episode handle carried by workers. This allows a single environment instance to serve arbitrarily many concurrent episodes of vastly different shapes (e.g., a single-step QA task and a multi-turn sandboxed agent).
• Interface: A uniform reset(handle) and step(handle, action) interface is implemented by all environments, from static decoders to physics simulators.
• Configuration vs. Code: Tasks are declared via configuration (suite and template configs) rather than code. The executed configuration is persisted as a snapshot artifact, ensuring reproducibility and allowing new benchmarks to be onboarded primarily through configuration changes.
• Orthogonal Judgment: Scoring is decoupled from task execution. Tasks define what is recorded; separate scorer components read the trace and emit rewards. This allows different judges (rule-based, model-based, trajectory-analytic) to be applied to the same task without modifying the runtime.

2.2 The Resource Abstraction

This abstraction absorbs heterogeneity in backend resource profiles and operational irregularity.

• Control Planes: Expensive resources (LLM inference and sandboxed runtimes) are detached from the main orchestrator and delegated to symmetric control planes (Inference Plane and Sandbox Plane).
• Handle Protocol: The orchestrator interacts with these planes via a narrow acquire(kind, episode_id, constraints) and release(handle) protocol. The orchestrator does not manage the internal lifecycle (autoscaling, container deployment, recovery) of these resources; it only manages the handles.
• Independent Concurrency: Stages are decoupled into independent queues with their own worker pools and concurrency budgets. This prevents slow stages (e.g., I/O-bound sandbox execution) from starving fast stages (e.g., GPU-bound generation) or vice versa.
• Symmetry: Both the inference plane (managing heterogeneous engine fleets like vLLM/SGLang) and the sandbox plane (managing stateless and stateful containers) expose the same interface to the orchestrator, hiding their internal operational complexity.

2.3 The Result Abstraction

This abstraction absorbs heterogeneity in event types across layers.

• Unified Trace Schema: Every event from every subsystem (runtime dialogue, judge rationale, inference events, lease events, trajectory steps) is written into a single structured record (TraceRecord).
• Identity and Lineage: Records share a hierarchical identity tuple (run_id, suite_id, task_id, episode_id, epoch) and causal lineage (event_id, parent_event_id). This enables cross-source analysis (e.g., joining inference latency with trajectory anomalies) as a simple database join or graph traversal.
• Trace as Artifact: The trace is the durable artifact, not the aggregate score. Aggregates are computed downstream from the trace. This allows for "drill-down" diagnostics long after a run completes, localizing regressions to specific events or resource leases.
• Decoupled Reporting: Reporters consume the trace off the hot path. Adding new analyses or metrics does not require modifying the evaluation pipeline.

3. Key Contributions

1. Unified Runtime for Heterogeneous Stacks: DeepInsight is the first infrastructure to carry the full Physical AI spectrum (from foundation-model decoding to whole-body control) on a single runtime without sacrificing benchmark fidelity or throughput.
2. Three Invariants: The system is built on three shared invariants (episode driver, resource-handle protocol, trace identity scheme) that allow new benchmarks to be onboarded largely by configuration.
3. Cross-Layer Diagnosis: By unifying the trace, DeepInsight enables the localization of regressions that begin in one layer and surface in another, a capability impossible for federations of separate harnesses.
4. Production-Grade Performance: The architecture supports near-linear scaling across nodes and reproduces peer-framework results within stated error budgets.

4. Experimental Results

4.1 System 2 (Foundation Model Evaluation)

DeepInsight was evaluated against mature open-source frameworks (lm-eval, Inspect AI, VLMEvalKit, lmms-eval) on text, multimodal VQA, and omni-modal benchmarks using the Qwen3.6-27B and Qwen3-32B models.

• Accuracy Alignment: DeepInsight reproduced reference scores and peer-framework readings within the residual spread of the peer runs themselves. It was closest to the reference on the majority of benchmarks across text (5/8, 4/9), multimodal VQA (8/15), and omni-modal (4/5) categories.
• Throughput: On a single 8×A100 node, DeepInsight outperformed peers:
  • 1.29× faster than lm-eval (due to asynchronous pipelining reclaiming idle time from variable-length episodes).
  • 1.13× faster than Inspect AI (due to stage decoupling preventing shared concurrency bottlenecks).
  • 1.03–1.04× faster than VLMEvalKit/lmms-eval (where prefill-bound workloads offered less room for pipelining gains).
• Scaling: DeepInsight demonstrated near-linear scaling across nodes (2.03× speedup on 2 nodes, 4.00× on 4 nodes) for a 27-suite workload, a capability peer frameworks lack.

4.2 System 1 and System 0 (Embodied Evaluation)

• System 1 (Visuomotor): The infrastructure successfully drove closed-loop simulations for navigation (VLN-CE), manipulation (LIBERO), and motion generation. It supported both objective metrics and subjective human-preference evaluations (audio-conditioned motion) within the same trace schema.
• System 0 (Whole-Body Control): DeepInsight enabled a "trace-backed" release workflow. While aggregate metrics (Success Rate, MPJPE) nominated a policy, the trace allowed for a behavior-level diagnostic that identified local failures (e.g., hip-joint dynamics, contact stability) that blocked deployment despite a high aggregate score.
• Full-System Validation (System 2–1–0): A composed vehicle-guide task demonstrated cross-layer failure localization.
  • Results: 60.4% end-to-end success rate across 96 episodes.
  • Failure Analysis: The largest failure locus was System 1 execution (42.1%), followed by system-boundary handoffs (28.9%). The unified trace allowed the system to attribute failures to specific causes (e.g., "Navigation-to-locomotion executability mismatch") that would be invisible in isolated evaluations.

5. Significance and Claims

The paper claims that DeepInsight addresses a structural gap in Physical AI evaluation: the lack of a substrate that can absorb the full spectrum of operator heterogeneity without forcing reimplementation or losing cross-layer visibility.

• Diagnostic Payoff: The primary value is diagnostic. Because every layer writes to a shared trace, a regression surfacing in one layer can be traced back to its origin in another. This "cross-layer payoff" is unachievable with a federation of per-segment harnesses.
• Production Viability: The system is not a theoretical prototype; it is deployed in production across all three layers of an embodied humanoid stack. It onboards new benchmarks largely via configuration and scales efficiently.
• Modest Scope: The authors explicitly state that DeepInsight is an evaluation substrate for a specific embodied humanoid stack, not a universal definition of Physical AI. The System 1 and System 0 results are presented as demonstrations of coverage and extensibility rather than competitive benchmarks, with quantitative depth for these layers left to future work. The paper also notes that all rollouts reported are simulated, acknowledging the sim-to-real gap as a remaining uncertainty.

In summary, DeepInsight provides the "shared identity" necessary to diagnose complex, multi-layered physical AI systems, replacing the patchwork of incompatible tools with a unified, scalable, and trace-centric infrastructure.