Technical Summary: Robotics Harness Optimization (RHO)

Problem Statement

Code-as-Policies (CaP) has demonstrated that Large Language Models (LLMs) can write executable code to compose perception, planning, and control primitives for robotics. However, current CaP systems rely heavily on multi-turn code-generation loops at test time to handle execution feedback, visual differences, and error correction. This approach is often infeasible for real-time robot control due to latency and the computational cost of scaling inference calls.

Conversely, Vision-Language-Action (VLA) models, which map pixels and language directly to actions, struggle with generalization. When faced with perturbed environments (e.g., changed object positions or task descriptions), VLAs like OpenVLA and $\pi_0.5$ suffer catastrophic performance drops (e.g., 0.0% and 12.83% success on LIBERO-PRO perturbations, respectively).

The core challenge is to develop a robot control paradigm that:

1. Avoids iterative LLM inference at deployment time (enabling real-time control).
2. Generalizes robustly to distribution shifts and perturbations where VLAs fail.
3. Produces interpretable, debuggable policies rather than black-box neural networks.

Methodology: Robotics Harness Optimization (RHO)

The authors introduce Robotics Harness Optimization (RHO), a paradigm that shifts the computational burden from test-time generation to training-time evolutionary search. Instead of optimizing a single prompt or function, RHO evolves multi-file neurosymbolic policy repositories (Repositories-as-Policies).

Core Components

1. HELIX (Hierarchical Evolution via LLM-Informed eXploration):

   • A reflective evolutionary optimizer that lifts the search space from single artifacts to whole multi-file repositories.
   • Mutator: A tool-enabled coding agent (e.g., Codex, Claude Code) that acts as the mutation operator. Within a single generation step, the agent can:
     • Edit multiple files in the repository.
     • Write and execute ephemeral tests.
     • Run candidate policies in the robot environment via a whitelisted interface.
     • Read tracebacks and stderr.
     • Query a visual-difference module (VDM) to compare scene states.
   • Reflection: The agent receives a reflection prompt summarizing failures and successes from the previous generation to guide the next mutation.

2. Repositories-as-Policies:

   • The candidate artifact is a deployable software system containing a task router, control routines, perception wrappers, grasp helpers, and, if applicable, the LLM harness.
   • These are neurosymbolic: symbolic code composes learned neural perception/grasping services with motion primitives.

3. Robot-Environments-as-a-Service (REaaS) & Isolation:

   • REaaS: Provides isolated, whitelisted access to robot environments (simulators) for the coding agent to execute candidates and gather feedback.
   • Isolation Mechanisms: To prevent reward hacking (e.g., agents hardcoding solutions or reading evaluator internals), RHO enforces strict boundaries. The mutator workspace is a sidecar proxy that excludes evaluator source code, benchmark splits, and judge prompts.

4. Optimization Loop:

   • Selection: Candidates are evaluated on a training minibatch. A child is accepted if it strictly improves the total shaped reward over the parent (or meets a relaxed $\ge$ threshold).
   • Pareto Frontier: Instead of a single chain of improvements, RHO maintains a Pareto frontier of candidates based on coverage dominance. A repository is retained if it is the best performer on at least one validation instance and not dominated by another surviving repository. This ensures diversity and prevents premature convergence.
   • Deployment: The final deployed policy is the repository with the highest mean validation reward from the terminal frontier. Crucially, no LLM code generation occurs at deployment time; the repository executes as a static Python controller (or a fixed harness for LLM-in-the-loop systems).

Key Contributions

1. Robotics Harness Optimization (RHO): A novel paradigm that evolves interpretable, neurosymbolic multi-file repositories rather than single prompts or functions, shifting optimization to training time to eliminate test-time inference loops.
2. Tool-Enabled Reflective Mutation: The first evolutionary optimizer where the mutator is a tool-enabled coding agent capable of interacting with the environment, reading tracebacks, and editing multiple files within a single mutation step.
3. Robust, Generalizable Policies: Demonstrating that RHO produces policies that outperform both multi-turn agentic systems and VLA baselines in single-turn evaluations, particularly under distribution shifts.

Experimental Results

The authors evaluated RHO on three benchmarks: CaP-Bench Robosuite, LIBERO-PRO, and Robotec's RAI O3DE.

1. CaP-Bench Robosuite (Manipulation)

• Setting: Single-turn execution (S4) with no LLM code-generation calls at deployment.
• Performance: RHO achieved a 70.0% success rate (490/700 trials).
• Comparison: This exceeds the strongest published multi-turn record (68.29% by CaP-Agent0, which required up to 66 LLM calls per trial). When CaP-Agent0 was restricted to single-turn execution, its performance collapsed to 24.0%.
• Efficiency: RHO achieved this with zero LLM calls during the control loop at deployment.

2. LIBERO-PRO (Generalization under Perturbation)

• Setting: Perturbed pick-and-place tasks (object/goal/spatial swaps and task description changes) where VLAs typically fail.
• Performance: RHO achieved 45.0% success (1351/3000 trials) after 63 generations of training on unperturbed data.
• Comparison:
  • 2.5$\times higher than the prior multi-turn CaP SOTA (CaP-Agent0 at 18.17%).
  • **3.5$\times** higher than$\pi_0.5$ (12.83%).
  • Significantly outperformed OpenVLA (0.0%).
• Significance: RHO generalizes to perturbed settings where VLAs collapse, without requiring demonstrations for the perturbed tasks.

3. RAI O3DE (LLM-in-the-Loop Deployment)

• Setting: A ROS 2 manipulation benchmark where the control loop includes an LLM.
• Performance: RHO evolved a multi-file repository of prompts, tools, and control code, improving held-out success from 23.5% to 44.3%.
• Efficiency: Achieved this with 20% less wall-clock time and 27% fewer tool calls compared to the baseline.
• Ablation: A single-file ablation (GEPA-style) increased tool calls by 16% and wall-clock time by 34%, demonstrating the superiority of the multi-file repository structure.

Structural Prior Analysis

Experiments comparing multi-file repositories against single-file stubs or rigid structural priors (Behavior Trees, Finite State Machines) showed:

• Multi-file repositories (70.0%) significantly outperformed single-file stubs (63.86%).
• Rigid priors (Behavior Trees: 33.86%, FSMs: 18.14%) collapsed, suggesting that the repository structure provides the right balance of scaffolding to guide the agent toward modularity without constraining the solution space.

Significance and Claims

The paper claims that RHO represents a fundamental shift in how robot policies are discovered:

• From Test-Time to Training-Time: By front-loading the search and refinement into training, RHO eliminates the latency and instability of iterative LLM inference during real-time control.
• Interpretability and Debuggability: Unlike end-to-end VLAs, RHO produces neurosymbolic code where advanced heuristics (e.g., quantile-robust geometry, IK-feasibility pre-gating, dual-perception cross-validation) are explicitly discovered and encoded. These mechanisms are human-readable and verifiable.
• Generalization: RHO demonstrates that evolutionary search over multi-file repositories can discover robust strategies that generalize to novel object arrangements and task descriptions, outperforming both current agentic stacks and VLA models.
• Scalability: The framework scales the "experience" gathered by an agent. While CaP-Agent0 gathers feedback turn-by-turn at deployment, RHO gathers this experience once during evolutionary search and compiles it into a static, efficient repository.

The authors conclude that while RHO inherits the spatial reasoning limits of the underlying LLM, it successfully recovers the modular structure necessary for task-and-motion planning, offering a path toward robust, interpretable, and deployable robot control.