💻 computer science

Addressing the Orchestration Gap in Generalist Robots via Physical Agency

This paper introduces "Pigey," a high-level orchestrator that bridges the "orchestration gap" by decomposing complex tasks into subgoals and managing existing frozen vision-language-action policies through a closed-loop physical agency, thereby achieving significant performance improvements on reasoning-heavy robotic tasks without requiring additional data or fine-tuning.

Original authors: Liane Galanti, Dhruv Shah, Tri Dao

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liane Galanti, Dhruv Shah, Tri Dao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to be a helpful roommate. You don't just want a machine that can move its arm to pick up a cup; you want a roommate who can figure out why it's picking up the cup, notice if the cup is hidden under a book, realize that a bottle of glue is dangerous for a toddler, and remember where it put the toys after the room gets messy. This is the world of "generalist robots," a field where scientists are trying to build machines that can do almost anything a human can do in a home, not just one specific job.

To do this, researchers usually rely on two main ingredients. First, there's perception and control, which is like the robot's eyes and muscles—teaching it how to see an object and how to grab it without dropping it. Second, there's reasoning, which is the robot's brain—teaching it to understand instructions like "put the safe things on the plate" or "find the toy that is hiding." For a long time, the big idea in robotics was to smash these two ingredients together into one giant, super-smart brain. The hope was that if you showed the robot enough videos of people doing tasks, it would learn to see, think, and act all at once. But as the paper suggests, this "all-in-one" approach often hits a wall. The robot might be great at moving its arm but terrible at figuring out that the doll is actually under the box, or it might fail to realize it dropped the toy and needs to try again.

This brings us to the paper's big idea: instead of forcing one giant brain to do everything, what if we gave the robot a manager? The authors, Liane Galanti, Dhruv Shah, and Tri Dao, propose a system called Pigey (Physical Agency). Think of Pigey not as a new muscle or a new brain, but as a project manager who stands between the robot's "brain" (a pre-trained AI) and its "muscles" (its physical actions).

Here is how Pigey works: Imagine you ask your robot to "Pick up the doll and put it in the basket." A standard robot might look at the scene, see a box, and blindly try to grab the box, failing because the doll is hidden underneath. Pigey, however, acts like a detective. It looks at the scene, thinks, "Wait, I don't see the doll. It must be hidden," and then commands the robot to move the box. Once the doll is revealed, Pigey says, "Okay, now grab the doll." If the robot drops the doll, Pigey notices, says, "Oops, that failed," and tells the robot to try again. Crucially, Pigey doesn't need to learn how to grab or move; it just uses the robot's existing skills but directs them with a smart, step-by-step plan.

The researchers tested this by taking two "frozen" (meaning untrained and unchangeable) robot skills—one good at picking up hard objects and one good at handling soft, tricky things—and letting Pigey orchestrate them. They didn't teach the robot anything new; they just changed how the robot was told to act. The results were surprising. On a tough simulation test called LIBERO-PRO, the standard robot only succeeded 12.8% of the time. But when Pigey took the wheel, the success rate jumped to 53.3%—more than four times better. On real-world tasks involving tricky reasoning, like figuring out which items are safe for a child or clearing a table for a vegetarian guest, the standard robot often failed completely (near 0% success), while Pigey got it right over 90% of the time.

The paper argues that the problem wasn't that the robot's muscles were weak or that it needed more data to learn how to move. The problem was an "orchestration gap." The robot had the skills, but it lacked the manager to tell it when to use them, how to check if they worked, and what to do when things went wrong. By adding this layer of high-level planning and verification, the authors show that we can make existing robots much smarter without the expensive and time-consuming process of collecting thousands of new videos to retrain them. It's a reminder that sometimes, the best way to upgrade a robot isn't to build a bigger brain, but to give it a better boss.

Technical Summary: Addressing the Orchestration Gap in Generalist Robots via Physical Agency

Problem Statement

General-purpose robotics requires a full-stack integration of perception, world knowledge, intent reasoning, planning, success detection, recovery, and low-level control. Current state-of-the-art Vision-Language-Action (VLA) models attempt to compress all these capabilities into a single learned policy via large-scale pre-training. However, this approach often fails on tasks requiring abstract reasoning (e.g., negation, conditionals, safety constraints) or closed-loop recovery (e.g., handling grasp failures or hidden objects).

The authors identify that when a policy fails on complex instructions, the bottleneck is rarely the low-level motor control itself, but rather the lack of a process to decompose goals, verify outcomes, and recover from errors. Existing solutions either rely on scaling robot data (which is expensive and indirect for teaching reasoning) or use symbolic planners that lack robustness in real-world, pixel-conditioned environments. The paper posits that a significant "orchestration gap" exists: frozen motor skills often possess the necessary low-level capabilities but fail to execute them correctly due to a lack of high-level orchestration.

Methodology: Pigey (Physical Agency)

The authors propose Pigey, a closed-loop inference-time orchestrator that augments existing frozen motor policies without additional data collection or post-training. Pigey acts as a high-level agent manager that decomposes abstract instructions into executable subgoals, selects appropriate backends, verifies outcomes, and initiates recovery.

System Architecture

Pigey operates as a frontier Vision-Language Model (VLM) running a closed loop over two frozen, complementary motor backends:

TAMP Backend (TiPToP): A Task and Motion Planning system used for precise pick-and-place of rigid objects. It provides geometric precision and explicit grasp signals.
VLA Backend ( $\pi0.5$ ): A pre-trained VLA policy used for deformable objects, contact-rich interactions, and recovery actions where geometric planning is unreliable.

The Orchestration Loop

The agent $\phi$ maps observations, history, and instructions to tool calls. The loop consists of four stages:

Perceive: The agent observes the scene, receiving camera views, robot state, and object labels from an open-vocabulary detector.
Reason: The agent decomposes the high-level instruction into short, concrete subgoals (e.g., "pick the red cup") and selects the appropriate backend (TAMP for rigid, VLA for deformable/recovery).
Act: The agent calls the selected backend with the subgoal. The backend executes the motion and returns control.
Verify & Recover: This is the critical differentiator. The agent verifies the outcome using:
- Deterministic Signals: Gripper width sensors (is_grasped flag) and explicit failure flags from the backend.
- Visual Signals: Post-action wrist images to confirm the object was grasped or placed correctly.
- Recovery Logic: If verification fails (e.g., a slip, a hidden object, or an occupied destination), the agent does not proceed. Instead, it re-perceives the scene, re-plans, retries the action, or escalates to the alternative backend (e.g., switching from TAMP to VLA if a grasp fails).

Crucially, the agent never emits raw motor commands; it delegates all motion to the frozen backends, intervening only to decide what to do next and whether the previous step succeeded.

Key Contributions

Full-Stack Framing of Robot Generalization: The paper reframes robot generalization as a problem where perception, reasoning, planning, and control must work together. It argues that failures are often attributable to missing orchestration components (decomposition, verification, recovery) rather than a lack of motor data.
Pigey Framework: A closed-loop orchestrator that supplies the task-level stack (planning, skill selection, verification, recovery) over frozen VLA policies and parametrized skills. It achieves this with zero additional data collection or post-training.
Bridging the Orchestration Gap: The authors demonstrate that the same frozen motor skills achieve significantly higher success rates when driven by Pigey compared to direct prompting. This gap is most pronounced in reasoning-limited tasks (world knowledge, conditionals, multi-step logic) rather than motor-bound tasks.

Experimental Results

The system was evaluated on 30 real-world tasks using a Franka FR3 robot (DROID platform) and the LIBERO-PRO simulation benchmark.

Real-World Performance (DROID)

Reasoning-Limited Tasks: On tasks requiring world knowledge, conditional logic, multi-step reasoning, and safety reasoning, the raw $\pi0.5$ VLA averaged near 0% success. Pigey lifted this to 90–100%.
Simple Control: For simple pick-and-place tasks, the raw VLA achieved 95%, while Pigey achieved 100%, demonstrating that orchestration also improves reliability on standard tasks by detecting and retrying failed grasps.
Overall: Across 30 tasks, Pigey achieved a 97.3% success rate compared to 16.7% for the raw VLA and 48.7% for the open-loop TAMP baseline.

Simulation Performance (LIBERO-PRO)

On the LIBERO-PRO benchmark, which tests robustness to perturbations (object swaps, goal swaps, spatial changes), the frozen $\pi0.5$ -LIBERO policy achieved a mean success rate of 12.8%.
Pigey improved this to 53.3% (a 4× increase) without any task-specific fine-tuning or changes to policy weights.
The system outperformed other baselines like CaP-Agent0 (18.2%) and raw $\pi0$ (0%).

Significance and Claims

The paper claims that a substantial portion of what appears to be a "missing motor capability" in generalist robots is actually a missing orchestration capability. By decoupling high-level reasoning from low-level control, the authors show that:

Inference-time orchestration is a powerful lever: It can recover capabilities that would otherwise require expensive robot data collection and policy retraining.
The "Orchestration Gap" is real: The difference between what frozen skills can achieve alone versus within an agentic loop is significant, particularly for tasks involving negation, safety, and recovery.
Closed-loop verification is essential: Breaking open-loop plans into verified steps (grasp $\to$ verify $\to$ place) allows the system to handle dynamic changes and failures that blind planners cannot.

The authors conclude that before investing in new robot data to teach reasoning, researchers should first measure how much capability can be recovered by wrapping existing frozen policies in a robust inference-time agent. While Pigey introduces latency and cost due to API calls to frontier models, it offers a path to generalization that avoids the prohibitive costs of scaling robot interaction data.