💻 computer science

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Harness VLA is a memory-augmented agentic framework that enhances the reliability of frozen Vision-Language-Action models in complex manipulation tasks by treating them as retryable contact-rich primitives and composing them with a fixed library of analytic primitives, thereby achieving significant performance gains on benchmarks like LIBERO-Pro and RoboCasa365 without requiring model finetuning.

Original authors: Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li, Zhihao Liu, Chunyang Zhu, Jiaxing Qiu, Yuchen Yan, Jiyuan Liu, Wenhao Tang, Zhengru Fang, Yi Nie, Changxu Wei, Yu Wang, Wenbo Ding, Chao Yu

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li, Zhihao Liu, Chunyang Zhu, Jiaxing Qiu, Yuchen Yan, Jiyuan Liu, Wenhao Tang, Zhengru Fang, Yi Nie, Changxu Wei, Yu Wang, Wenbo Ding, Chao Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot brain that is amazing at one specific thing: grabbing weirdly shaped objects, turning sticky knobs, or pressing buttons. It's like a world-class chef who can chop vegetables and sear steak perfectly. But if you ask this chef to plan a whole dinner party, navigate a crowded kitchen, or figure out where the salt is when the table has been moved, they get totally lost. They might try to chop the salt shaker because they were trained on a specific table layout, or they might freeze when the instructions change slightly.

This is the problem with current "Vision-Language-Action" (VLA) robots. They are great at the "contact" part (touching and moving things) but terrible at the "planning" part when things get messy or different from their training.

Enter Harness VLA. Think of this not as a new robot brain, but as a brilliant project manager who takes over the job of running the kitchen.

The Team-Up: The Manager and the Specialist

In this new system, the robot brain (the VLA) is "frozen." That means we don't retrain it or teach it new tricks. It stays exactly as it is, a specialist in contact-rich tasks. The Harness VLA system wraps a "manager" (an AI agent) around this specialist.

Here is how they work together:

The Manager (The Planner): This is the boss. It looks at the task (e.g., "Put the milk in the fridge"), checks the room, and breaks the job down into tiny, logical steps. It uses a small, fixed set of "analytic" tools—like a GPS for moving the arm, a simple command to open the gripper, or a move to rotate the wrist. These tools are like a robot's basic reflexes: predictable, reliable, and perfect for moving through empty space.
The Specialist (The Frozen VLA): The manager only calls the specialist when things get tricky. When the robot needs to actually grab a slippery milk carton, turn a faucet, or press a button, the manager says, "Okay, specialist, you're up!" The specialist does its magic for a few seconds, then hands control back to the manager.

The Secret Sauce: A Memory Notebook

The real magic isn't just the team-up; it's the Memory.

Imagine the manager has two notebooks:

The Task Notebook: After the robot successfully solves a problem once, the manager writes down the sequence of steps it took. But here's the cool part: instead of writing down exact coordinates like "move to x=1.2, y=0.5," it writes down logic like "move to the red bowl." This way, if the bowl is in a different spot tomorrow, the manager can still use the same plan, just re-aiming the steps to the new location.
The Global Notebook: This is a collection of "rules of thumb" and "lessons learned" from many different tasks. It contains notes like, "If the gripper closes but the object doesn't move, it's a fake grab—try again," or "Always check the success signal before celebrating."

When a new task comes in, the manager checks these notebooks. If the robot fails, the manager doesn't just give up. It reads the "failure rules," adjusts the plan, re-positions the robot, and tries the specialist's help again. It's like a human learning to ride a bike: you fall, you remember what went wrong, you adjust your balance, and you try again.

What This System is NOT

The paper is very clear about what this system doesn't do. It explicitly argues against two common ideas:

It does NOT try to make the robot brain bigger or smarter. The authors didn't train a new, massive AI model to do everything. They kept the brain frozen and small.
It does NOT give the robot an infinite library of new skills. The manager doesn't invent new tools on the fly. It uses a small, fixed set of tools (Move, Rotate, Grab, Release) and learns how to combine them perfectly.

The paper suggests that trying to make one giant AI do everything (planning, moving, and grabbing) is actually the problem. By splitting the job, the system becomes much more reliable.

The Results: Does it Work?

The authors tested this system in several virtual worlds, from simple tabletop games to complex kitchen simulations. The results were quite impressive:

On a standard test called LIBERO-Pro, where the instructions were changed or objects were moved to new spots, the Harness VLA system succeeded 38.6 percentage points more often than the best previous methods.
On a kitchen simulation called RoboCasa365, it improved success rates by 25.4 percentage points.
On a dual-arm robot test called RoboTwin, it reached a success rate of 58.4%.

The paper shows that by letting a smart manager handle the planning and memory, and only using the frozen "specialist" brain for the actual grabbing, the robot can handle messy, real-world changes much better than before. It's a reminder that sometimes, the best way to build a super-robot isn't to give it a bigger brain, but to give it a better manager.

Technical Summary: Harness VLA

Problem Statement

Language-conditioned robotic manipulation requires a system capable of both precise, contact-rich control and robust reasoning over language, scenes, and long-horizon tasks. Current approaches face a fundamental trade-off:

End-to-End Vision-Language-Action (VLA) Models: These models excel at local, image-conditioned contact tasks (e.g., grasping irregular objects, turning faucets) but are trained on in-distribution trajectories. They often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, or when short skills must be composed into longer routines. A single unstable contact failure can derail an entire monolithic rollout.
LLM Coding Agents: These agents provide strong semantic and compositional reasoning by synthesizing executable programs over perception and control APIs. However, purely analytic primitives (e.g., inverse kinematics transport, gripper control) struggle with irregular grasping, constrained placement, and articulated-object interaction. Furthermore, scaling these systems often requires expanding the skill library, forcing the agent to validate the safety and reusability of newly generated skills.

The core challenge is to combine the local visuomotor strengths of VLAs with the compositional reasoning of agents without requiring finetuning of the VLA or expanding the primitive vocabulary at deployment time.

Methodology: Harness VLA

The authors propose Harness VLA, a memory-augmented agentic framework that treats a frozen VLA not as a monolithic policy, but as a retryable, contact-rich primitive within a fixed library of analytic controllers.

1. System Architecture

The system operates via a turn-based, autoregressive interaction between a high-level Agentic Planner ( $\Pi$ ) and a physics engine.

Fixed Primitive Library ( $\mathcal{P}$ ): The planner is restricted to a small, fixed set of primitives. This includes:
- Analytic Primitives: Deterministic, model-based controllers for non-contact tasks (e.g., MOVE TO, ROTATE WRIST, SET GRIPPER, NAVIGATE TO). These handle free-space transport, staging, posture adjustment, and release.
- VLA Primitive (VLA ACT): A single learned primitive that encapsulates a frozen, pretrained VLA. It is invoked only for local, contact-rich phases (e.g., grasping, insertion, fixture actuation).
Memory Modules:
- Task Specific Memory: Stores successful primitive composition traces (JSONL format) from a reference seed. These traces are parameterized (replacing concrete coordinates with symbolic queries) to allow few-shot re-grounding in new spatial layouts.
- Global Memory: Stores reusable success rules and failure models (e.g., "avoid empty grasps," "re-stage if contact fails") distilled from exploration.

2. Execution Loop

The framework follows a Read-Eval-Print Loop (REPL) style execution:

Exploratory Bootstrapping: On a reference seed, the planner autonomously discovers a solution by iterating through primitive compositions. It uses a RESET primitive to retry failures. Upon success, it abstracts the trace into Task Specific Memory and extracts heuristics for Global Memory.
Deployment Evaluation: On unseen perturbations (e.g., position swaps, instruction redirections), the planner retrieves the Task Specific Memory trace. It dynamically re-grounds the trace using live RGB-D observations and consults Global Memory to avoid known failure modes. The planner orchestrates the sequence: using analytic primitives for staging and transport, and invoking VLA ACT only when the robot is in a favorable local configuration for contact.

3. Key Design Principles

Asymmetric Delegation: Semantic grounding, long-horizon composition, and scene-level reasoning are handled by the planner. The frozen VLA is strictly reserved for local contact-rich operations where its training distribution is informative.
Retryable Contact: The planner treats VLA ACT as a local attempt. If a contact phase fails (e.g., unstable grasp), the planner can re-stage the robot (change approach pose or viewpoint) and retry the VLA invocation, rather than failing the entire episode.
No Finetuning: The VLA backbone remains frozen. The system extends the VLA's operating range by changing when and how it is invoked, not by updating its weights.

Key Contributions

Memory-Augmented Agentic Framework: A novel architecture that exposes a frozen VLA as a primitive within a fixed library, enabling long-horizon, perturbed manipulation without VLA finetuning or vocabulary expansion.
Empirical Analysis of Fixed Primitives: Demonstrates that a small, fixed primitive library is sufficient when the planner learns to orchestrate it. Repeated planner-staged invocations can reframe brittle VLA attempts, while analytic primitives solve the non-contact structure.
Robust Benchmark Performance: The framework achieves state-of-the-art results across standard and heavily perturbed benchmarks, significantly outperforming direct VLA baselines and prior agentic methods.

Experimental Results

The authors evaluated Harness VLA on four benchmark families: LIBERO, LIBERO-Pro, RoboCasa365, and RoboTwin C2R.

Standard LIBERO: Harness VLA (using a Codex or Claude Code planner) achieved a 96.0% aggregate success rate, preserving competitive performance with the underlying frozen VLA (95.3%) while enabling perturbation robustness.
LIBERO-Pro (Perturbed): Under instruction-redirection and position-swap perturbations, Harness VLA significantly outperformed the strongest baseline (RATS).
- Harness VLA (CC): 82.4% overall success.
- Harness VLA (Codex): 72.1% overall success.
- Improvement over RATS (43.8%): +38.6 percentage points.
- Improvement over the frozen RLinf baseline (50.0%): +32.4 percentage points.
RoboCasa365 (Household Kitchen): Harness VLA improved over the frozen RLDX-1 baseline (30.0%) to 55.4% (Codex) and 48.6% (CC), a gain of +25.4 percentage points.
RoboTwin C2R (Bimanual Zero-Shot): In a clean-to-randomized transfer setting, Harness VLA raised the success rate of the frozen LingBot-VLA backend (50.4%) to 58.4%.

Mechanism Analysis:

Semantic Re-grounding: The planner explicitly parses task descriptions and re-binds targets, preventing the VLA from repeating training-time behaviors when instructions change.
Adaptive Invocation: Success rates rise rapidly with the first few planner-staged VLA invocations, demonstrating that retrying contact attempts after re-staging is critical for robustness.
Division of Labor: Analytic primitives handle the majority of non-contact transport and staging (e.g., 84.2% of calls in LIBERO), reserving the VLA for the specific contact-rich phases where it is most effective.

Significance and Claims

The paper claims that Harness VLA demonstrates that pretrained VLAs are most effective when isolated to contact-rich visuomotor control, while abstracting semantic and spatial bindings to a higher-level agent prevents the catastrophic failures common in monolithic deployments.

The authors argue that their approach offers a practical path to robust manipulation without the computational cost of finetuning large VLA models for every new task or environment. By treating the VLA as a "contact specialist" and the agent as the "orchestrator," the system extends the utility of frozen models to out-of-distribution scenarios. The work highlights that the key to robustness lies not in expanding the skill library, but in learning the operating range of fixed primitives through memory-guided composition and adaptive retry mechanisms.

Limitations: The authors note the system relies on an open feedback loop between the planner and VLA, lacks joint fine-tuning via environmental rewards (suggesting future work with reinforcement learning), and is constrained by the absence of fine-grained image captioning for highly cluttered scenes. They also suggest future integration with automatic skill-discovery systems (like ASPIRE) to expand the primitive vocabulary when necessary.