🤖 machine learning

VLCP: Vision Language Control Policy Closed-Loop Code Replanning for Robot Manipulation

VLCP introduces a training-free robot manipulation policy that keeps a vision-language model frozen and enables closed-loop control by having it dynamically rewrite its own Python control code every $K$ steps based on multi-view observations, achieving a tenfold success rate improvement over open-loop baselines through in-episode failure recovery.

Original authors: Dhia Naouali, Minghan Wu, Claudia Wong, Abhinav Puthran, Omar G. Younis

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dhia Naouali, Minghan Wu, Claudia Wong, Abhinav Puthran, Omar G. Younis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of robotics, there is a persistent gap between what a computer can understand and what it can actually do. For years, scientists have tried to bridge this by teaching robots to speak the language of movement, forcing powerful artificial intelligence models to learn a new, specialized vocabulary of motor commands. This approach requires massive amounts of training data and often strips the robot of its ability to reason, leaving it to memorize specific actions rather than understand the situation. A more recent idea suggests that instead of teaching a robot a new language, we should let it speak the one it already knows best: computer code. By asking an AI to write a short program that tells the robot how to move, researchers can keep the AI's reasoning intact. However, a major flaw in this method has been that the program is written once, before the robot starts moving. If the robot misses a grasp or bumps into something, the program keeps running blindly, carrying that mistake to the end of the task with no way to fix it.

A new study introduces a system called VLCP, which changes the rules by letting the robot rewrite its own instructions while it is still working. The researchers built a framework where a frozen, pre-trained artificial intelligence model acts as a supervisor. Instead of sending a single set of commands at the start, the system pauses every fifty steps to look at the robot's current situation through cameras and sensors. It then asks the AI to write a fresh, short Python function to handle the next fifty steps based on what it just saw. If the robot failed to pick up an object in the previous phase, the AI notices the failure in the images, writes a new piece of code to correct the approach, and tries again, all within the same attempt. This creates a closed loop where the control software itself is edited in real time, rather than just the robot's movements being adjusted.

The researchers tested this system in a simulated environment containing fifty-seven different tasks, ranging from simple object picking to complex sequences in kitchen and living room settings. They compared their method against a version of the same system that wrote the code only once at the beginning and never looked back. The difference was stark. The system that rewrote its code as it went succeeded in thirty-five percent of the tasks, while the one that stuck to its original plan succeeded in only three percent. This tenfold improvement held true across every type of scene they tested. The key to this success was not just that the robot could see its mistakes, but that it could change the very instructions that caused them. In nearly thirty percent of the cases where the robot initially failed to grab an object, the system caught the error, rewrote the approach, and successfully lifted the item later in the same attempt. Without this ability to edit the code mid-task, those failures would have been permanent.

To ensure this process was practical, the team measured how much computing power it required. Because the AI was reusing most of the same background information for every check, the system only needed to process a small amount of new data each time. This allowed the robot to pause, think, and rewrite its instructions roughly ten times during a single task without becoming too slow or expensive to run. The study also showed that this method did not rely on the robot memorizing specific phrases or instructions. When the researchers changed the wording of the task descriptions, the system's performance remained steady, whereas other models that relied on training data struggled significantly when the language changed slightly. This suggests the system is truly understanding the goal rather than just matching patterns.

The researchers were careful to note that their results come from a computer simulation, not a physical robot in a real room. The system currently relies on perfect digital information about where objects are located, which is something real-world cameras cannot always provide with such certainty. Additionally, the time it takes for the AI to generate a new piece of code means this approach is better suited for tasks that do not require split-second reactions. Despite these limitations, the study demonstrates a clear path forward: by keeping the AI in the language of code and allowing it to fix its own mistakes as they happen, robots can become much more capable without needing to be retrained on thousands of human demonstrations. The work suggests that the future of robot control may not lie in teaching machines new ways to move, but in giving them the freedom to edit their own plans as they learn.

Technical Summary: VLCP (Vision Language Control Policy)

Problem Statement
Current approaches to deploying frontier Vision-Language Models (VLMs) in robotics typically rely on Vision-Language-Action (VLA) models. These require fine-tuning pretrained VLMs to emit low-level action tokens (e.g., joint positions or specialized motor tokens) that the model never encountered during pretraining. This adaptation risks discarding the model's inherent reasoning capabilities and necessitates large demonstration datasets. Furthermore, existing "Code-as-Policy" methods generate control code once before execution begins. If a failure occurs during execution (e.g., a missed grasp), these open-loop systems cannot recover because the policy is fixed. Conversely, existing closed-loop methods that do react to observations often operate at the task or language level (re-selecting subtasks or retrying fixed policies) rather than rewriting the underlying control code that caused the failure.

Methodology
VLCP proposes a training-free, closed-loop framework that keeps the VLM frozen and uses it to generate a short-horizon Python control function. The core innovation is closing the loop at the level of the control code itself within a single episode.

Code-as-Policy Representation: Instead of emitting motor tokens, the frozen VLM writes a Python function control(obs, J) that takes multi-view RGB observations, proprioceptive state, and the end-effector Jacobian as inputs. This function outputs a dictionary of joint position targets for the simulator's PD controller.
Closed-Loop Replanning: The episode is divided into blocks of $K$ $K$ steps (default $K=50$ $K = 50$ ). At the start of each block, the VLM is queried to generate a new control function based on the current state.
- Observation: The VLM receives synchronized multi-view RGB images, numeric state (end-effector pose, joint positions, object poses), and an explicit state delta.
- Rewrite: If the previous block failed (e.g., a grasp attempt missed), the VLM observes the failure signature in the new images and rewrites the control function to correct the approach for the next block. This allows for recovery during the episode, not just between trials.
Skill Library: Helper modules (skills) written during any replan are saved to a shared on-disk directory. These are re-injected verbatim into the system prompt of subsequent replans (both within the same episode and across episodes), allowing the policy to accumulate and reuse skills without fine-tuning.
Prompt Engineering: The prompt is structured to maximize token caching. It includes a static prefix (documentation, API, examples), a dynamic skill library, and a "goal pin" (task description) that survives context truncation. Images are only included in the live user turn to prevent context bloat.

Key Contributions

VLCP Framework: A training-free closed-loop manipulation system that re-queries a frozen VLM every $K$ steps to rewrite the control function based on live observations and state deltas.
Cross-Episode Skill Library: A mechanism where helper modules generated during replans are persisted and re-injected into future prompts, enabling skill accumulation without retraining.
Controlled Ablation (The Loop): A rigorous comparison against an open-loop degenerate case ( $K=T$ , where the VLM is queried only once per episode). This isolates the contribution of the within-episode replanning loop, holding the simulator, API, model, and task set constant.
Quantified Recovery: Direct measurement of within-episode recovery, demonstrating that the system can detect and fix failed grasps mid-episode.
Feasibility Analysis: A demonstration that the approach is computationally affordable via prompt caching, requiring only $\sim10$ VLM calls per episode with high cache hit rates.

Results
Evaluated on a 57-task sweep across three scene families (Object Picks, Kitchen Scenes, Living-Room Scenes) in MuJoCo/RoboVerse:

Success Rate: The closed-loop VLCP ( $K=50$ ) achieved a pooled success rate of 35.1%. In contrast, the identical system running in open-loop mode ( $K=T$ ) achieved only 3.5%. This represents a tenfold improvement with non-overlapping 95% confidence intervals across all scene families.
Recovery Mechanism: Analysis of logged traces revealed that 27.3% of failed grasps were successfully recovered by a subsequent replan. The open-loop variant had a 0% recovery rate by construction.
Generalization: Under instruction reframing (changing the phrasing of task instructions), VLCP maintained stable performance (40.0% in both original and modified conditions), whereas a fine-tuned VLA baseline (VLA-0) showed significant performance collapse on specific tasks when instructions were rephrased.
Efficiency: The system achieved a median of 84% input token cache hit rate. An episode required a median of 10 replan calls, with a median latency of 18.8 seconds per replan. All 778 generated control blocks parsed and compiled successfully (100% validity).

Significance and Claims
The paper claims that VLCP demonstrates that closing the loop at the level of the control code is a distinct and effective frontier for robot manipulation.

Preservation of Reasoning: By keeping the VLM frozen and using it to write code (a language it is fluent in) rather than fine-tuning it for actions, the method preserves the model's pretraining reasoning capabilities.
The Locus of Correction: The authors argue that prior methods fail because they retry fixed policies or escalate to planners without fixing the root cause (the control code). VLCP is among the few methods that rewrite the failing artifact mid-episode.
Training-Free Viability: The results suggest that a zero-demonstration, training-free approach can achieve strong absolute performance (35.1%) on complex manipulation tasks, provided the system can self-correct via code rewriting.
Continual Learning Substrate: The cross-episode skill library offers a natural path for continual learning, where competence compounds across episodes without retraining.

The authors acknowledge limitations, noting that the current implementation relies on simulator-provided object poses (bypassing perception noise) and is constrained by VLM inference latency, making it suitable for slower manipulation tasks rather than high-frequency reactive control. They emphasize that the contribution is the loop itself, not a specific generated program.

Technical Summary: VLCP (Vision Language Control Policy)

More like this