Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation

This paper introduces Act-Observe-Rewrite (AOR), a framework enabling multimodal language models to iteratively improve robot manipulation policies by synthesizing and rewriting executable Python controller code based on visual feedback and failure analysis, achieving high success rates across tasks without demonstrations, reward engineering, or gradient updates.

The Gist

Imagine you are teaching a robot to build a tower of blocks. In the old days, you had two main options:

1. The "Drill Sergeant" Method: You show the robot a video of a human doing it perfectly 10,000 times. The robot memorizes the movements by rote. If you change the lighting or the table, the robot gets confused and fails.
2. The "Trial and Error" Method: You let the robot try, fail, and try again, but you have to manually tweak its "brain" (its neural network) after every mistake. This takes a long time and requires a supercomputer.

This paper introduces a third way: The "Self-Taught Programmer."

The authors call this Act–Observe–Rewrite (AOR). Here is how it works, explained with a simple story and some analogies.

The Story: The Robot and the "Ghost Writer"

Imagine a robot arm (the Actor) trying to pick up a red cube and put it on a table.

1. Act: The robot tries to do the task. It moves its arm, grabs the cube, and places it.
2. Observe: The robot fails. Maybe it missed the cube, or it dropped it. But instead of just saying "I failed," a special AI assistant (a Multimodal LLM) watches the video of the failure. It looks at the robot's code (the instructions telling the robot how to move) and the video of the mistake side-by-side.
3. Rewrite: The AI assistant doesn't just say, "Try harder." It acts like a Ghost Writer for the robot's brain. It opens the robot's source code, finds the exact line causing the error, and rewrites it.

Then, the robot compiles this new code and tries again. No human touched the code. No massive training data was used. The robot literally wrote a better version of itself after every failure.

The Magic Analogy: The "Code vs. The Dial"

To understand why this is special, imagine a radio.

• Old Methods (Parameter Tuning): Imagine the robot is a radio with a dial. If the signal is bad, you just turn the dial slightly left or right. You are guessing. If the radio is broken because the antenna is in the wrong place, turning the dial won't help.
• This Paper's Method (Code Rewriting): Imagine the robot is a radio, but the AI assistant is an engineer. If the signal is bad, the engineer doesn't just turn a dial. They open the radio, look at the circuit board, and realize, "Ah, the antenna is wired backward!" They solder a new wire (rewrite the code) to fix the root cause.

Why is this a big deal?
Most robot learning methods only "turn the dial." They tweak numbers. But this paper shows that if you let the AI rewrite the actual instructions, it can fix deep, structural problems that numbers can't solve.

The Three Challenges They Solved

The researchers tested this on three tasks, and the "Ghost Writer" solved them in very human-like ways:

1. The "Wrong Map" Problem (Lift Task):

   • The Mistake: The robot kept hovering 8 inches above the cube. It thought the cube was floating in the air.
   • The Old Way: You'd have to guess, "Maybe the camera is too sensitive?"
   • The AOR Way: The AI looked at the code and the video. It said, "The camera uses a different coordinate system (like a map where North is Down). The code is reading the map upside down." It rewrote the math to flip the image. Success.

2. The "Colorblind" Problem (Pick & Place Can):

   • The Mistake: The robot was looking for a "silver" soda can, but in the camera's view, the can looked red. The robot couldn't find it.
   • The AOR Way: The AI saw the robot staring at empty space. It looked at the code: search for silver. It realized, "Wait, the lighting makes it look red!" It changed the code to search for red. Success.

3. The "Clumsy Hand" Problem (Stack Task):

   • The Mistake: The robot could pick up the first cube, but when it tried to stack a second one, its fingers kept bumping the bottom cube and knocking it over.
   • The AOR Way: The AI saw the bump in the video. It rewrote the approach path to be more careful. It got the robot to 91% success.
   • The Limit: The robot eventually got stuck. It knew why it was failing (it was bumping the block), but it couldn't figure out how to move its fingers differently to avoid it. It hit a wall. This shows the system isn't perfect yet, but it's honest about its limits.

The "No-Go" Zone (What Makes This Unique)

Usually, to get a robot to learn, you need:

• Thousands of videos of humans doing the task.
• Complex math to reward the robot for good moves.
• Supercomputers to train the brain.

AOR needs none of that.

• No Videos: It learns from its own mistakes.
• No Rewards: It doesn't need a score; it just needs to see the code didn't work.
• No Training: It doesn't "learn" in the background; it literally rewrites its own software between attempts.

The Bottom Line

This paper proposes a new way to build robots: Don't train the robot to be smart; give it a smart editor that rewrites its own manual.

It's like giving a robot a "Ctrl+Z" (Undo) button, but instead of just undoing the move, it rewrites the instructions so the mistake never happens again. It turns robot learning from a "black box" (where we don't know why it works) into a transparent process where we can read the code and say, "Ah, that's why it failed, and here is the fix."

While it's not perfect yet (it sometimes gets stuck on very tricky physical problems), it proves that robots can learn to fix their own code just by watching themselves fail and thinking about it.

Technical Summary

Here is a detailed technical summary of the paper "Act–Observe–Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation" by Vaishak Kumar.

1. Problem Statement

The paper addresses a critical gap in robot manipulation learning: how to adapt a robot's control policy to a specific deployment configuration without gradient updates, pre-collected demonstrations, or manual reward engineering.

While Vision-Language-Action (VLA) models generalize well across tasks, they often fail on specific hardware setups due to subtle issues like sensor biases, coordinate system mismatches, or kinematic constraints. Current solutions typically require expensive retraining or rely on fixed skill libraries that cannot be modified when failures occur. The authors ask: Can a multimodal Large Language Model (LLM) learn to manipulate physical objects by reasoning about its own failures and rewriting the control code itself?

2. Methodology: The Act–Observe–Rewrite (AOR) Framework

AOR is a two-timescale framework where an LLM agent iteratively synthesizes and improves a robot's executable Python controller.

Core Architecture

The framework operates in a loop consisting of a Fast Loop (robot execution) and a Slow Loop (LLM reasoning):

1. Act (Execution): The robot executes an episode using a current Python controller class. The controller handles low-level motor control, vision processing, and state management.
2. Observe (Data Collection): Upon episode termination, the system collects:
   • Structured Outcomes: Reward, duration, phase logs, and diagnostic signals (e.g., minimum distance to target, oscillation flags).
   • Visual Evidence: Key-frame images captured at phase transitions and regular intervals.
   • Memory: These data points are stored in an episodic memory buffer.
3. Rewrite (Policy Synthesis): A multimodal LLM (specifically Claude Code in the experiments) analyzes the memory and the current controller source code.
   • Diagnosis: The LLM answers three structured questions: What was the dominant failure mode? Was the root cause in vision, logic, or parameters? What is the single most impactful change?
   • Synthesis: The LLM generates an entirely new Python controller class. It does not just tune parameters; it can rewrite the logic, fix geometric formulas, or alter the state machine structure.
   • Safety: The new code is compiled in a sandbox. If it fails to compile or lacks the required interface, the system falls back to the previous working controller. Actions are also clamped to safety limits.

Key Distinctions from Prior Work

• vs. Reflexion/ReAct: Unlike verbal self-reflection which modifies high-level strategies or text descriptions, AOR modifies the executable motor-control code.
• vs. Code as Policies: Unlike one-shot code generation, AOR includes a closed feedback loop where the LLM observes the physical outcome of its code and iteratively refines it.
• vs. VLA Models: AOR does not update neural weights. It treats the policy as a human-readable, auditable program that can be debugged via code logic rather than opaque weight adjustments.

3. Key Contributions

1. Architectural Shift: The paper proposes that making the full controller implementation the unit of reasoning enables a qualitatively different form of in-context learning. The agent can diagnose root causes (e.g., "the y-coordinate is negated in the back-projection") rather than just observing that a failure occurred.
2. Autonomous Debugging: The framework demonstrated the ability to autonomously discover and fix subtle, undocumented bugs in the simulation environment (e.g., mismatches between OpenGL and OpenCV camera conventions) without human intervention.
3. Zero-Shot Adaptation: The system achieves high success rates on manipulation tasks without any demonstrations, reward function engineering, or gradient-based training.
4. Interpretability: Because the policy is code, the reasoning process is transparent. Humans can read the LLM's diagnosis and the resulting code changes to understand exactly why the robot failed and how it was fixed.

4. Experimental Results

The framework was evaluated on three Robosuite tasks using a UR5e arm and a multimodal LLM agent.

Task	Description	Success Rate	Iterations (LLM Calls)	Key Findings
Lift	Pick up a red cube.	100%	3	Rapidly converged by fixing a 2.5cm depth vision bias and switching to a "stationary grasp" (holding position while closing gripper).
PickPlaceCan	Pick a cola can and place in a bin.	100%	2	Diagnosed that the can rendered as red (not silver) and that a bin marker was contaminating the centroid calculation. Fixed by changing color segmentation and filtering logic.
Stack	Stack a red cube on a green cube.	91%	20	Fixed systematic vision pipeline bugs (back-projection sign errors, extrinsic matrix mismatches) reducing error from 5-8cm to ~1.5cm. Residual 9% failure was due to gripper contact with the target cube, which the agent diagnosed but could not resolve within the iteration budget.

Comparison: When tested with a different coding agent (OpenAI Codex/GPT-5), the system failed to solve even the simplest task, highlighting that the framework's success is dependent on the reasoning capabilities of the underlying LLM.

5. Significance and Limitations

Significance:

• New Paradigm: AOR establishes a viable path for "write, test, rewrite" robot learning, offering a low-cost alternative to data-driven training for specific deployment scenarios.
• Debugging Tool: The ability to autonomously identify code-level bugs (like coordinate system mismatches) suggests AOR could serve as a powerful debugging tool for any vision-based robot system.
• Complementarity: It complements VLA models; while VLAs provide broad generalization, AOR provides targeted, interpretable debugging for specific failures.

Limitations:

• Simulation Only: Experiments were conducted in Robosuite. Real-world deployment would require handling higher noise and uncertainty.
• Search Efficiency: The agent tests hypotheses sequentially (one trial at a time), which is slow. Parallel hypothesis testing is a potential future improvement.
• Local Optima: In the "Stack" task, the agent correctly diagnosed the residual failure (gripper contact) but failed to generate a novel strategy to avoid it, getting stuck in a local optimum of the search space.
• Agent Dependence: Performance is tightly coupled with the LLM's ability to reason about geometry and code; weaker models may fail to diagnose complex physical failures.

Conclusion

The paper demonstrates that multimodal coding agents can effectively learn robot manipulation policies through iterative self-correction. By treating the controller as code, the system bridges the gap between high-level reasoning and low-level physical execution, enabling robots to adapt to new environments and fix their own bugs without retraining.