🤖 machine learning

MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution

The paper introduces MEMENTO, a memory-guided memetic framework that evolves executable code-as-policies for long-horizon embodied tasks by combining LLM-driven generation with structured feedback from an evolved evaluator, achieving superior performance and sim-to-real transfer compared to existing baselines.

Original authors: Alkis Sygkounas, Victor Aregbede, Amy Loutfi, Andreas Persson

Published 2026-07-28

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Alkis Sygkounas, Victor Aregbede, Amy Loutfi, Andreas Persson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to do a complex chore, like building a tower of blocks or making a sandwich. This isn't just about pressing a single button; it's a long sequence of steps where every move depends on the one before it. If the robot grabs the wrong block or opens the wrong door, the whole plan falls apart. This is the world of "embodied AI," where software meets the messy, physical world. The big challenge here is the "credit assignment problem": when the robot fails at the end of a long task, how do you know which specific step went wrong? Was it the first move, or the tenth?

To solve this, researchers are trying a clever trick called "code-as-policy." Instead of training a robot with vague instructions or trial-and-error rewards, they write the robot's brain as actual computer code. This is like giving the robot a recipe book instead of just a feeling. If the recipe fails, you can read it, see exactly which instruction was confusing, and rewrite it. But writing these recipes by hand is hard, and guessing the right code is even harder. This is where Large Language Models (LLMs)—the same AI brains that write essays and chat with us—come in. They can generate code, but they often need help figuring out why a piece of code failed so they can fix it properly.

This is the story of a new method called MEMENTO, which acts like a super-smart, memory-keeping editor for robot code. The researchers wanted to see if they could evolve better robot recipes by letting an AI write them, test them, and then use a special "memory" to remember what didn't work, so it wouldn't make the same mistakes again. They tested this on two tricky games: one where a robot arm has to solve a Tower of Hanoi puzzle (moving blocks without breaking the rules), and another where a robot has to navigate a virtual house to make a snack.

Here is how MEMENTO works, and what the team discovered.

The Problem with "Guess and Check"

Imagine you are trying to write a program to solve a puzzle. You ask an AI to write the code. It writes something, you run it, and it fails. If you just ask the AI to "try again" with a new, random idea, you might get a slightly different failure, but you aren't really learning. It's like trying to find a hidden key in a giant field by throwing darts blindfolded.

Previous methods tried to fix this by generating many different versions of the code at once and picking the best one. But the researchers found that this approach often missed the mark. It was like having a hundred people suggest different ways to fix a leaky pipe, but none of them actually looked at the specific spot where the water was dripping. They lacked a way to take a single good idea, tweak it carefully, remember what tweaks failed, and then combine the best parts of different ideas.

Enter MEMENTO: The Memory-Keeping Editor

The authors introduced MEMENTO, which stands for Memory-Guided Memetic Code-as-Policy Evolution. The name sounds fancy, but the idea is surprisingly human. Think of MEMENTO as a master editor working with a single "star" draft of a robot's brain.

The process happens in two main acts:

Act 1: Building the Judge
Before the robot even starts learning, MEMENTO creates a special "Judge" (an evaluator). This Judge is a piece of code that watches the robot try the task and gives it a score. But it doesn't just say "Pass" or "Fail." It gives a detailed report card: "You got the score, but you dropped the block here," or "You opened the door too fast." The researchers evolved this Judge first, making sure it was good at spotting exactly what went wrong in the robot's performance.

Act 2: The Three-Branch Search
Once the Judge is ready, the real evolution begins. MEMENTO doesn't just throw darts; it uses three specific strategies to improve the robot's code, all while keeping a "memory" of past failures:

The Hill-Climber (The Careful Tinkerer): This branch takes the current best code and makes tiny, careful changes. If a change makes the robot better, it keeps it. If a change makes it worse, it remembers why that specific change failed. This "memory of rejected ideas" stops the AI from making the same mistake twice. It's like a hiker who tries a path, finds a dead end, marks it on a map, and tries a different path, ensuring they don't walk into the same bush again.
The Macro-Mutator (The Big Idea Generator): Sometimes, tiny tweaks aren't enough. This branch takes the best code and makes big, bold changes—like rewriting a whole paragraph of the recipe. It's the "what if we tried a completely different approach?" moment.
The Crossover (The Mixer): This is the magic ingredient. It takes the best result from the Careful Tinkerer and the best result from the Big Idea Generator and mixes them together. It's like taking the best ingredients from two different recipes to create a new, super-dish.

After trying these three paths, MEMENTO picks the single best result to be the "Elite" for the next round. This cycle repeats, slowly refining the code until the robot can solve the task.

The Results: Does It Actually Work?

The researchers tested MEMENTO in two very different worlds:

Robosuite Tower of Hanoi: A robot arm stacking four blocks.
AI2-THOR: A robot navigating a virtual kitchen to put an apple in a microwave and bread in a fridge.

They compared MEMENTO against two other popular methods (Eureka and REvolve). The results were clear: MEMENTO won.

In the Tower of Hanoi task, MEMENTO achieved a success rate of 0.97 ± 0.06 (meaning it solved the puzzle almost every time), while the other methods struggled, with one only getting 0.53 and the other failing completely (0.00). In the kitchen task, MEMENTO hit a perfect 1.00 ± 0.00 success rate, while the others managed only 0.40 and 0.00.

But the real magic wasn't just in the training room. The researchers tested if the robot could handle new situations it had never seen before.

New Shapes: When they changed the blocks from cubes to weirdly shaped cylinders or asymmetrical objects, MEMENTO still solved the puzzle 0.87 of the time. The other methods crashed to 0.23 or 0.00.
New Rooms: When they moved the kitchen robot to a completely new house layout it had never seen, MEMENTO still succeeded 0.78 of the time. The others dropped to 0.08.

This suggests that MEMENTO didn't just memorize the specific blocks or the specific kitchen; it actually learned the logic of the task.

The "Aha!" Moments (Ablation Studies)

To prove that their specific design was the secret sauce, the researchers tried removing parts of MEMENTO to see what happened.

No Memory: When they removed the "memory" of rejected ideas, the robot got stuck making the same mistakes over and over.
No Big Changes: When they only let the robot make tiny tweaks (no "Macro-Mutation"), it couldn't escape local traps.
No Mixing: When they stopped mixing the best ideas (no "Crossover"), the robot couldn't combine the right strategies.
No Judge Evolution: When they used a simple, untrained Judge instead of the evolved one, the robot couldn't figure out how to win.

Every single time they removed a piece, the performance dropped significantly, proving that the whole system was necessary.

From Simulation to Reality

The most exciting part? They took the best code MEMENTO evolved in the computer simulation and put it on a real physical robot (a Franka robot arm). They didn't retrain it or change the code; they just let it run.

The robot successfully solved the Tower of Hanoi puzzle in the real world 90% of the time (9 out of 10 tries). The one failure wasn't because the code was wrong; it was because the robot's camera misread the position of a block. This proved that the "code-as-policy" approach isn't just a simulation trick—it works in the real, messy physical world.

What This Means

The paper suggests that for robots to learn complex, long tasks, we need more than just random guesses. We need a system that remembers what didn't work, makes both small and big changes, and mixes the best ideas together. MEMENTO shows that by treating robot code like a recipe that can be edited, tested, and refined with a memory of past failures, we can teach robots to solve puzzles that were previously too hard for them.

The authors are careful to note that this was tested on specific tasks and that the method relies on the specific language model they used. They didn't claim this solves every robot problem, but they did show a very promising path forward for teaching robots to think in code.

Technical Summary: MEMENTO for Code-as-Policy Evolution

Problem Statement

Long-horizon embodied control tasks present a significant challenge due to the "long-horizon credit-assignment problem." In these tasks, success is only observable after a sequence of many dependent actions, where early actions determine the state encountered by later ones. Existing approaches, such as demonstration-based learning or reinforcement learning with temporal abstraction, often fail to expose the specific decision logic of a failed policy or identify which parts of that logic require revision.

While representing policies as executable programs ("code-as-policy") allows for direct inspection and revision of decision logic, improving these programs via Large Language Models (LLMs) is difficult. Binary task success is too sparse to guide refinement, and structured feedback is hard to specify manually. Furthermore, existing LLM-driven evolutionary methods primarily select among independently generated variants, lacking a sequential local improvement phase and failing to retain a memory of previously failed revisions.

Methodology: MEMENTO

The authors propose MEMENTO, a memory-guided single-elite memetic framework designed to evolve executable code-as-policy programs. The framework operates in two distinct stages:

1. Evaluator Evolution
Before searching for policies, MEMENTO evolves a specialized evaluator ( $E^\star$ ).

Input: An initial policy ( $p_0$ ) is executed for $N_{roll}$ episodes to create a fixed rollout set ( $T_0$ ).
Process: A pool of candidate evaluators (LLM-generated programs) is iteratively refined. Each candidate maps the rollout set to a scalar fitness score ( $F$ ) and structured feedback metrics ( $M$ ).
Selection: Candidates are scored based on their ability to correlate with task success, and the best evaluator is selected via LLM majority voting.
Output: A fixed, evolved evaluator $E^\star$ that provides both a fitness signal for selection and structured metrics to condition future code revisions.

2. Policy Evolution under Fixed Evaluator
Once $E^\star$ is fixed, the framework searches for the optimal policy program starting from a single elite candidate ( $C_g$ ). Each generation explores three distinct branches, with the best output becoming the elite for the next generation:

Sequential Hill-Climbing (Micro-mutation): A local search branch that iteratively refines the current accepted candidate. Crucially, it maintains a memory ( $H$ ) of rejected proposal edit summaries. This memory conditions the LLM to avoid repeating failed edits, guiding the search away from known local minima.
Macro-Mutation: Independent, large-scale edits applied directly to the generation's elite candidate to introduce significant structural changes.
Crossover: A recombination of the best candidates from the hill-climbing and macro-mutation branches, allowing the combination of partial executable strategies.

The selection mechanism is elitist: the candidate with the highest fitness score ( $F$ ) from the three branches is selected as the next elite ( $C_{g+1}$ ).

Key Contributions

The paper identifies four primary contributions:

Single-Elite Memetic Framework: A search method for executable policy programs that uniquely combines sequential local improvement (with rejected-proposal memory), macro-mutation, and crossover.
Evaluator-Evolution Stage: A pre-search phase that produces a fixed evaluator capable of providing both scalar fitness for selection and structured rollout feedback for conditioning code revisions.
Empirical Evaluation: Comprehensive testing on two long-horizon domains (Robosuite and AI2-THOR), including ablation studies on zero-shot generation, evaluator evolution, and search operators, as well as generalization tests on held-out object geometries and scenes.
Sim-to-Real Transfer: A physical deployment of the evolved Robosuite policy on a Franka robot, demonstrating the feasibility of transferring simulation-evolved code to real-world settings.

Experimental Results

MEMENTO was evaluated on Robosuite Franka Tower-of-Hanoi and AI2-THOR household interaction, comparing it against adapted baselines Eureka and REvolve.

Performance: MEMENTO significantly outperformed baselines. On Robosuite, it achieved 0.97 ± 0.06 success rate compared to 0.53 ± 0.12 for REvolve and 0.00 for Eureka. On AI2-THOR, MEMENTO achieved 1.00 ± 0.00 success, while REvolve achieved 0.40 ± 0.10 and Eureka 0.00.
Generalization: The evolved policies demonstrated strong generalization. In Robosuite, policies transferred from cubic objects to asymmetric and cylindrical objects. In AI2-THOR, policies trained on three kitchens successfully generalized to 26 unseen kitchens.
Ablation Studies:
- Zero-shot generation failed to solve either domain (0% success), proving the necessity of evolutionary search.
- Unevolved evaluators (zero-shot LLM evaluators) failed to guide policies to task completion, highlighting the need for evaluator evolution to align feedback with long-horizon success.
- Operator removal: Removing either the crossover branch or the macro-mutation branch resulted in a failure to solve the tasks, indicating that both local refinement and global variation/recombination are necessary.
Sim-to-Real: The best Robosuite policy was deployed on a physical Franka robot, achieving a 0.90 success rate over 10 trials, with the single failure attributed to a perception error.

Significance and Claims

The paper claims that MEMENTO addresses the limitations of existing code-as-policy evolution by integrating memory-guided local search with evolutionary variation. The authors argue that:

Memory is critical: Retaining a history of rejected edits prevents the LLM from cycling through the same failed modifications, enabling effective hill-climbing in complex code spaces.
Evaluator evolution is essential: A fixed, evolved evaluator provides a richer, more aligned signal than a static or zero-shot evaluator, which is crucial for guiding the search in sparse-reward, long-horizon environments.
Hybrid search is superior: The combination of micro-mutations (local refinement), macro-mutations (global exploration), and crossover (recombination) is necessary to solve tasks that neither pure local search nor pure population-based evolution can solve alone.

The work concludes that memory-guided memetic search is a viable mechanism for improving executable embodied policies using sparse rollout feedback, successfully bridging the gap between simulation and reality.