Memory as Plans: World-Action Modeling with Memory-Grounded Planning
The paper introduces MaP-WAM, a Memory-as-Plans framework that addresses the non-Markovian nature of complex robotic tasks by decomposing memory-dependent modeling into memory-grounded planning and plan-conditioned execution, utilizing compact episodic memory representations to achieve state-of-the-art performance on both benchmarks and real-robot tasks while maintaining constant inference latency.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Robots have long been masters of the immediate moment. If you place a cup in front of a robotic arm, it can reach out, grasp it, and lift it with impressive precision. This ability relies on a simple rule: the robot decides its next move based entirely on what it sees right now. However, the real world is rarely so simple. Many tasks require a robot to remember things that are no longer visible. Imagine asking a robot to find a specific button that was covered up minutes ago, or to swap two objects that it saw in different locations earlier in the day. To succeed, the machine must hold a mental picture of the past, not just the present. This is the challenge of memory-dependent manipulation, a hurdle that has kept robots from performing complex, multi-step chores in our homes and workplaces.
For years, researchers tried to solve this by feeding the robot more and more video history. They would give the machine a growing window of every frame it had ever seen, hoping that if it saw enough, it would remember what it needed. But this approach hits a hard wall. As the history gets longer, the computer running the robot slows down, eventually grinding to a halt because it runs out of memory. Other teams tried to summarize the past into a short list of words, but in doing so, they often threw away the fine details—the exact color of an object or its precise position—that are crucial for success. The result was a trade-off: either the robot was fast but forgetful, or it was detailed but too slow to be useful.
A new approach, developed by a team of researchers, changes the strategy entirely. Instead of forcing the robot to constantly re-read its entire history while it moves, they teach it to create a plan first. Think of it as the difference between reading a map while driving versus having a navigator who gives you a single, clear instruction for the next leg of the journey. The researchers call their system MaP-WAM. It works by breaking a long task into smaller segments. Before the robot starts moving, it looks at its long-term memory—sparse, carefully selected snapshots of the past—and writes down a specific plan for the next step. This plan includes both a description of what to do and a visual guide of what the scene should look like as the robot works.
Once this plan is written, the robot executes it without needing to see the entire history again. It only needs to look at the current view and the plan it just made. This keeps the robot's "working memory" small and fast, allowing it to run smoothly even as the task gets longer. To make sure the robot doesn't get lost or drift off course, the system also tracks its progress. It constantly checks its current position against the visual guide in the plan. If the robot sees that it is slightly behind or ahead of where it should be, it adjusts its internal clock to match the plan, correcting any errors before they pile up. This creates a closed loop where the robot plans, acts, checks its progress, and then updates its memory for the next step, all while keeping its computational load steady.
The team tested this method in both computer simulations and on a real robot arm. In the simulations, which involved tasks like swapping blocks or finding hidden buttons, the new system succeeded 83.3% of the time, outperforming previous methods that struggled with memory. On a real robot, it achieved an 88% success rate on a task requiring the robot to find a specific button and a 68% success rate on a task requiring it to press buttons in a specific sequence. Crucially, as the tasks grew longer and the history of past actions increased, the time it took for the robot to decide on its next move stayed roughly the same, hovering around 827 milliseconds. In contrast, older methods that tried to process the full history of frames became so slow and memory-heavy that they crashed after about 1,700 frames.
The researchers found that the key to this success was not just having memory, but how they used it. By converting long, complex histories into compact, memory-grounded plans, they allowed the robot to retain fine-grained visual evidence without the cost of processing it in real-time. They also discovered that explicitly tracking progress was essential; without it, the robot would often confuse visually similar moments, such as pressing a button versus releasing it, leading to repeated mistakes. The system proved that a robot does not need to carry its entire past in its head to act wisely; it only needs to know how to turn that past into a clear, actionable plan for the present. This shift from reactive memory to proactive planning offers a promising path toward robots that can handle the messy, multi-step realities of human life.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.