🤖 AI

PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft

PEAM is a novel framework for embodied agents in Minecraft that transforms memory from inference-time retrieval into parameter-resident skills by utilizing a dual-system architecture of slow deliberative reasoning and fast reflexive execution, where failures serve as critical training signals for contrastive learning and a self-triggered mechanism autonomously decides when to internalize experiences to enable continual learning without catastrophic forgetting.

Original authors: Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play Minecraft. Currently, most robots learn like a student who never memorizes anything. Every time they face a zombie or need to build a furnace, they have to stop, open a massive notebook of past experiences, search for a similar story, read it, and then try to figure out what to do. This is slow, uses up a lot of "brain space" (computer memory), and if the notebook gets too big, they get confused.

PEAM (Parametric Embodied Agent Memory) is a new way for robots to learn that changes the game. Instead of just keeping a notebook, PEAM helps the robot internalize its experiences. It turns "I remember how I did this" into "I know how to do this."

Here is how it works, broken down into simple concepts:

1. The Two-Brain System: The Thinker and the Doer

PEAM uses a "slow" and a "fast" brain working together:

The Slow Thinker (Deliberative LLM): This is the expert planner. It thinks deeply, writes code, plans complex steps, and tries to solve hard problems. If it fails, it figures out why and tries again.
The Fast Doer (Parametric Skills): This is the reflexive muscle memory. Once the Slow Thinker solves a problem (like crafting a sword), it doesn't just save the story. It teaches the Fast Doer the skill directly. Next time, the Fast Doer can do it instantly without asking the Slow Thinker.

2. The "Internalization" Process: From Notebook to Muscle Memory

Think of learning to ride a bike. At first, you have to think about balancing, pedaling, and steering (Slow Thinker). Eventually, you just do it without thinking (Fast Doer). PEAM automates this for a robot.

Learning from Failure: Most robots ignore mistakes or just write them down as text notes. PEAM treats failure as a training signal. It looks at a "Failed Attempt" and the "Correction" that fixed it. It teaches the robot: "Don't do X (the failure); do Y (the correction)." This is like a coach saying, "You fell because you leaned too far left; next time, lean right."
The "Worthiness" Score: The robot doesn't learn everything. It has a filter (called Parameterization-Worthiness) that asks: "Is this skill useful enough to memorize? Is it stable? Is it something we do often?" If the answer is yes, it gets internalized. If it's a one-time fluke, it stays in the notebook.

3. The "Specialized Gym" (Isolated Adapters)

This is the paper's biggest innovation to prevent "forgetting."
Imagine a gym with different rooms: a Crafting Room, a Combat Room, and a Farming Room.

In old systems, learning to fight might accidentally overwrite how the robot knows how to craft (like mixing up your gym clothes).
In PEAM, the robot has physically isolated rooms (called adapters). When it learns to fight, it only updates the "Combat Room." The "Crafting Room" remains untouched. This means the robot can learn new skills forever without forgetting the old ones.

4. The Self-Triggered Alarm Clock

How does the robot know when to stop practicing and start memorizing?

Old way: "Let's practice for 100 tries, then memorize." (This is rigid and might memorize too early or too late).
PEAM way: The robot has a self-triggered alarm. It watches its own failure rate. If it starts failing a specific task more often than usual, the alarm goes off: "Hey, we are struggling with this. Let's stop and internalize the solution immediately." This works no matter what the task is, without needing a human to set a specific number.

Why is this better?

The paper tested this in Minecraft and found three main benefits:

Speed: Because the robot doesn't have to search a notebook every time, it acts much faster (about 40% faster in the tests).
Efficiency: It uses way less computer power (about 85% less "text" to process) because it doesn't need to re-read old stories.
No Forgetting: Because of the "specialized rooms," learning a new combat move didn't make the robot forget how to build a house.

Summary

PEAM is like taking a student who relies on a textbook for every single question and turning them into an expert who has actually memorized the skills. It learns by doing, learns from mistakes, keeps different skills separate so they don't get mixed up, and knows exactly when to stop practicing and start remembering. The result is a robot that is faster, smarter, and doesn't forget what it has learned.

Technical Summary: PEAM - Parametric Embodied Agent Memory

1. Problem Statement

Current Large Language Model (LLM) based embodied agents, particularly in complex environments like Minecraft, rely heavily on non-parametric memory. In these systems, past trajectories, reflections, and skills are stored externally (e.g., in vector databases or skill libraries) and re-injected into the prompt at inference time. While this allows for recall, it presents significant limitations:

Latency and Cost: Every decision cycle requires retrieving and serializing past experiences, increasing context window usage and inference latency.
Static Policy: The agent's underlying parametric policy (the backbone model or trainable module) remains unchanged across tasks. Even if an external library grows, the agent does not "learn" to execute skills more efficiently; it merely retrieves them.
Missing Consolidation: Existing systems handle the acquisition of experience (via skill libraries) but lack a mechanism for consolidation—the process of transforming selected episodic traces into durable, internalized parametric competence.

The paper posits that durable memory in cognitive systems arises from two complementary systems: a fast episodic store and a slow, distributed parametric store, coupled by offline consolidation. PEAM aims to operationalize this for embodied agents by creating a pathway for experience to move from external recall into the agent's trainable parameters.

2. Methodology

PEAM (Parametric Embodied Agent Memory) is a two-tier framework designed to internalize experience into parametric skills.

Architecture

Slow Tier (Deliberative): A large LLM (e.g., GPT-4o) handles open-ended reasoning, code synthesis, and outcome verification. It explores the environment, detects failures, and generates corrected trajectories.
Fast Tier (Parametric): A multimodal Mixture-of-Experts (MoE) LoRA architecture built on a Qwen3-VL-8B-Instruct backbone. It executes consolidated skills reflexively with low latency.
Episodic Store: An external memory that stages successful trajectories and failure-correction pairs generated by the slow tier.

Core Mechanisms

PEAM addresses three critical questions: What to internalize, When to internalize, and How to internalize.

A. How: Success and Failure-Correction Consolidation

PEAM treats failure as a first-class training signal. The episodic store contains:

Success Trajectories ( $D_{succ}$ ): Verified successful executions.
Failure-Correction Pairs ( $D_{cpair}$ ): Pairs of a failed trajectory ( $\tau_f$ ) and a subsequent corrected success ( $\tau_c$ ) under the same context.

The fast module updates specific per-category isolated adapters ( $\theta_c$ ) using a joint objective:
$\mathcal{L}_{PEAM}(\theta_c) = \mathcal{L}_{BC}(\theta_c; D^{(c)}_{succ}) + \lambda \mathcal{L}_{PEAM-DPO}(\theta_c; D^{(c)}_{cpair})$

Behavioral Cloning (BC): Ensures the adapter learns the absolute syntactic scaffolding (e.g., code wrappers) required for the action parser to function.
PEAM-DPO (Direct Preference Optimization): A contrastive loss where the corrected trajectory is preferred over the failed one. This teaches the agent how to correct actions, not just what succeeds.
Isolation: Updates are restricted to the specific category adapter (e.g., Crafting, Combat), preventing interference with other skills.

B. What: Parameterization-Worthiness (PV) Score

Not all experiences should be internalized. PEAM uses a scoring function to rank candidates based on:

Cost Savings ( $U_{cost}$ ): Estimated retrieval cost saved (frequency $\times$ code length).
Stability ( $U_{stab}$ ): Consistency of success across contexts.
Redundancy ( $P_{redun}$ ): Penalizes similarity to already internalized skills.
Forgetting Risk ( $R_{forget}$ ): A structural proxy; if a skill shares a category with existing parameters, it risks interference (though isolation mitigates this, the score accounts for category identity).

Only candidates in the top- $q$ quantile of PV scores are considered for consolidation.

C. When: Self-Triggered Consolidation (STC)

Instead of a fixed schedule, PEAM uses a scale-free, self-triggered mechanism. Consolidation is triggered when a skill's recent failure rate significantly exceeds its historical baseline, determined by a statistical test (Z-score) rather than a task-specific absolute threshold. This allows the trigger to transfer across different task distributions without re-tuning.

3. Key Contributions

Parametric Memory Framework: PEAM shifts embodied agent memory from inference-time retrieval to parameter-resident skills, reducing latency and token consumption while improving long-horizon performance.
Failure as a Training Signal: Unlike systems that convert failures into textual reflections for prompts, PEAM directly trains on failure-correction trajectory pairs using a joint BC and DPO objective. This enables the agent to learn the specific behavioral differences between failure and success.
Architectural Forgetting Resistance: By using physically isolated per-category LoRA adapters, PEAM achieves parameter-level continual learning. Internalizing a combat skill does not overwrite the parameters used for crafting, eliminating catastrophic forgetting by design rather than regularization.
Scale-Free Consolidation Triggers: The introduction of a self-triggered consolidation mechanism that relies on relative failure statistics allows the agent to self-evolve without manual, task-specific threshold tuning.

4. Experimental Results

Experiments were conducted in Minecraft (1.19) using the VOYAGER framework, comparing PEAM against retrieval-based agents (VOYAGER, Optimus-1), non-parametric memory variants (Reflexion), and parametric baselines (Full Fine-Tuning, Shared LoRA, EWC).

Task Success: PEAM achieved 69.7% success on long-horizon tasks, outperforming the strongest retrieval baseline (VOYAGER at 54.5%) by +15.2 percentage points (McNemar $p=0.018$ ).
Efficiency: PEAM reduced median per-call latency by 42% (5.5s $\to$ 3.2s) and token consumption per task by 85% (31.2K $\to$ 4.6K) by eliminating the need to re-inject skill libraries at inference time.
Forgetting Resistance: In sequential consolidation experiments (Craft $\to$ Gather $\to$ Combat), PEAM showed 0% forgetting on previously learned skills. In contrast, shared LoRA baselines lost 32.4% performance, and full fine-tuning lost 78.5%.
Ablation Insights:
- BC + DPO: A pure DPO adapter failed to generate parser-compatible code (0/12 cases), whereas the joint objective succeeded (12/12). The BC term is essential for maintaining deployable syntax.
- Isolation: Removing per-category isolation caused significant forgetting, confirming that structural isolation is the primary driver of retention.
- Trigger Robustness: The self-triggered mechanism maintained consistent performance across different task distributions (craft-heavy vs. combat-heavy) without re-tuning, unlike fixed-threshold baselines.

5. Significance and Claims

The paper claims that PEAM demonstrates a viable pathway for embodied agents to transition from accumulating external histories to internalizing selected experiences as part of their core policy.

Efficiency vs. Retrieval: The results suggest that while richer retrieval (e.g., multimodal context) helps, the operational benefits of internalizing skills into parameters (latency, token reduction, and direct execution) provide gains that retrieval alone cannot capture.
Structural Continual Learning: The work argues that forgetting resistance in embodied agents should be supported by architecture (parameter isolation) rather than solely by regularization techniques.
Methodological Findings: The paper highlights that forward-pass preference margins (DPO scores) do not always predict the deployability of generated code, emphasizing the need for joint objectives that preserve syntactic constraints. It also notes that trajectory re-slicing can serve as a tractable substitute for cross-distribution evaluation when real-world paired data is scarce.

The authors conclude that embodied agents should not merely act as fixed policies with external memory; they require a consolidation pathway where selected experience becomes part of the policy itself, enabling self-evolution and efficient execution.