🤖 AI

Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deployment

The paper presents H $^2$ -EMV, a hierarchical episodic memory framework that enables robots to learn what to forget through user interaction, thereby maintaining high question-answering accuracy while significantly reducing storage and computation costs for scalable, personalized lifelong deployment.

Original authors: Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot butler named Robo. You've had Robo for years, and every single second of its life is recorded: every cup it picked up, every door it opened, and every conversation it had.

If you asked Robo, "Where did I put my keys?" or "Why did the soup burn last Tuesday?", a perfect memory system would have the answer. But here's the problem: Robo's brain is running out of space. If it keeps everything, it will eventually get so slow and cluttered that it can't think at all. It's like trying to find a specific receipt in a shoebox that has been filled with every piece of paper you've ever owned.

This paper introduces a solution called H2-Emv. It teaches robots how to forget on purpose, but in a smart way.

Here is how it works, using some everyday analogies:

1. The "Living Tree" (Online Memory)

Instead of writing a diary at the end of the day (which takes too long), Robo builds a tree as it lives.

The Leaves: Every tiny moment (like "I saw a red cup") is a leaf.
The Branches: As time passes, the robot groups these leaves into branches (e.g., "Making coffee").
The Trunk: The branches merge into the trunk (e.g., "Morning Routine").
The Magic: This happens while the robot is working. It doesn't wait until the end of the day to organize its thoughts. This keeps the tree growing organically.

2. The "Rotting Fruit" (Forgetting)

In nature, fruit rots and falls off a tree. Robo's memory works the same way.

The Timer: Every memory gets a "use-by date." If you haven't asked about a specific event in a while, that memory starts to "rot" (expire).
The Decision: Before the memory falls off the tree, a smart AI (like a librarian) checks it. It asks: "Is this important?"
The Catch: If the robot just forgets everything old, it might forget the keys. If it keeps everything, the tree gets too big.

3. The "Smart Librarian" (Relevance Estimation)

How does the robot know what is important? It uses a Rule Book written in plain English.

The Rules: The robot has rules like: "Always remember where I put valuable items" or "Always remember when I meet a new person."
The Check: When a memory is about to rot, the librarian checks the Rule Book. If the memory fits a rule (e.g., "Oh, this is where the keys were!"), the librarian gives it a fresh coat of paint and extends its life. If it's just "I walked to the kitchen," it gets dropped.

4. The "Feedback Loop" (Learning from You)

This is the most human-like part. The robot doesn't know your rules perfectly at first.

The Mistake: You ask, "Where did I put my keys?" The robot says, "I don't know, I forgot."
The Correction: You say, "Hey! You should always remember where I put my keys!"
The Update: The robot's librarian takes your feedback and rewrites the Rule Book. Next time, the "keys" memory won't rot.
The Result: Over time, the robot learns your specific priorities. It forgets the boring stuff but remembers exactly what you care about.

Why is this a big deal?

The researchers tested this on a real humanoid robot (Armar-7) and in simulations. Here is what they found:

It saves space: The robot's memory size shrank by 45% because it stopped hoarding useless junk.
It's faster: Answering questions became 35% faster because the robot wasn't searching through a mountain of trash.
It gets smarter: In the second round of testing, the robot's accuracy jumped by 70%. It learned from its mistakes and adapted to the user's needs.

The Bottom Line

Think of H2-Emv as teaching a robot to have a human-like memory. Humans don't remember every second of their lives; we remember the important stuff and forget the rest. By teaching robots to do the same—by forgetting the boring details and keeping the relevant ones based on your feedback—we can finally have robots that can work with us for years without crashing their brains or annoying us with irrelevant data.

It's not about remembering everything; it's about remembering what matters.

1. Problem Statement

Robots operating in long-term human-robot collaboration (lifelong deployment) generate a continuous stream of multimodal sensory data. To interact naturally, they must be able to verbalize past experiences (Episodic Memory Verbalization, or EMV) in response to user queries (e.g., "Where did you put my keys?").

However, maintaining a complete, unbounded memory of all experiences presents three critical challenges:

Storage Limits: Continuous data accumulation eventually exceeds hardware capacity.
Compute Latency: Querying an ever-growing memory tree increases the token cost and latency for Large Language Models (LLMs) during question answering.
Relevance Ambiguity: There is no universal definition of what is "relevant" to remember; relevance varies by user, task, and context. Static forgetting mechanisms (e.g., time-based decay) often discard information that users later deem important.

The paper argues that selective forgetting, adapted to user-specific notions of relevance through incremental learning, is essential for scalable, personalized robotic memory.

2. Methodology: H2-Emv Framework

The authors propose H2-Emv, a framework that enables humanoid robots to learn what to remember through interaction. The system is built upon a hierarchical, tree-based Episodic Memory (EM) structure and introduces three core innovations:

A. Online Hierarchical Memory Construction

Unlike previous systems that construct memory trees retrospectively (offline) after a recording is complete, H2-Emv builds the memory online as events occur.

Structure: The memory is a tree with scene graph instances at the lowest level ( $\ell=1$ ), event summaries ( $\ell=2$ ), goal summaries ( $\ell=3$ ), and higher-level abstractions ( $\ell>3$ ).
Time-Aware Incremental Grouping: When a new observation arrives, an LLM performs incremental grouping and summarization. Crucially, the system uses time-based visibility cutoffs to prevent merging temporally distant events (e.g., preventing a robot's "night" state from merging with the next day's activities), ensuring the hierarchy remains semantically meaningful despite gaps in the data stream.

B. Decay-Based Forgetting with Relevance Estimation

To prevent unbounded growth, the system employs a forgetting mechanism where nodes have a default lifetime ( $\tau$ ).

Expiration Check: When a node expires, it is not immediately deleted. Instead, an LLM estimates a relevance factor ( $\alpha$ ).
Relevance Rules: The LLM is conditioned on a set of natural-language relevance rules. If the estimated relevance is high, the node's lifetime is extended ( $\tau_{new} = \tau_{old} + \alpha \cdot \Delta t$ ). If low, the node is replaced by a "forgotten" placeholder (retaining only time range and a summary header) to maintain tree structure without storing full details.
Mechanism: This acts as an offline time-based decay mechanism enhanced by an LLM-derived importance factor.

C. Incremental Learning of Relevance

Since relevance is subjective, the system learns the relevance rules dynamically from user feedback.

Feedback Loop: If a user asks a question and the robot cannot answer (because the detail was forgotten), the user can provide feedback (e.g., "You should always remember where you put the keys").
Rule Update: An LLM processes this feedback to update, add, or remove the natural-language relevance rules.
Propagation: These updated rules immediately influence both the relevance estimation during the forgetting process and the summarization process during online memory construction, allowing the robot to adapt its memory retention strategy to the specific user's priorities.

3. Key Contributions

Online EM Construction: Shifting from retrospective (offline) to incremental (online) tree construction, reducing query-time latency by shifting the computational load to the background.
Learnable Forgetting: Introducing a mechanism where forgetting is not static but driven by LLM-based relevance estimation conditioned on user-defined, learnable rules.
Feedback-Driven Adaptation: Demonstrating that a robot can improve its memory retention strategy over time based on minimal user feedback, effectively personalizing the EM for specific users.
Real-World Deployment: Validating the system on the Armar-7 humanoid robot over a multi-week period, handling 20.5 hours of real-world multimodal data.

4. Experimental Results

The system was evaluated using simulated household tasks (TEACh dataset) and real-world recordings from Armar-7.

Efficiency: H2-Emv reduced memory size by 45% and query-time compute (LLM token cost) by 35% compared to non-forgetting baselines.
Performance Evolution:
- Round 1: Systems with forgetting initially performed worse than non-forgetting systems (as expected, due to information loss).
- Round 2: After receiving feedback on forgotten items, H2-Emv's accuracy increased by 70% in second-round queries. It successfully adapted to user priorities, outperforming non-forgetting baselines in the second round on real-world data.
Ablation Studies:
- Hierarchical Structure: Non-hierarchical (flat) baselines failed to scale, suffering from exploding token costs and poor retrieval accuracy as history length increased.
- Online vs. Offline: Online construction with forgetting performed better than offline construction with forgetting in the long term, as offline trees were severely degraded by initial time-based forgetting before relevance learning could take effect.
- Relevance Learning: Systems without relevance learning stagnated or declined in performance over time, confirming that static rules are insufficient for lifelong deployment.

5. Significance and Future Work

Significance:
This work addresses a fundamental bottleneck in lifelong robotics: the trade-off between memory capacity and retrieval utility. By treating forgetting not as a limitation but as a learnable, adaptive process, H2-Emv enables robots to maintain a manageable memory footprint while improving their ability to answer user questions over time. It bridges the gap between theoretical memory models and practical, long-term deployment on physical hardware.

Future Directions:

Multimodal Feedback: Incorporating visual cues (e.g., user satisfaction gestures) alongside text feedback.
Multi-User Scenarios: Handling conflicting relevance definitions when multiple users interact with the same robot.
Cognitive Inspiration: Integrating cognitive science concepts, such as prioritizing "surprising" or novel events and emotional context, to refine relevance estimation.
Real-Time LLMs: Optimizing the system for smaller, faster foundation models to reduce latency in real-time interactions.

In conclusion, H2-Emv demonstrates that learning to forget is as critical as learning to remember for the successful deployment of autonomous agents in dynamic, long-term human environments.