🤖 machine learning

Continual Self-Improvement with Lightweight Experiential Latent Memories

This paper proposes an efficient online method that converts transient reasoning traces into compact, modular latent memories via lightweight self-supervised training, enabling large language models to achieve continual self-improvement and superior reasoning performance without catastrophic forgetting.

Original authors: Vaggelis Dorovatas, Nancy Kalaj, Rahaf Aljundi

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vaggelis Dorovatas, Nancy Kalaj, Rahaf Aljundi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart student (a Large Language Model) who is incredibly good at solving math problems. However, this student has a strange quirk: they have no memory of what they just learned.

Every time you give them a new problem, they start from scratch. Even if they spend hours thinking, making mistakes, correcting themselves, and finally finding the right answer, once they hand in the solution, that entire "thinking process" vanishes. They don't get any smarter for the next problem. They are like a genius who forgets their own homework the moment they turn it in.

This paper introduces a system called ELM (Experiential Latent Memory) to fix this. It's a way for the AI to keep a "cheat sheet" of its own thinking so it can get better over time, without needing a teacher to grade its work.

Here is how it works, broken down into simple concepts:

1. The Problem: Why "Reading Notes" Doesn't Work

The researchers first tried a simple approach: In-Context Learning (ICL).

The Analogy: Imagine the student tries to learn by reading a transcript of their own previous thoughts. "Last time I solved a geometry problem, I wrote: 'Step 1: Find the angle. Step 2: Use the formula.'"
The Result: This didn't work well. The paper found that just reading the text of the solution is like reading a recipe without ever cooking the meal. It misses the "flavor" of the actual thinking process—the dead ends, the confidence levels, and the hidden logic that happened inside the model's brain but never made it to the text. It's too shallow to help with new, tricky problems.

2. The Solution: The "Flashcard" System

Instead of writing down the whole story, the researchers created a system where the AI creates tiny, invisible "Flashcards" for every problem it solves.

The Process:
1. The Test: The AI tries to solve a math problem.
2. The Self-Check: Since there's no teacher, the AI generates many different answers to the same problem. It then looks at them and says, "Okay, 7 out of 10 of these say the answer is 42. So, 42 is probably right." This is called Majority Voting. It acts as a self-grading system.
3. The Lesson: The AI takes that specific problem and its self-graded answer and trains a tiny, lightweight "Flashcard" (called a Soft Prompt).
4. The Storage: This Flashcard is stored in a "Memory Pool." It is incredibly small (only 0.001% of the model's size) so it doesn't slow anything down.

3. Using the Flashcards: The "Librarian"

When a new math problem comes in, the system acts like a librarian:

Search: It looks at the new problem and asks, "Do we have a Flashcard in our pool that is similar to this?"
Retrieve: If it finds a match, it pulls out that tiny Flashcard and attaches it to the new problem.
The Double-Check: The AI solves the problem twice: once without the Flashcard (Zero-Shot) and once with the Flashcard.
The Verdict: A "Verifier" (a safety guard) compares the two answers. If the Flashcard version is better, it uses that. If the Flashcard makes things worse (because sometimes the self-grading was wrong), the Verdict picks the original answer.

4. Why This is a Big Deal

No Forgetting: Because the AI doesn't rewrite its entire brain (which would cause it to forget old skills), it just adds these tiny, isolated Flashcards. It's like adding a new chapter to a book without erasing the old ones.
Efficiency: It doesn't need a supercomputer to learn. It learns on the fly, one problem at a time, using very few steps.
Better than Offline Training: Surprisingly, the paper found that learning from one problem at a time using this method often works better than training the whole model on a massive dataset of thousands of problems all at once. It seems the AI learns the "essence" of the reasoning better when it focuses on one specific experience at a time.

Summary

Think of ELM as giving a forgetful genius a personalized, self-updating pocket notebook.

Instead of writing down the whole story of a problem, it writes a tiny, invisible "hint" based on what it learned.
It checks its own work to make sure the hint is good.
When a new problem arrives, it checks its notebook for a similar hint.
If the hint helps, it uses it; if it hurts, it ignores it.

The result is a system that gets smarter every time it solves a problem, turning its own "thinking time" into permanent, reusable knowledge, all without needing a human teacher.

Technical Summary: Continual Self-Improvement with Lightweight Experiential Latent Memories

Problem Statement

Large Language Models (LLMs) currently achieve strong reasoning capabilities through scaling inference-time compute (e.g., Chain-of-Thought, extended thinking). However, these models remain fundamentally stateless after deployment. They discard rich, self-produced reasoning traces generated during inference, failing to accumulate experience or convert computational effort into reusable knowledge for future problems.

Existing approaches to address this face significant limitations:

Training-Free In-Context Learning (ICL): Methods that store raw textual reasoning traces suffer from a "textual bottleneck." They store surface-level outputs rather than the underlying reasoning structure or intermediate states, leading to shallow generalization and failure to transfer effectively across tasks.
Offline Reinforcement Learning (RL): While unsupervised RL methods using internal rewards (e.g., majority voting) show promise, they typically operate in an offline regime, requiring access to the full test dataset upfront. Furthermore, they rely on full-model updates, which risk catastrophic forgetting, optimization instability, and do not scale well as test-time data accumulates due to interfering self-supervisory signals.

The core challenge is to enable truly online, continual self-improvement where an LLM learns from a stream of problems one by one, without external supervision, without prior access to future data, and without forgetting its initial capabilities.

Methodology: Experiential Latent Memory (ELM)

The authors propose ELM, an online method that distills inference-time compute into compact, modular latent memories. The approach consists of three main components:

1. Memory Creation (Per-Sample Learning)

Instead of updating the main model, the system learns a lightweight, instance-specific memory for each encountered problem.

Internal Reward Signal: In the absence of ground-truth labels, the model generates multiple candidate responses ( $N$ ) for a given input. A consensus answer is derived via majority voting, serving as a proxy label.
Training Objective: The model is trained using Group Relative Policy Optimization (GRPO). The advantage is computed by normalizing rewards within the group of sampled outputs relative to the consensus.
Parameterization: The memory is parameterized as a Soft Prompt (a sequence of trainable continuous vectors prepended to the input).
- Efficiency: These soft prompts constitute only $\sim 0.001\%$ of the total model parameters.
- Isolation: The base model remains frozen. Each memory is trained independently for a few gradient steps (e.g., 10 steps) using the internal reward signal. This modular design prevents interference between samples and avoids catastrophic forgetting.

2. Memory Retrieval

When a new test sample arrives, the system retrieves the most relevant memory from the pool.

Key Construction: Keys are derived either from the soft prompt embeddings (Input Retrieval) or by pooling activations from the model's internal Key-Value (KV) projections (KV Retrieval).
Matching: Retrieval is performed by computing cosine similarity between the query vector of the new input and the stored memory keys.

3. Response Routing (Verification)

Since memories are learned via self-supervision, they can be noisy and occasionally degrade performance (turning a correct zero-shot answer into an incorrect one).

Dual Generation: For a given input, the system generates both a zero-shot response and a memory-augmented response.
Verification: If the responses agree, the answer is returned. If they disagree, a Verifier (a Process Reward Model) selects the higher-scoring response. This routing mechanism ensures reliability and mitigates memory-induced regressions.

Key Contributions

Latent Experiential Memories: Introduction of a method to learn modular, instance-level latent representations (soft prompts) directly from test-time compute, capturing abstract reasoning structures rather than surface text.
Online Continual Protocol: A strictly online framework where models improve continuously as experience accumulates, without requiring prior access to the full dataset or external labels.
Efficiency and Modularity: A design that isolates experience in lightweight units ( $\sim 0.001\%$ parameters), enabling stable learning without full-model updates or catastrophic forgetting.
Empirical Validation: Demonstration that learning from individual samples with internal rewards can yield generalization gains that rival or surpass full-dataset offline training.

Experimental Results

The authors evaluated ELM on challenging mathematical reasoning benchmarks (MATH500, AMC23, AIME24) using Llama-3.1-8B-Instruct and Qwen2.5-Math-7B.

Performance vs. Baselines:
- ELM significantly outperforms zero-shot and raw-data ICL baselines.
- On MATH500, ELM (trained entirely online without prior dataset access) outperformed both full-dataset offline GRPO and TTRL baselines.
- On AMC23 and AIME24, ELM substantially improved base performance (e.g., +12.5 points on Qwen for AMC23), often matching or exceeding offline full-dataset training results.
Generalization: Individual latent memories were shown to generalize effectively across the full test set, with soft prompts outperforming LoRA-based parameterizations in generalization capability.
Cross-Dataset Transfer: Memories trained on one dataset (e.g., AIME24) transferred effectively to another (e.g., AMC23), demonstrating that the learned representations capture generalizable reasoning structures.
Comparison to Offline Training: Surprisingly, training on individual samples with internal rewards often yielded better generalization than training the full model on the entire dataset, suggesting that full-model updates may suffer from capacity mismatches or interference in low-data regimes.

Significance and Claims

The paper positions ELM as a foundational step toward models capable of continuous self-improvement. Its significance lies in:

Bridging the Gap: It successfully converts transient, expensive inference-time compute into persistent, reusable knowledge without external supervision.
Scalability: By using modular, lightweight memories, the method scales with accumulated experience without the computational overhead or stability issues associated with full-model fine-tuning.
Overcoming Limitations: It addresses the "textual bottleneck" of ICL by operating in latent space and solves the "offline constraint" of RL by functioning in a true streaming setting.

The authors acknowledge limitations, noting that memory quality varies (some are noisy), and the current reliance on a verifier is a necessary mitigation. They leave future work to explore memory compositionality, eviction strategies, and methods to identify high-quality samples for memory creation. However, they assert that ELM provides a viable, efficient path for LLMs to internalize their reasoning explorations and improve steadily over time.