🤖 machine learning

Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

Inspired by human learning, this paper proposes a "Sleep" paradigm for Large Language Models that enables continual learning and self-improvement by consolidating short-term memories into long-term parameters through a two-stage process of "Knowledge Seeding" distillation and unsupervised "Dreaming" via reinforcement learning.

Original authors: Ali Behrouz, Farnoosh Hashemi, Vahab Mirrokni

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Behrouz, Farnoosh Hashemi, Vahab Mirrokni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Goldfish" AI

Imagine you have a brilliant student who can solve any math problem if you give them a cheat sheet right in front of them. This is how current Large Language Models (LLMs) work today. They are amazing at In-Context Learning: if you give them a story or a set of rules in the chat, they can use that information immediately.

But here is the catch: As soon as the chat window closes or the conversation gets too long, the student forgets everything. They have no way to write these new rules into their permanent memory (their "brain"). They suffer from a form of anterograde amnesia—they can remember who they were before they started school, but they can't learn anything new that sticks.

If you try to force them to learn new things by retraining them, they often forget their old skills (a problem called "Catastrophic Forgetting"). If you try to update them too often, it's too expensive and messy.

The Solution: Giving AI a "Sleep Cycle"

The authors propose a new way for AI to learn, inspired by how humans learn. They argue that AI shouldn't just be "awake" (processing data) all the time. Instead, it needs a Sleep phase.

Think of the AI's learning process like a human's day:

Wake Time (Active Learning): The AI is awake, talking to users, and reading new information. It stores this info in a "short-term memory" (like a sticky note on a desk). This memory is fragile and will disappear if the desk is cleared.
Sleep Time (Consolidation): When the AI "goes to sleep," it stops talking to users. Instead, it goes into a deep internal processing mode to move those sticky notes into its long-term brain.

The paper breaks "Sleep" into two distinct stages, just like human sleep has different stages (Deep Sleep and Dreaming).

Stage 1: Deep Sleep (Memory Consolidation)

The Analogy: Moving from a Backpack to a Library
Imagine you are a student who just learned a new fact. Right now, it's in your backpack (your fast, unstable memory). If you keep adding more things to the backpack, it gets messy, and you might lose old items.

During this "Deep Sleep" stage, the AI does two things:

Expands the Library (Parameter Expansion): The AI builds a new, bigger section in its library (adding new parameters/experts). This is like building a new wing on a library so you don't have to throw old books away to make room for new ones.
Knowledge Seeding (Upward Distillation): The AI takes the fragile information from its "backpack" (fast memory) and carefully copies it into the new library wing (slow, stable memory).
- The Twist: Usually, we teach a small student from a big teacher. Here, the AI teaches a bigger version of itself using its own smaller, faster version as the teacher. It's like a junior architect drawing up plans, and then a senior architect (the bigger model) using those plans to build a permanent structure, ensuring the junior's ideas aren't lost.

Why this matters: This prevents the AI from forgetting old skills while learning new ones. It turns temporary "sticky notes" into permanent "library books."

Stage 2: REM Sleep (Dreaming)

The Analogy: The Creative Writer's Block
Once the AI has organized its memories, it enters the "Dreaming" phase. In humans, REM sleep is when we dream and process emotions. For the AI, this is a self-improvement session.

The AI generates its own "dreams"—which are just synthetic examples or practice problems it creates for itself.

The Process: The AI asks itself, "What if I tried to solve this?" It creates a fake scenario, tries to solve it, and then checks if it did a good job.
The Reward: If the AI solves its own dream correctly, it gets a "reward" (a digital high-five). If it fails, it learns from the mistake.
The Goal: This allows the AI to practice and refine its new skills without needing a human teacher to give it homework. It's like a musician practicing scales in their head while sleeping, getting better without anyone watching.

The Results: Does It Work?

The authors tested this "Sleep" paradigm on several tough challenges, and the results were promising:

Learning New Languages: When the AI had to learn to translate two new, rare languages one after another, the "Sleep" model remembered both. The standard model forgot the first one when learning the second.
Long Stories: When reading very long documents (like a whole novel), the "Sleep" model could find specific details buried deep inside, whereas other models got lost.
Math and Reasoning: The model got better at solving complex math problems by using its "Dreaming" phase to practice.
No Catastrophic Forgetting: Most importantly, the model learned new things without losing its old abilities.

Summary

The paper suggests that for AI to truly learn and grow like humans, it needs to stop, rest, and process what it learned.

Wake: Gather information (fragile).
Deep Sleep: Move information to permanent storage and expand the brain (consolidation).
Dream: Practice and refine skills on its own (self-improvement).

By mimicking human sleep, the AI becomes a "continual learner" that can grow smarter over time without forgetting who it used to be.

Technical Summary: Language Models Need Sleep

1. Problem Statement

Current Large Language Models (LLMs) exhibit a critical limitation in their ability to learn continually. While they excel at in-context learning (ICL) and instant prediction, they remain largely static after initial deployment. This creates a "knowledge cutoff" vulnerability where models cannot acquire new facts or skills without expensive re-pretraining or parameter updates that risk Catastrophic Forgetting (CF).

Existing continual learning approaches face a dilemma:

Re-pretraining: Computationally prohibitive for frequent updates.
Fine-tuning/Adaptation: Often leads to CF, where proficiency on original tasks degrades as new ones are learned.
In-Context Learning: Limited by the context window; knowledge is fragile and lost once the session ends.

The paper draws an analogy to anterograde amnesia in humans, where the inability to transfer short-term memories to long-term storage limits knowledge to the immediate present. Current Transformer architectures mirror this: attention mechanisms act as volatile short-term memory (high-frequency updates), while MLP layers act as static long-term memory (zero-frequency updates). The core challenge is enabling LLMs to continuously transfer fragile, short-term in-context knowledge into stable, long-term parametric knowledge without forgetting prior capabilities.

2. Methodology: The "Sleep" Paradigm

Inspired by human neuroplasticity and the distinct roles of sleep stages (NREM for consolidation, REM for dreaming), the authors propose a Sleep Paradigm for continual learning. This paradigm replaces the conventional "train/test" split with two alternating phases: Active (Wake) time and Sleep time.

Phase 1: Memory Consolidation (Offline)

This phase mimics Non-Rapid Eye Movement (NREM) sleep, focusing on stabilizing fragile memories.

Periodic Parameter (De)Activation: The model utilizes a Continuum Memory System (CMS) where different MLP blocks update at different frequencies ( $f_1 \geq \dots \geq f_c$ ). During sleep, the model performs a gradual parameter expansion. New low-rank experts are added to slower (more stable) frequency blocks to receive knowledge from faster (more volatile) blocks.
Knowledge Seeding (Upward Distillation): The core mechanism for transferring knowledge. A "smaller" version of the model (high-frequency, volatile parameters) acts as a teacher, distilling knowledge into a "larger" version (expanded low-frequency parameters).
- Self-Knowledge Seeding (SKS): The model distills its own high-frequency knowledge into its newly expanded low-frequency parameters.
- Generalized Knowledge Distillation (GKD) with Imitation Learning: The objective combines on-policy distillation with Reinforcement Learning (RL).
  - Distillation: The student predicts tokens based on the teacher's logits on self-generated sequences.
  - Imitation Learning: The student is rewarded via RL for mimicking the teacher's sampling distribution. Rewards are based on both semantic similarity (using a frozen reward model) and absolute token-level similarity (Levenshtein distance).
Synaptic Pruning: After consolidation, the low-rank parameters in the high-frequency block (which have transferred their knowledge) are reset, freeing capacity for future learning while preserving the consolidated knowledge in the stable block.

Phase 2: Dreaming (Self-Improvement)

This phase mimics Rapid Eye Movement (REM) sleep, focusing on self-modification and exploring novel connections.

Synthetic Data Generation: The model generates "dreams" (synthetic data) based on its current state and context.
Curriculum Selection: Not all dreams are used. The model assigns an importance score to each dream based on the gradient of the supervised fine-tuning (SFT) objective. It selects the top- $k$ most impactful dreams and adds random samples to maintain diversity.
Iterative Self-Improvement: The model undergoes supervised fine-tuning (using LoRA) on the selected high-quality synthetic data. This process is designed to improve proficiency on recently acquired knowledge while avoiding the catastrophic forgetting often seen in iterative self-improvement loops.

3. Key Contributions

Sleep Paradigm for Continual Learning: A conceptual shift from static train/test phases to a lifecycle of "Active" and "Sleep" states, enabling lifelong adaptation.
Knowledge Seeding (KS): A novel upward distillation process where a smaller, high-frequency model version distills knowledge into a larger, expanded low-frequency version. This addresses the capacity bottleneck in continual learning.
Generalized Distillation with Imitation Learning: A new objective function combining on-policy distillation with RL-based imitation learning to ensure the student not only learns the knowledge but also learns how to use it (sampling behavior).
Periodic Parameter Expansion: A mechanism to gradually unlock and expand parameters (via low-rank experts) during sleep, allowing for plasticity without interference with existing stable knowledge.
Dreaming for Self-Improvement: A robust self-improvement phase that uses gradient-based data selection to generate synthetic curricula, mitigating the risks of iterative self-training.

4. Experimental Results

The authors evaluated the Sleep paradigm on four challenging downstream tasks:

Class-Incremental Learning: On CLINC, Banking, and DBpedia datasets, the "Hope" architecture (augmented with Sleep) outperformed baselines including In-Context Learning (ICL), Elastic Weight Consolidation (EWC), and external learner approaches (InCA). It successfully converted prompt-level adaptation into durable parametric memory.
Long-Context Understanding: On LongHealth, QASPER, and MK-NIAH (Needle-in-a-Haystack), Sleep-based models consistently outperformed ICL and efficient attention baselines (DuoAttention). Increasing the number of consolidation stages improved performance, while increasing the persistence of the most stable memory (lower frequency) was found to be a trade-off between retention and adaptability.
Continual Learning (Language Acquisition): In a task requiring sequential learning of two unseen languages (Manchu and Kalamang), ICL suffered sharp performance drops and catastrophic forgetting. Sleep-based models retained gains across sequential learning, with performance improving monotonically with more consolidation stages.
Knowledge Incorporation & Few-Shot Reasoning:
- Knowledge Incorporation: On SQuAD, Sleep achieved the highest accuracy in integrating new factual information, outperforming SEAL and standard fine-tuning.
- Mathematical Reasoning: Sleep improved reasoning capabilities on AIME and HMMT benchmarks, outperforming base models, SFT, and GRPO.
- Few-Shot Learning (ARC): Sleep achieved an 80% success rate on the ARC benchmark, significantly outperforming ICL, Test-Time Training (TTT), and SEAL.
Scalability: On the BABILong benchmark, the Sleep model maintained stability up to 10 million tokens, whereas other models degraded significantly beyond 128k–256k tokens.

5. Significance and Claims

The paper claims that the Sleep paradigm offers a fundamental solution to the static nature of current LLMs. By decoupling the acquisition of new information (Active/Wake) from the consolidation and self-improvement of that information (Sleep), the framework enables:

Robust Continual Learning: The ability to learn incrementally without catastrophic forgetting.
Efficient Knowledge Transfer: Moving from fragile, context-bound memories to stable, parametric knowledge through "Knowledge Seeding."
Self-Improvement: The capacity to recursively improve capabilities via "Dreaming" without human supervision.

The authors emphasize that their approach differs from recent on-policy self-distillation (OPSD) works by introducing upward distillation (expanding capacity rather than just re-conditioning a fixed model) and a two-phase process (separating consolidation from self-improvement) to prevent the training collapse and capability erosion often associated with iterative self-training. The results suggest that incorporating biological sleep-like mechanisms is crucial for the next generation of adaptive, lifelong learning systems.