🤖 machine learning

Learning, Fast and Slow: Towards LLMs That Adapt Continually

This paper introduces Fast-Slow Training (FST), a framework that combines fixed model parameters ("slow" weights) with optimized context ("fast" weights) to enable LLMs to learn from textual feedback more efficiently, achieve higher performance, and maintain greater plasticity while significantly reducing catastrophic forgetting compared to traditional parameter-updating methods.

Original authors: Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but rigid student (the Large Language Model) how to solve complex puzzles like math problems, coding challenges, or fact-checking riddles.

Traditionally, when we want this student to get better, we force them to memorize the new rules by rewriting their brain (updating the model's internal parameters). This is like taking a sponge, soaking it in new water, and then squeezing it dry. The problem? Once you squeeze it, the old water (their original general knowledge) leaks out. They get really good at the specific puzzle you taught them, but they forget how to be a good general thinker, and they struggle to learn the next puzzle. This is called "catastrophic forgetting."

On the other hand, you could just give the student a cheat sheet (a prompt) for each specific puzzle. This is cheap and fast, but the student still has to rely on their original brainpower. If the puzzle is too hard, the cheat sheet alone isn't enough to get a perfect score.

The "Fast and Slow" Solution
The paper introduces a new training method called Fast-Slow Training (FST). It combines the best of both worlds by treating the student's learning process like a two-track system:

The Slow Track (The Brain): This is the model's permanent weights. It learns slowly, like a long-term memory. It focuses on keeping the student's core reasoning skills and general knowledge intact.
The Fast Track (The Cheat Sheet): This is the "context" or prompt. It learns very quickly, like a temporary sticky note. It absorbs the specific, tricky details of the current task without permanently changing the student's brain.

How It Works (The Analogy)
Imagine you are training a chef (the AI) to cook a new, difficult dish.

Old Way (Slow Learning Only): You force the chef to memorize the recipe by rewriting their entire culinary philosophy. They become great at this one dish, but they forget how to cook anything else, and they can't adapt if you ask them to cook a slightly different version later.
The FST Way: You keep the chef's fundamental skills (Slow Track) exactly as they are. Instead, you give them a dynamic, evolving recipe card (Fast Track).
- Every time the chef tries the dish and makes a mistake, a "coach" (the system) looks at the error and instantly updates the recipe card with a specific tip (e.g., "Don't add salt until step 3").
- The chef uses this updated card to try again.
- Only after the chef has mastered the dish using these cards do we make tiny, careful adjustments to their fundamental cooking style (the Slow Track).

Why This Is Better (The Results)
The paper claims this approach wins in three major ways:

It's Faster and Cheaper: Because the "recipe card" (Fast Track) can absorb new information instantly, the system learns the task much faster. The paper says it needs up to 3 times fewer attempts to reach the same level of performance as the old method.
It Doesn't Forget: Because the chef's fundamental brain (Slow Track) isn't being constantly rewritten, they don't lose their general cooking skills. The paper shows the model stays much closer to its original, smart self, meaning it doesn't "forget" how to be a general reasoner.
It's Ready for the Next Challenge: Since the chef's brain wasn't over-specialized on the first dish, they can immediately start learning a new dish without needing to "unlearn" the old one. The paper tested this by switching tasks mid-training, and the FST models adapted smoothly, while the old models completely stalled.

The Secret Sauce: A Team of Chefs
The paper also mentions a clever trick: instead of using just one recipe card, they maintain a team of different recipe cards (a population of prompts). Some cards are great for math, others for coding. The system mixes them up, allowing the "Slow Track" to see a variety of ways to solve problems, which helps it learn even better without getting confused.

In Summary
This paper argues that we shouldn't force AI models to memorize every new task by rewriting their brains. Instead, we should let them keep their general intelligence (Slow) while giving them a super-fast, adaptable set of instructions (Fast) to handle specific tasks. This makes them smarter, faster to train, and better at learning new things without forgetting the old ones.

Technical Summary: Learning, Fast and Slow: Towards LLMs That Adapt Continually

Problem Statement

Large Language Models (LLMs) are typically adapted to downstream tasks via parameter updates, such as Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). While effective, this approach forces the model to absorb all task-specific information into a single, persistent set of weights. This creates a fundamental bottleneck: updates that improve in-domain performance simultaneously shift the model away from its base behavior, reducing output entropy, degrading out-of-distribution (OOD) generalization, and causing "plasticity loss"—the inability to adapt effectively to future tasks. Conversely, in-context learning (e.g., prompt optimization) allows for rapid, cheap adaptation without modifying weights but generally fails to match the performance ceilings achievable through parameter updates. The paper argues that restricting learning to either in-context or in-weights mechanisms is suboptimal and that current methods fail to leverage the complementary strengths of both.

Methodology: Fast-Slow Training (FST)

The authors introduce Fast-Slow Training (FST), a framework that jointly optimizes two complementary adaptation channels:

Slow Weights ( $\theta$ ): The model parameters, updated infrequently and expensively via Reinforcement Learning with Verifiable Rewards (RLVR). These encode long-lived, persistent behaviors.
Fast Weights ( $\Phi$ ): A population of textual contexts (prompts, instructions), updated frequently and cheaply via Reflective Prompt Evolution (GEPA). These capture transient, task-specific signals.

The Training Loop

FST interleaves two update loops within cycles of length $T$ :

Fast Loop (Context Optimization): Using the current policy $\pi_\theta$ , the system generates rollouts. A frozen "reflection" LLM analyzes these rollouts (including thoughts, tool calls, errors, and feedback) to propose textual mutations. GEPA maintains a Pareto frontier population of $K$ prompts rather than a single best prompt. This population preserves diversity, allowing different prompts to specialize on different subsets of the task distribution.
Slow Loop (Parameter Update): The policy $\theta$ is updated using Group Relative Policy Optimization (GRPO) with a clipped surrogate objective (CISPO). Crucially, during the generation of rollouts for the RL update, the system samples from the entire population of $K$ fast prompts. The advantage calculation normalizes rewards across the entire group, mixing variation induced by both the prompts and the sampling process.

This co-evolution allows the fast weights to absorb task-specific lessons (e.g., specific heuristics or error corrections) immediately, steering the model toward better reasoning. The slow weights then update under this evolving context, preventing the model from over-specializing to a single static prompt or drifting too far from the base policy.

Key Contributions

Framework Integration: The paper instantiates the theoretical concept of fast/slow weights (originally from neuroscience and early neural network literature) for LLM post-training, specifically combining RLVR for weights and GEPA for prompts.
Pareto-Frontier Prompt Population: Unlike prior work that optimizes a single prompt, FST maintains a diverse population of prompts. This allows the RL agent to compare responses across different conditioning behaviors within the same advantage computation, providing richer gradient signals.
Interleaved Optimization: The method avoids the sequential "train then tune" paradigm. Instead, it allows the policy and the prompt population to co-evolve, where the prompt population adapts to the current policy, and the policy adapts to the evolving prompts.

Experimental Results

The authors evaluate FST on three reasoning domains: CodeIO (code output prediction), Polaris (math), and HoVer-hard (multi-hop fact verification), using Qwen3-8B as the base model.

Data Efficiency: FST reaches the running peak performance of standard RL in significantly fewer steps: 3.0× fewer on CodeIO, 1.4× fewer on Math, and 3.0× fewer on HoVer-hard.
Performance Asymptote: FST consistently reaches a higher performance ceiling than RL alone. For example, on HoVer-hard, the fitted asymptote improves from 17.3% (RL) to 25.0% (FST).
Reduced Model Drift: At matched reward levels, FST-trained models exhibit up to 70% lower KL divergence from the base policy compared to RL-only models. This indicates the slow weights remain closer to the base configuration.
Preserved Plasticity: In a two-phase training experiment (training on Task X, then Task Y), FST-initiated models retain the ability to learn Task Y effectively. In contrast, RL-only models often collapse to near-zero performance on the second task, demonstrating severe plasticity loss.
Continual Learning: In a setting where task domains switch every 200 steps, FST adapts quickly to new objectives, whereas RL-only models stall or fail to acquire new tasks after the initial one.
Mechanism Analysis:
- Observation 1: Fast weights acquire task signals much faster than slow weights. In a synthetic star-graph task, FST achieved measurable rewards by step ~50, driven by prompt evolution, while RL-only remained near zero until step ~300.
- Observation 2: The performance ceiling is raised because both channels optimize for reward. Distilling the fast weights into the slow weights (FST-distill) improves performance over prompt-only methods but fails to reach the ceiling of the joint FST approach, confirming that both components are necessary for maximum performance.

Significance and Claims

The paper claims that effective LLM post-training should not be viewed as parameter learning followed by prompt tuning, but rather as optimization over multiple adaptation channels.

Division of Labor: By routing task-specific and rapidly evolving improvements through fast textual weights, the slow parametric weights are freed to consolidate persistent, general reasoning behaviors.
Continual Learning: This division of labor offers a path toward post-training methods that are more data-efficient, less destructive (lower KL drift), and more amenable to continual learning, addressing the "plasticity loss" phenomenon observed in deep continual learning.
Information Efficiency: The framework leverages rich textual feedback (thoughts, errors) in the context update, overcoming the "1-bit-per-episode" information limit often associated with binary RLVR rewards.

The authors conclude that FST represents a promising direction for reasoning models, suggesting that not all task-specific information needs to be distilled into model weights, and that maintaining a separation between declarative (prompt) and procedural (weight) knowledge is crucial for general-purpose reasoners.