💬 NLP

An evolutionary perspective on modes of learning in Transformers

This paper draws an analogy between evolutionary biology and Transformer learning to demonstrate that environmental stability favors in-weight learning (IWL) while environmental volatility combined with reliable cues favors in-context learning (ICL), with transitions between these strategies governed by asymptotic optimality and optimization costs.

Original authors: Alexander Y. Ku, Thomas L. Griffiths, Stephanie C. Y. Chan

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Y. Ku, Thomas L. Griffiths, Stephanie C. Y. Chan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to solve problems. This robot has two main ways of learning:

The "Hardwired" Way (In-Weights Learning): It studies a textbook for years, memorizing the answers so deeply that they become part of its brain. Once learned, it never forgets, but it can't change its mind easily.
The "Contextual" Way (In-Context Learning): It doesn't memorize the answers. Instead, it looks at the examples you give it right now (like a cheat sheet) and figures out the pattern on the fly. It's flexible, but it needs you to keep showing it examples.

This paper asks a fascinating question: When does the robot choose to memorize, and when does it choose to look at the cheat sheet?

The authors, using ideas from evolutionary biology, argue that the robot's choice depends entirely on how predictable its world is.

The Biological Analogy: The Water Flea and the Evolutionary Clock

To explain this, the authors use a story about a tiny water creature called a Daphnia (a water flea).

The Plastic Strategy (The Cheat Sheet): If a water flea smells a predator nearby, it instantly grows a protective helmet and a sharp tail spine. This is plasticity. It changes its body based on immediate cues. If the predator leaves, it shrinks the helmet back to save energy.
- Robot equivalent: Using the prompt (the examples you give it) to solve a new problem immediately.
The Evolutionary Strategy (The Hardwired Brain): If the water flea lives in a world where predators are always there, it's too slow to grow a helmet every time. Instead, over thousands of generations, the species evolves to be born with the helmet permanently attached.
- Robot equivalent: Memorizing the solution into its permanent weights so it doesn't need the examples anymore.

The Big Rule:

If the environment is chaotic and changes fast (predators come and go), the robot (or flea) should rely on the Cheat Sheet (ICL). It's too risky to hardwire a solution for a world that changes tomorrow.
If the environment is stable and boring (predators are always there), the robot should Memorize (IWL). Why waste energy looking at a cheat sheet if the answer never changes?

The Experiment: Two Games

The researchers tested this theory with two different games for their AI models:

Game 1: The Wobbly Wave (Sinusoid Regression)

The Task: Predict the shape of a wavy line.
The Twist: Sometimes the wave changes shape every second (chaos). Sometimes it stays the same for hours (stable).
The Result: When the wave was chaotic but the examples given were clear, the robot relied on the Cheat Sheet. But when the wave was stable, the robot quickly stopped looking at the cheat sheet and memorized the wave shape.

Game 2: The Mystery Characters (Omniglot Classification)

The Task: Identify strange alien characters.
The Twist: The robot is given a few examples of what a character means, then asked to guess a new one.
The Result: Here, the robot loved the Cheat Sheet so much that even when the world was stable, it kept using it! Why? Because memorizing 1,600+ alien characters is really hard (expensive), but looking at the cheat sheet is easy.

The "Cost" of Learning

This is the paper's most surprising discovery. It's not just about what the environment needs; it's about what is easiest to learn first.

Imagine you are trying to learn a new language.

Scenario A: You need to memorize 10,000 vocabulary words (Hard). But you can just copy the sentence structure from a book (Easy). You will start by copying (Cheat Sheet) because it's faster, even if you eventually want to memorize the words.
Scenario B: You need to memorize a simple rule like "add an 's' to make it plural" (Easy). But copying the whole sentence is messy and confusing (Hard). You will start by memorizing the rule (Hardwired) because it's the quickest path to a solution.

The researchers found that Transformers act exactly like this. They always grab the "cheaper" learning strategy first, even if the "expensive" one is better in the long run.

In the Character game, copying (Cheat Sheet) was cheap, so they started there.
In the Wave game, memorizing the rule (Hardwired) was cheap, so they started there.

The "Genetic Assimilation" Surprise

The paper also explains a weird phenomenon where a robot starts using a cheat sheet but then stops using it later, even though the cheat sheet still works.

Think of it like a child learning to ride a bike with training wheels.

Phase 1: The child uses training wheels (Cheat Sheet) because they are safe and easy.
Phase 2: As they practice, they get so good at balancing that they don't need the training wheels anymore. They take them off and ride on their own (Memorized).
The Result: The training wheels are gone, but the skill remains.

In AI terms, the robot starts by using the prompt (ICL) to solve the problem. As it trains, it "hardwires" that solution into its brain (IWL). Eventually, it doesn't need the prompt anymore. The prompt becomes redundant, just like the training wheels.

The Takeaway

This paper tells us that AI isn't just a magic black box; it behaves like a living organism trying to survive in its environment.

If the world is unstable: AI stays flexible and uses the context (Cheat Sheet).
If the world is stable: AI hardwires the solution (Memorization).
The "First Step": AI always picks the path of least resistance first. It does the "cheap" thing first, then slowly builds the "expensive" but permanent solution if it needs to.

By understanding these rules, we can design better AI training methods, teaching them to be flexible when they need to be, and solid when they need to be, just like nature has done for millions of years.

1. Problem Statement

Transformers exhibit two distinct learning strategies:

In-Weights Learning (IWL): Permanent refinement of model parameters during training (analogous to genetic evolution).
In-Context Learning (ICL): Ephemeral modulation of inference using information maintained in activations (analogous to phenotypic plasticity).

While the emergence of ICL is well-documented, the learning dynamics governing the transition between these strategies remain poorly understood. Empirical observations show that ICL can be transient (appearing early and disappearing) or delayed (appearing late). Existing rational adaptation frameworks explain which strategy is asymptotically optimal but fail to explain why and when the model transitions between them during training.

The paper posits that Evolutionary Biology offers a superior explanatory lens. Specifically, it hypothesizes that the competition between ICL and IWL is governed by the same principles that dictate the trade-off between plasticity and genetic evolution in biological systems: environmental predictability across different timescales.

2. Methodology

The authors operationalize two key dimensions of environmental predictability and test them in controlled meta-learning benchmarks.

Key Dimensions

Environmental Stability: The invariance of the target task across different prompts during training.
- High Stability: The task remains constant (analogous to a stable environment favoring genetic fixation).
- Low Stability (Volatility): The task changes frequently (analogous to a fluctuating environment favoring plasticity).
Cue Reliability: The sufficiency of information within a single prompt to specify the target task.
- High Reliability: Prompt examples are noise-free and accurate.
- Low Reliability: Prompt examples contain significant noise or are misleading.

Experimental Setup

Two standard meta-learning benchmarks were adapted to manipulate these dimensions:

Parametric Sinusoid Regression:
- Task: Predict $y = A \sin(x + \phi)$ .
- Stability Control: An AR(1) process governs the evolution of parameters $(A, \phi)$ across steps.
- Reliability Control: Gaussian noise added to prompt targets.
Few-Shot Omniglot Classification:
- Task: Binary classification of characters based on a global mapping $M_t$ .
- Stability Control: The persistence probability ( $\gamma$ ) of the global mapping $M_t$ over time.
- Reliability Control: The probability ( $\rho$ ) that prompt labels match the true mapping (flipping labels with probability $1-\rho$ ).

Evaluation Protocol

To distinguish between ICL and IWL, the authors use a conflict-based evaluation:

ICL Target: The label implied by the prompt examples.
IWL Target: The label implied by the current training environment (weights).
Metric: An ICL Preference Score ( $S_{ICL}$ ) is calculated based on the ratio of prediction errors relative to these two targets. $S_{ICL} \approx 1$ implies pure ICL; $S_{ICL} \approx 0$ implies pure IWL.

3. Key Contributions

Evolutionary Framework for Learning Dynamics: The paper establishes a direct correspondence between biological adaptation (plasticity vs. genetic assimilation) and Transformer learning modes (ICL vs. IWL).
Identification of Determinants: It demonstrates that environmental stability and cue reliability systematically dictate the asymptotic preference for ICL or IWL.
Cost-Based Arbitration: It introduces the concept that acquisition cost (inductive bias fit) drives the trajectory of learning. Models initially adopt the "cheaper" strategy (lower optimization cost) before transitioning to the "asymptotically optimal" strategy if the environment demands it.
Explanation of Transience: The framework explains both ICL transience (ICL $\to$ IWL) and IWL transience (IWL $\to$ ICL) as a function of task structure and learning costs.

4. Key Results

A. Asymptotic Strategy Determinants

Stable Environments $\to$ IWL: When the environment is stable (high $\alpha$ ), models converge to IWL, effectively "genetically assimilating" the task into weights. This renders the prompt redundant.
Volatile Environments + Reliable Cues $\to$ ICL: When the environment is volatile but prompts provide reliable cues, models rely heavily on ICL (plasticity) to adapt to each new instance.
The "Cost" Exception: In the Omniglot task with perfectly reliable cues, ICL persists even in stable environments. This suggests that if the cost of learning IWL (memorizing a massive vocabulary) is prohibitively high, the model retains the cheaper ICL strategy despite environmental stability.

B. Task-Dependent Transience Patterns

The direction of strategy transition depends on which strategy is easier to learn initially:

ICL Transience (ICL $\to$ IWL): Observed in Omniglot under high stability.
- Mechanism: ICL (local pattern matching) is a low-cost circuit to discover early. IWL (global memorization of 1623 characters) is high-cost. The model starts with ICL and gradually "assimilates" the task into weights as training progresses.
IWL Transience (IWL $\to$ ICL): Observed in Sinusoid Regression under medium stability.
- Mechanism: IWL (approximating a sine wave) is a simple, low-cost function for neural networks. ICL (implementing a regression algorithm like gradient descent within the forward pass) is a complex, high-cost circuit. The model starts with IWL and only develops ICL capabilities later in training (similar to "grokking").

C. Causal Validation via Learning Cost

Metric: The authors used Prequential Codelength (cumulative negative log-likelihood) to quantify the cost of learning a strategy.
Manipulation: By reducing the Omniglot vocabulary size ( $|C|$ ), the cost of IWL (memorization) was drastically lowered.
Result: In the reduced vocabulary setting, the learning trajectory reversed: the model initially preferred IWL (because it became the cheaper strategy) before switching to ICL. This confirms that acquisition cost, not just environmental optimality, dictates the initial learning phase.

5. Significance and Implications

Ecological View of Training: The paper argues that training environments should be viewed ecologically. The statistical structure of the data (stability and noise) fundamentally shapes the internal learning mechanisms of the model.
Predicting Model Behavior: By understanding the "cost" of a strategy relative to a task's structure, researchers can predict whether a model will exhibit transient ICL or delayed ICL.
Baldwin Effect in AI: The authors suggest a potential "Baldwin Effect" in Transformers, where early plasticity (ICL) might scaffold the acquisition of more robust, weight-based solutions (IWL) for complex tasks, accelerating adaptation on rugged fitness landscapes.
Training Methodology: This framework offers principled guidance for designing training curricula. For instance, to encourage robust IWL, one might need to manipulate environmental stability or reduce the complexity of the IWL solution space to ensure the model converges on the desired strategy.

In summary, the paper bridges evolutionary biology and deep learning, demonstrating that Transformers are not just optimizing for loss but are dynamically arbitrating between learning modes based on environmental predictability and optimization costs.