💬 NLP

Language Models Struggle to Use Representations Learned In-Context

Despite evidence that large language models can encode new semantic representations from in-context information, this study reveals that both open-weights and state-of-the-art closed-source models struggle to effectively utilize these learned representations to adapt their behavior for downstream tasks.

Original authors: Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Smart Student" Who Can't Take Notes

Imagine a brilliant student (the AI) sitting in a classroom. The teacher writes a complex, made-up rule on the board using strange symbols (like "ink" means "city" and "toy" means "air"). The student looks at the board, studies it, and seems to understand the pattern perfectly.

The researchers of this paper wanted to know: If the student understands the pattern while looking at the board, can they actually use that understanding to solve a new problem later, once the board is cleared?

The short answer is: No, not really. Even though the student seems to "get it" in the moment, they struggle to apply that knowledge when asked to do something new with it.

The Experiment: Three Different Tests

The researchers ran three main tests to see how well AI models could learn and use these new, made-up rules.

1. The "Memory Lane" Test (Next-Token Prediction)

The Setup: Imagine a game where you walk through a secret city made of random words. You walk from "Toy" to "Ink" to "City." The AI sees this path.
The Twist:

Scenario A: The AI sees the path written in its own response (like it's writing a diary). It can predict the next word easily.
Scenario B: The teacher (the user) writes the path, and then asks the AI to predict the next word after a pause.
The Result: In Scenario B, the AI gets stuck. It's as if the student studied the map, but when asked to draw the next street on a blank piece of paper, they forgot the map existed. The AI learned the pattern, but it couldn't "pull it out" of its memory to use it.

2. The "Shape-Shifter" Test (Adaptive World Modeling)

The Setup: The AI is shown a random walk through a grid (like a 4x4 checkerboard) made of random words. Then, the teacher gives a few examples of a new rule, like "Move two steps to the right." The AI has to apply this rule to a new word.
The Result: The AI fails miserably.

Why? The researchers checked the AI's "brain" (its internal math) and found that it did actually learn the shape of the grid. It knew the map.
The Problem: Even though the map was stored in its brain, the AI couldn't use that map to solve the puzzle. It's like having a perfect GPS in your pocket but being unable to read the directions when you need to turn. The knowledge was "inert"—it was there, but it was useless for the task at hand.

3. The "Super-Thinker" Test (Reasoning Models)

The Setup: The researchers tried the same tests on the newest, most powerful AI models (the "reasoning models"). These are AIs that are trained to "think out loud" by writing long chains of reasoning before giving an answer.
The Result: These super-models did better, but they still weren't perfect.

They were great at simple, one-dimensional lines (like a straight road).
They completely collapsed when faced with 2D grids (like a city block).
Even when they "thought out loud," they struggled to turn the internal map they had learned into a usable tool. They could describe the map if asked, but they couldn't reliably use it to navigate.

The Core Discovery: "Inert" Knowledge

The main finding of the paper is a concept the authors call "Inert Representations."

Think of it like this:

Encoding: The AI is like a sponge. When you pour water (new information) on it, it soaks it up immediately. It holds the water.
Deploying: But when you squeeze the sponge to get the water out to water a plant (solve a new task), the water doesn't come out.

The paper proves that just because an AI can "encode" (store) a new pattern in its brain during a conversation, it doesn't mean it can "deploy" (use) that pattern to solve a problem later. The knowledge is trapped.

What About the "Thinking" AIs?

The researchers hoped that the newest models, which "think out loud" (write long reasoning chains), might fix this problem. Maybe if they talk through the problem, they can unlock the trapped knowledge?

The paper says: Partially, yes, but not really.
These advanced models are slightly better at using the information, but they still fail significantly when the task gets complex (like moving in two dimensions). They are still struggling to turn their "in-context learning" into a flexible tool.

The Conclusion

The paper concludes that while AI models are getting better at recognizing patterns in the moment, they are not yet "adaptable agents." They cannot reliably learn a new rule from a conversation and then immediately use that rule to solve a different problem.

To build truly adaptable AI, we need to figure out how to make that "squeezing" process work—how to make the knowledge the AI learns in the moment actually useful for the tasks it faces next.

1. Problem Statement

While Large Language Models (LMs) have demonstrated impressive capabilities in next-token prediction and complex reasoning, a critical gap remains in their ability to function as truly adaptable artificial agents. Specifically, the paper investigates whether LMs can flexibly deploy rich, structured representations of data learned entirely in-context (i.e., without parameter updates) to solve novel downstream tasks.

Previous work (Park et al., 2025a) established that LMs can encode novel semantics into their latent hidden states when exposed to a specific context (e.g., learning the topology of a random walk). However, it is unclear if these representations are inert (decodable but unused) or flexibly deployable (causally driving responses to new tasks). The core question is: Can an LM learn a structure in-context and then actively use that structure to solve a different task requiring that same structure?

2. Methodology

The authors designed a series of experiments using synthetic tasks to disentangle encoding (learning the representation) from deployment (using the representation).

A. Core Task: Graph Tracing & In-Context Representation Learning (ICRL)

Setup: Models are presented with a sequence of arbitrary tokens generated by a random walk over a latent state space (topologies include 1D lines and 2D grids of 16 or 25 states).
Metric: The authors use Dirichlet Energy (DE) and Distance Correlation (DC) to measure if the model's hidden layer representations have reorganized to reflect the geometric topology of the state space.
Goal: Confirm that the model successfully learns the topology in its latent space.

B. Experiment 1: Next-Token Prediction with Delay

Objective: Test if the learned topology can be used to predict the next token after a delay.
Conditions:
1. Prefilled: The random walk sequence is provided as part of the model's own generated response (immediate deployment).
2. Instruction: The sequence is provided in the user prompt, requiring the model to generate an [ANSWER] token and then predict the next word after a gap (delayed deployment).
Hypothesis: If representations are flexible, performance should be similar in both conditions. If they are inert, performance should drop significantly in the Instruction condition.

C. Experiment 2: Adaptive World Modeling (AWM)

Objective: A novel task designed to force the model to use the learned topology to infer a transformation rule.
Setup:
1. Phase 1: The model observes a random walk (learning the topology).
2. Phase 2: Few-shot examples define a rule mapping states (e.g., $s_{i,j} \to s_{i+2,j}$ ).
3. Phase 3: The model must apply this rule to a held-out state.
Challenge: Since tokens are arbitrary, the model must rely on the in-context learned topology to solve the mapping.
Variations: Tested with "1-step" (rule observed in walk) and "2-step/3-step" (rule not observed) rules.

D. Experiment 3: Frontier Reasoning Models

Subjects: State-of-the-art "reasoning models" (Gemini-2.5 Pro/Flash, GPT-5) that utilize Chain-of-Thought (CoT) reasoning.
Objective: Determine if externalized reasoning chains can compensate for inert hidden representations.
Analysis: Tested on AWM and a "Structure Description" task (asking the model to verbally describe the learned topology).

3. Key Contributions

Introduction of Adaptive World Modeling (AWM): A new benchmark task specifically designed to probe the flexible deployment of in-context semantics.
Decoupling Encoding from Deployment: The study provides empirical evidence that successful in-context representation learning (encoding) does not guarantee the ability to use those representations for downstream tasks (deployment).
Analysis of Reasoning Models: The first systematic evaluation of whether frontier reasoning models overcome the "inert representation" limitation found in standard open-weights models.

4. Key Results

A. Open-Weights Models (Gemma, OLMo)

Encoding Success: Models successfully learned the topology in their latent representations (DE and DC metrics improved with context length) in both Instruction and Prefilled conditions.
Deployment Failure:
- Next-Token Prediction: Models performed well in the Prefilled condition but struggled significantly in the Instruction condition (delayed deployment), despite having equivalent latent representations.
- AWM Task: Models failed to solve the AWM task when required to infer the topology in-context. Accuracy was near baseline.
- Crucial Finding: When the topology was provided explicitly in the prompt (bypassing the need to learn it in-context), accuracy jumped to >75%. This proves the failure was not in rule induction, but in deploying the in-context learned representation.
- Representation Degradation: Hidden representations of tokens in the few-shot examples (AWM phase) showed lower fidelity to the topology compared to the random walk phase, suggesting the learned structure was not being carried over flexibly.

B. Frontier Reasoning Models (Gemini, GPT-5)

Partial Improvement: Reasoning models outperformed open-weights models, particularly on 1D (line) topologies.
Persistent Limitations: Performance collapsed on 2D (grid) topologies.
Mechanism: When asked to explicitly describe the learned topology, these models achieved "fair but not perfect" accuracy. This suggests that while CoT chains allow for some flexible deployment of semantics, the process is imperfect and does not fully overcome the inertness of the underlying hidden representations.

5. Significance and Implications

The "Inert Representation" Problem: The paper challenges the assumption that in-context learning creates a fully functional "world model." It suggests that LMs often encode context-dependent structures in a way that is inert—they exist in the latent space but cannot be causally accessed to drive behavior in new contexts.
Limitations of Current Architectures: Even the most advanced reasoning models (CoT) only partially mitigate this issue. This implies that current transformer architectures, even with extended reasoning, may lack the mechanisms required for truly adaptable agents that can construct and deploy internal world models on the fly.
Future Directions: The authors suggest that achieving flexible deployment may require:
- Targeted training regimens.
- Mechanistic interventions to make in-context semantics available to earlier model layers.
- Architectural changes that support the dynamic linking of learned representations to downstream reasoning.

In summary, the paper demonstrates that while LMs are excellent at recording new information in their context window, they currently struggle to act on that information when it requires a flexible, causal application of the learned structure to a novel task.