💬 NLP

Activation-Based Active Learning for In-Context Learning: Challenges and Insights

This paper investigates the potential of using MLP activation-based signals for selecting in-context examples in large language models but concludes that such methods are ineffective due to a lack of correlation with task performance, suggesting that future research should explore techniques like Sparse Autoencoders to address the underlying issue of superposition.

Original authors: Yaseen M. Osman, Geoff V. Merrett, Stuart E. Middleton

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yaseen M. Osman, Geoff V. Merrett, Stuart E. Middleton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but very new student (a Large Language Model) how to solve a specific type of puzzle. Instead of giving them a long textbook to study, you give them a few examples right before the test. This is called In-Context Learning.

The big question researchers have been asking is: Which examples should we pick? If we pick the "best" examples, the student gets a better grade. If we pick bad ones, they might get confused.

The Big Idea: Listening to the Student's Brainwaves

The authors of this paper had a clever hypothesis. They thought, "Maybe we can look at the student's brain activity while they are looking at an example to see if it's a good one."

In computer terms, these "brainwaves" are called activations. When the model processes an example, certain parts of its internal machinery (specifically the MLP layers, which are like the model's memory banks) light up. The researchers wanted to measure how brightly these lights flashed. They thought:

Brighter lights = A more important, high-quality example.
Dimmer lights = A boring, useless example.

They tried to use these "brainwave readings" to automatically select the best examples for the student. They even tried different ways of letting the student look at the examples (like letting them read the whole page at once vs. reading line-by-line) to see if that changed the brainwave patterns.

The Experiment: Testing the Theory

The team ran a massive experiment. They used two popular AI models (Llama and Qwen) and tested them on four different types of tasks:

True/False questions (BoolQ)
Multiple-choice science questions (ARC-Challenge, OpenBookQA)
Math word problems (GSM8K)

For each task, they took 1,000 potential examples. They measured the "brainwaves" (activations) for each one using different math tools (like checking the average brightness, the spread of the light, or the biggest single flash). Then, they actually tested the model with those examples to see how well it performed.

Finally, they compared the brainwave scores against the actual test scores. They were looking for a strong link: Did the examples with the brightest brainwaves actually help the model get better grades?

The Result: A "Negative" Finding

Here is the punchline: No, it didn't work.

The researchers found almost no connection between the brainwave patterns and how well the model did.

The correlation was extremely weak (at best, a score of 0.33 out of 1.0, which is barely a whisper of a relationship).
Sometimes, the examples with the "brightest" brainwaves actually made the model perform worse than if they had picked random examples.
Even when they tried different ways of masking (hiding parts of the text to change how the model reads), the result was the same: the brainwaves didn't predict success.

Why Did It Fail? The "Superposition" Analogy

The authors offer a theory for why this happened, using a concept called Superposition.

Imagine a crowded room where everyone is trying to talk at once. In a normal room, if you want to hear one person, you just listen to their voice. But in this AI model's "room" (its internal dimensions), there are more conversations happening than there are people to listen to.

To fit all these conversations into the limited space, the model mixes them together. A single "light" in the brain isn't just about one specific fact; it's a complex, tangled mix of many different facts and ideas. Because the signals are so jumbled and mixed up, looking at the raw "brightness" of the lights doesn't tell you if a specific example is good or bad. It's like trying to judge the quality of a specific ingredient in a smoothie just by looking at the color of the whole drink; the colors are too blended to tell you what's inside.

The Takeaway

The paper concludes that you cannot use raw brainwave measurements (MLP activations) to pick good examples for AI. It's a dead end for this specific method.

However, the authors suggest a path forward. Since the signals are so mixed up, we might need a special tool to "unmix" them first. They propose using something called Sparse Autoencoders (SAEs), which act like a sophisticated filter to separate the tangled conversations in the room so we can actually hear the individual voices. Until we have that filter, we shouldn't rely on these activation signals to choose our examples.

In short: The idea of using the AI's internal "glow" to pick examples was a great theory, but the experiment showed it doesn't work in practice because the AI's internal signals are too messy to interpret directly.

Technical Summary: Activation-Based Active Learning for In-Context Learning

Problem Statement
In-context learning (ICL) has emerged as a primary method for enhancing Large Language Model (LLM) performance without fine-tuning, relying instead on the selection of high-quality examples within the prompt. While deep active learning has been applied to ICL using techniques like model uncertainty, data diversity, and similarity metrics, there remains a gap in utilizing recent advances in understanding transformer internal activations. Specifically, it is unclear whether the raw activation patterns of the Multi-Layer Perceptron (MLP) layers—which are crucial for knowledge memorization—can serve as a fine-grained signal to optimize the selection of in-context examples. This paper investigates the hypothesis that model activations correlate with example quality and task performance.

Methodology
The authors conducted a comprehensive empirical study using two base models, Llama-3.2-3B and Qwen2.5-3B, across four diverse datasets: BoolQ, ARC-Challenge, OpenBookQA (classification tasks), and GSM8K (generative math reasoning).

The methodology involved the following steps:

Masking Strategies: To capture complex relationships between candidate examples and test samples, the authors experimented with four attention masking configurations:
- Causal Mask: The default setting blocking attention to future tokens.
- No Mask: Removing causal masking to allow bidirectional processing.
- Row Test-Based: Masking only the test query tokens.
- Col Test-Based: Masking only the test key tokens.
Scoring Mechanism: For each candidate example, the model performed a forward pass with 1-shot prompts (example + delimiter + test sample). The authors extracted activation matrices from all $L$ MLP layers.
Metrics: To reduce activation matrices to a single score, the authors utilized:
- The first four statistical moments: mean (absolute), variance, skewness, and kurtosis.
- The "max activation": the largest absolute activation value within the matrix.
- The "per-vector max": the mean of the maximum activations across token vectors.
Evaluation: The authors calculated a shot_score for each example by averaging prompt scores across the test set. They then computed the Spearman correlation coefficient between these scores and the actual task performance (accuracy or exact match) of the selected examples.

Key Results
The study yielded a definitive negative result regarding the initial hypothesis:

Lack of Correlation: There is no reliable correlation between MLP activation-based scores and in-context example performance. The absolute Spearman correlation coefficient was at most 0.33 across all tasks and models tested.
Statistical Significance: Many experiments showed low p-values ( $p < 0.001$ ), indicating a statistically significant lack of correlation.
Performance Misalignment: Selecting examples based on maximum, median, or minimum scores did not yield the best-performing examples. In some cases, "optimal" selections based on these metrics resulted in performance drops below zero-shot baselines.
Robustness: This lack of correlation persisted regardless of the masking strategy (causal, no mask, or test-specific) or the specific metric used (moments or max activations).

Significance and Contributions
The paper's primary contribution is the most comprehensive analysis to date of MLP activation-based deep active learning for ICL, providing a critical "negative result" that guides future research directions.

Empirical Evidence: The authors demonstrate that raw MLP activations, even when viewed through massive activations or statistical moments, do not serve as a proxy for example quality.
Hypothesis on Superposition: The paper hypothesizes that the failure of these metrics is due to superposition, where models represent more features than their dimensionality allows, causing features to be entangled across dimensions. This suggests that linear or simple statistical metrics cannot disentangle these complex, non-linear interactions.
Future Direction: Consequently, the authors argue that practitioners should not use raw MLP activations for sampling in-context examples without further disentangling the MLP. They propose Sparse Autoencoders (SAEs) as a promising future direction to isolate interpretable features before applying active learning strategies.

In conclusion, while the study does not validate the use of activation-based sampling for ICL, it provides essential insights by ruling out a specific class of methods and pointing toward feature disentanglement techniques like SAEs as necessary for future progress in this domain.

The Big Idea: Listening to the Student's Brainwaves

The Experiment: Testing the Theory

The Result: A "Negative" Finding

Why Did It Fail? The "Superposition" Analogy

The Takeaway

More like this