🤖 machine learning

In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification

This paper reveals that in-context learning models are mechanically constrained to treat demonstrated tokens as an exhaustive vocabulary, causing accuracy to collapse when label slots are filled with homogeneous or even semantically valid tokens, a phenomenon localized to specific neural circuits that override semantic understanding with format-driven token retrieval.

Original authors: Ming Liu

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ming Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly literal-minded robot how to play a guessing game. You show it a few examples (demonstrations) of how the game works, and then you ask it to guess the answer for a new situation. This is called "In-Context Learning."

Usually, we think the robot learns the rules of the game from your examples. But this paper discovered a strange glitch: The robot isn't actually learning the rules; it's just copying the list of answers you gave it.

Here is the breakdown of what the researchers found, using simple analogies:

1. The "List of Answers" Trap

Imagine you are showing the robot how to identify animals.

Normal Scenario: You show it 4 pictures of dogs and 4 pictures of cats. It learns, "Oh, the answer is usually 'dog' or 'cat'."
The Glitch (Homogeneous Labels): You show it 8 pictures of cats, but you tell it the answer is "cat" for all of them. Then you show it a picture of a dog and ask, "What is this?"
- What you expect: The robot sees the dog and says, "That's a dog."
- What happens: The robot looks at your list of 8 "cat" answers, decides that "cat" is the only word allowed in the game, and says "cat" anyway—even though it's wrong.

The researchers found that if you give the robot a list of answers that are all the same, its accuracy crashes to near zero (sometimes less than 12%). It stops thinking about the picture and starts thinking, "I must pick a word from the list you just gave me."

2. The "Nonsense Word" Experiment

To prove the robot wasn't just confused by the word "cat," the researchers tried something weirder. They gave the robot examples where the answers were nonsense words like "foo," "bar," "vex," "nit," and "orb."

The Setup: They showed the robot 8 examples with these nonsense words. Then they showed it a picture of a real dog and asked for the answer.
The Result: Even though "foo" and "bar" make no sense for a dog, the robot refused to say "dog." Instead, it picked one of the nonsense words from your list 42% to 67% of the time.
The Lesson: The robot treats the list of words you gave it as a closed menu. If "dog" isn't on the menu you provided, the robot will starve rather than order off-menu. It doesn't matter if the menu items are nonsense; the robot just grabs one of them.

3. The "Double-Think" Mechanism (How it works inside)

The researchers looked inside the robot's brain (using a technique called "mechanistic interpretability") to see how this happens. They found a two-step process they call "Encode-then-Override."

Think of it like a writer drafting a story:

Step 1 (The Smart Part): The robot's lower brain layers read the picture of the dog and correctly think, "This is a dog." It has the right answer ready.
Step 2 (The Glitch): The robot's upper brain layers look at the list of examples you gave it. They see the list says "cat" (or "foo"). They decide, "No, the rules of this specific conversation say we can only use words from that list."
The Override: The upper layers wipe out the correct answer ("dog") and force the robot to output a word from the list instead.

It's like a student who knows the math problem is 2+2=4, but the teacher says, "In this class, we only use the numbers 5, 6, and 7." The student knows the truth, but the "class rules" (the examples) force them to write "5" instead.

4. It Happens Even with Big Robots

You might think, "Maybe this only happens to small, dumb robots." The researchers tested this on models ranging from very small to quite large (up to 8 billion parameters).

The Finding: The glitch happens to all of them. Even the smartest robots in their test group fell for the trap. In fact, the bigger robots sometimes fell for it harder, crashing from 86% accuracy down to 0% when the examples were misleading.

5. The "Last Word" Rule

The researchers also found a weird quirk about when the robot listens.

If you give it 7 examples of "cat" and 1 example of "dog" at the very end, the robot ignores the first 7 and listens to the last one.
If you put the "dog" example at the beginning, the robot ignores it completely.
Analogy: The robot has a very short attention span. It only cares about the most recent list of answers it saw, and it treats that list as the absolute law for the current question.

Summary

The paper concludes that these AI models don't always "learn" the concept of a task. Instead, they often just retrieve the list of answers you showed them and pick from that list, ignoring the actual meaning of the question. If you give them a list of nonsense, they will answer with nonsense. If you give them a list of all "cats," they will call a dog a "cat."

Why this matters (according to the paper):
If you are using these models for tasks like sorting emails or answering questions based on a database, and the examples you pull up happen to all have the same label (even by accident), the model will stop thinking and just copy that label, leading to massive errors. The fix? Make sure the examples you show the robot are diverse, and maybe put a correct example at the very end of the list to "reset" its brain.

Technical Summary: In-Context Fixation

Problem Statement

While prior research (Min et al., 2022) established that random demonstration labels have minimal impact on In-Context Learning (ICL) performance, this paper identifies a critical failure mode termed In-Context Fixation. The authors demonstrate that when demonstrations share homogeneous labels (even if semantically valid), model accuracy collapses to $\le$ 12% across various architectures and tasks.

Crucially, the paper argues that the trigger is not the correctness of the labels, but the content of the label slot itself. The model appears to treat the tokens occupying the label position in the demonstrations as an exhaustive answer vocabulary. This behavior contradicts latent-concept Bayesian accounts (Xie et al., 2022), which would predict that diverse nonsense labels should result in a diffuse posterior allowing semantic inference to recover. Instead, the model binds its output strictly to the demonstrated token inventory, overriding semantic plausibility.

Methodology

The study employs a multi-faceted approach combining behavioral analysis, causal mechanistic interpretability, and cross-architecture validation:

Experimental Setup:
- Tasks: Four single-token binary classification tasks (Category, Sentiment, Temperature, Size) and 4-way multiclass classification.
- Models: Six base models across three families (Pythia, Llama, Qwen) ranging from 0.8B to 8B parameters.
- Conditions:
  - Garden-Path (GP): All 8 demonstrations share one label opposite the test answer.
  - Varied Nonsense: Demonstrations use diverse nonsense tokens from a set $\{foo, bar, vex, nit, orb\}$ .
  - Multi-token: Using verbalizers like "very positive" to decompose format vs. content fixation.
- Metrics: Accuracy, probability mass on demonstrated sets vs. correct answers, and recovery rates via patching.
Mechanistic Analysis:
- Per-Item Paired Activation Patching: Used on Pythia-1B to swap attention layer activations from control prompts (correct labels) into garden-path prompts to identify causal circuits.
- Logit Lens: Applied to Llama-3.2-1B to trace the evolution of logits through layers, observing the "encode-then-override" trajectory.
- Direct Logit Attribution (DLA): To identify specific attention heads contributing to fixation.

Key Contributions

Task-General Phenomenon: The fixation effect is robust across four tasks, six models, and three model families (0.8B–8B). In all 12 task $\times$ model combinations, GP accuracy falls below 15%, with gaps of 64–100 percentage points compared to balanced controls.
Set-Level Fixation (Constrained Vocabulary Retrieval): The paper demonstrates that fixation operates over the entire demonstrated label set, not just a single token. Even with varied nonsense labels, the model concentrates 42–67% of probability on the demonstrated set while suppressing the semantically correct answer ( $P(dog) < 0.2\%$ ). This proves the model retrieves from the demonstrated inventory rather than inferring from latent concepts.
Generalization to Multiclass and Multi-Token:
- Fixation extends to 4-way classification, reducing accuracy to 0% when labels are homogeneous.
- In multi-token verbalizers, fixation decomposes into format-level (adoption of the "very X" template) and content-level (polarity override) components, which are experimentally dissociable.
First Causal Circuit for an ICL Failure Mode:
- In Pythia-1B, per-item paired patching recovers 98.4% of the accuracy gap.
- The circuit is localized to a layer-7-centered circuit (specifically involving layers 7, 10, and 11).
- A sparse coalition of just 4 heads (across 3 layers) achieves 87.6% recovery, though the circuit is distributed (redundant pathways exist).
Cross-Architecture Replication: Logit lens analysis on Llama-3.2-1B confirms the encode-then-override trajectory: intermediate layers (L9–L11) encode the correct answer, but upper layers (L12–L15) override it under garden-path conditions. Per-item patching on Llama confirms causal sufficiency (89% recovery with top-5 layers).

Results

Dose-Response: Accuracy drops monotonically as the number of misleading demonstrations increases. In Qwen3-8B, a single additional misleading demonstration (moving from $k=7$ to $k=8$ ) causes a near-discontinuous collapse from 42% to 0% accuracy.
Set-Level Boundaries: When a correct label (e.g., "dog") is excluded from the demonstrated set, its probability is suppressed by 1.6–9.4 $\times$ relative to balanced controls, regardless of the semantic validity of the demonstration labels.
Recency Bias: A single corrective demonstration at the final position (position 8) can fully reverse fixation (0% $\to$ 100% accuracy), overriding seven opposing demonstrations.
Calibration Failure: Contextual calibration methods fail to resolve fixation, leaving a 40pp residual gap.
Instruction Tuning: Instruction-tuned models exhibit fixation under raw prompting, though chat templates can reduce the gap in some models (e.g., Llama-3.2-3B-Instruct), suggesting the effect interacts with prompt formatting.

Significance and Claims

The paper claims to reveal that ICL output is constrained vocabulary retrieval rather than semantic inference or latent-concept Bayesian updating. The model binds its output to the demonstrated token inventory regardless of semantic plausibility.

Mechanistic Insight: The "encode-then-override" trajectory provides a mechanistic explanation for ICL failures: the model correctly computes the answer in mid-layers but discards it in upper layers in favor of the demonstrated label set.
Theoretical Implications: This finding challenges the view of ICL as purely implicit Bayesian inference. It suggests a "slot-conditioned set retrieval" mechanism where the label slot defines the support of the output distribution.
Practical Implications: The authors identify this as a concrete reliability failure mode for deployed systems (e.g., RAG pipelines, agent scaffolds). They propose that demonstration diversity is critical; specifically, appending a single diverse, correctly-labeled demonstration at the end of a prompt can mitigate the effect.
Security: The mechanism constitutes a prompt-injection attack surface where an adversary controlling demonstration selection can deterministically override model predictions by ensuring label homogeneity.

The paper concludes that while the effect persists at 8B parameters, the specific circuit architecture varies by model (e.g., Pythia vs. Llama), and the phenomenon warrants further investigation in larger models and reasoning tasks.