🤖 machine learning

When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning

This paper challenges the assumption that correct demonstrations always aid In-Context Learning by revealing that task-preserving perturbations can degrade performance through "contextual evidence shift," demonstrating that exemplar utility depends on how examples influence contextual inference rather than mere correctness.

Original authors: Chenghao Qiu, Chunli Peng, Yufeng Yang, Kuan-Hao Huang, Yi Zhou

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenghao Qiu, Chunli Peng, Yufeng Yang, Kuan-Hao Huang, Yi Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Being "Right" Isn't Always "Helpful"

Imagine you are trying to teach a robot how to sort laundry. You show it a few examples:

Example 1: A red sock goes in the "Red" pile.
Example 2: A blue shirt goes in the "Blue" pile.

The robot learns quickly. This is called In-Context Learning (ICL). The robot looks at the examples you give it (the "demonstrations") and guesses how to handle the new item you just handed it.

The common belief has been: "As long as the examples I show the robot are factually correct, the robot will learn better."

This paper proves that belief is wrong.

The authors discovered a strange phenomenon: You can give the robot examples that are 100% factually correct, but if you change how those examples look or sound, the robot might get confused and perform worse than if you gave it no examples at all.

The Experiment: The "Task-Preserving" Trick

To test this, the researchers didn't trick the robot with wrong answers. Instead, they played a game of "same task, different story." They called this Task-Preserving Perturbation.

Think of it like teaching someone to drive.

The Standard Way: You show them a video of a car stopping at a red light. (Correct example).
The "Perturbed" Way: You show them a video of a different car stopping at a red light, but this time the car is a bright yellow convertible, the weather is sunny, and the driver is wearing a hat. The action (stopping at a red light) is still correct. The rule hasn't changed.

However, the researchers found that if you show the robot too many of these "different-looking but correct" examples, the robot starts to get confused about what the real rule is. It starts thinking, "Oh, maybe the rule is about yellow convertibles in sunny weather," rather than "Stop at red lights."

The Core Concept: "Contextual Evidence Shift"

The paper introduces a fancy term for this confusion: Contextual Evidence Shift.

Imagine the robot is a detective trying to solve a mystery. The examples you give it are "clues."

Clean Clues: All the clues look similar and point clearly to the same suspect.
Perturbed Clues: The clues are still technically true, but they look very different from each other. Some are written in red ink, some in blue; some are long stories, some are short notes.

Even though every clue is true, the mixture of clues changes. The detective (the robot) starts to focus on the wrong details (like the color of the ink) instead of the main crime. The "evidence" the robot uses to make a decision has shifted, even though the facts haven't changed.

What They Found

The researchers tested this on three types of tasks:

Sentiment Analysis: Deciding if a movie review is positive or negative.
Logical Reasoning: Solving puzzles like "If A is true, and A implies B, is B true?"
Math Word Problems: Solving simple math stories.

The Results:

Small Robots Get Hurt More: Smaller, less powerful models (like a 7-billion parameter model) got very confused when the examples looked different. Their accuracy dropped significantly.
Big Robots Are Tougher: Larger, smarter models (like 70-billion parameter models) were much better at ignoring the "noise" and sticking to the real rule. They didn't get as confused.
More Confusion = Worse Results: The more "different-looking" examples they added, the worse the small robots performed. In some cases, the robot did worse with these "correct" examples than if they had given it no examples at all.

A Simple Analogy: The Music Playlist

Imagine you are trying to teach a DJ to play "Upbeat Pop" music.

Standard Examples: You give them a playlist of 10 upbeat pop songs. They learn the vibe perfectly.
Perturbed Examples: You give them 10 songs that are still upbeat pop, but you change the cover art, the album names, and the order of the tracks. The music is the same genre, but the "packaging" is weird.

If you give the DJ a mix of standard and weirdly packaged songs, a novice DJ might get confused and start playing slow jazz because they are focusing on the weird album covers instead of the beat. An expert DJ (the large model) would just ignore the covers and play the right music.

The Takeaway

The paper concludes that when we try to teach AI models using examples, we shouldn't just ask, "Is this example correct?"

We also have to ask, "Does this example fit the pattern of the other examples?"

If the examples are all "correct" but they look too different from each other, they can actually hurt the AI's ability to learn. The AI needs a consistent "vibe" or "context" to figure out the rules, not just a pile of factually true but stylistically chaotic data.

In short: Correctness is necessary, but it is not enough. The way you present the correct information matters just as much as the information itself.

Technical Summary: When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning

Problem Statement
In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks by conditioning on a small set of input–output demonstrations without updating parameters. A prevailing intuition suggests that demonstrations are beneficial because they provide correct examples of the target task mapping. However, this work identifies a counterintuitive phenomenon: exemplar correctness does not guarantee exemplar utility. The authors demonstrate that demonstrations can be individually valid under the task definition (i.e., the input–output pair satisfies the task mapping) yet still degrade ICL performance when used as contextual evidence. This "correctness–utility gap" arises because the model does not merely learn a mapping from individual pairs but infers a latent task, input regime, and decision rule from the mixture of all provided exemplars.

Methodology: Task Preserving Perturbations
To isolate the distinction between correctness and utility, the authors introduce task preserving perturbations. This framework involves modifying the input side of in-context exemplars while strictly maintaining the task definition, prompt format, and demonstration order. Crucially, every perturbed exemplar is paired with the target induced by the same task mapping $g_T$ , ensuring the demonstration remains "correct" in isolation.

The framework encompasses two regimes:

Label Updating Perturbations: Task-relevant semantics are altered, and the target is recomputed according to the task mapping (e.g., changing a sentiment sentence from positive to negative and updating the label).
Strict Target-Preserving Perturbations: The original target remains valid, but the input is modified to include distractions or reordering (e.g., adding irrelevant numerical facts to a math problem or inserting distracting premises in logical reasoning).

The authors define the perturbation ratio $\rho$ as the proportion of exemplars replaced by these task-preserving variants. They formalize the resulting failure mode as contextual evidence shift: perturbations alter the effective mixture of evidence the model uses for inference, shifting the model's inferred decision context even though the underlying task mapping remains unchanged.

Key Contributions

Identification of the Correctness–Utility Gap: The paper establishes that a demonstration can satisfy the condition of correctness ( $y_i = g_T(x_i)$ ) while having negative utility (increasing risk on clean queries) due to its interaction with the broader context.
Methodological Framework: The introduction of task preserving perturbations as a controlled intervention. Unlike prior adversarial studies that alter labels or task semantics, this method preserves the task mapping to specifically test how input distribution shifts affect ICL.
Theoretical Formalization: The concept of contextual evidence shift is formalized, showing how perturbations change the effective evidence mixture ( $m_\theta$ ) used for contextual inference, thereby separating the nominal perturbation ratio from the effective weight the model assigns to perturbed evidence.

Experimental Results
The authors evaluated open-weight instruction-tuned models (Llama-2, Llama-3.1, Qwen2.5/3.5, Gemma-4) across three domains: sentiment classification (SST-2), logical reasoning (ProverQA), and math word problems (PROBLEMATHIC).

Performance Degradation: Across all tasks, increasing the perturbation ratio $\rho$ substantially degraded ICL performance. In some cases (e.g., SST-2 with smaller models), performance dropped below zero-shot baselines.
Model Scale Sensitivity: The degradation was strongly scale-dependent. Smaller models (e.g., 7B–14B) exhibited significant vulnerability, while larger models (e.g., 70B+) remained relatively stable. For instance, on SST-2, Llama-2-7B accuracy dropped from 96.4% to 56.9% at 100% perturbation, whereas Llama-3.1-70B remained near 100%.
Task Difficulty: Harder tasks (e.g., complex logical reasoning or math problems) showed greater sensitivity to perturbations, particularly for smaller models.
Positional Effects: Perturbations placed near the query (tail perturbations) were disproportionately harmful to smaller models, which exhibited strong recency sensitivity. Attention map analysis revealed that larger models were better able to reassign attention to early, clean exemplars, mitigating the shift in evidence.
Control Experiments: Replacing exemplars with task-irrelevant factual statements caused only mild degradation, confirming that the harm stems from task-valid but distributionally shifted evidence rather than generic prompt disruption or task recognition failure.

Significance and Claims
The paper argues that robust ICL requires evaluating not only whether demonstrations are correct but also how they influence contextual inference. The authors claim that the effectiveness of a demonstration set depends on its global composition and the resulting evidence mixture, not just individual example quality.

The study concludes that the "correctness" of an exemplar is a local property, whereas "utility" is context-dependent. Consequently, standard ICL pipelines that select exemplars based solely on correctness or similarity may inadvertently introduce harmful contextual evidence. The authors position their work as an explanatory abstraction (contextual evidence shift) rather than a complete mechanistic account, noting limitations such as the focus on text-only ICL and controlled perturbations. They suggest future work should explore defense mechanisms that select or weight demonstrations by utility rather than correctness alone.