💬 NLP

Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

This paper reveals that many-shot chain-of-thought in-context learning behaves as in-context test-time learning rather than simple pattern matching, demonstrating that standard scaling rules fail for reasoning tasks and proposing Curvilinear Demonstration Selection (CDS) to optimize demonstration ordering for significant performance gains.

Original authors: Tsz Ting Chung, Lemao Liu, Mo Yu, Dit-Yan Yeung

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tsz Ting Chung, Lemao Liu, Mo Yu, Dit-Yan Yeung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart but inexperienced student how to solve a complex puzzle. You have a massive notebook (the "long context window") where you can write down hundreds of examples to help them.

This paper investigates what happens when you fill that notebook with hundreds of examples that include not just the question and answer, but also the step-by-step thinking process (called "Chain-of-Thought") used to solve it.

Here is the simple breakdown of what the researchers found:

1. The Old Rule vs. The New Reality

The Old Rule (For Simple Tasks):
If you are teaching a student to sort mail (a simple task), it doesn't matter much if you show them 20 examples or 200, or if you show them in a random order. The more examples you give, the better they get, and the order doesn't really matter. It's like throwing a bunch of red balls into a bucket; they all look the same, so the order is irrelevant.

The New Reality (For Reasoning Tasks):
When the task is hard—like solving a geometry proof or a detective mystery—the old rules break.

More isn't always better: If you just dump hundreds of reasoning examples into the notebook for a standard AI model, it often gets confused and performs worse.
Order matters a lot: Unlike sorting mail, the order in which you present these complex reasoning steps is critical. If you jump from a simple concept to a super hard one too quickly, the student gets lost.

2. Why "Similar" Examples Fail

Usually, when we want to help a student, we look for examples that look similar to the current problem.

The Trap: The paper found that for reasoning tasks, picking examples that look similar (e.g., two math problems both about triangles) is actually a bad idea.
The Analogy: Imagine you are teaching someone how to bake a cake. You show them a recipe for a chocolate cake, and then you ask them to bake a sponge cake. They look similar (both are cakes), but the process (the steps) is different. If the student tries to copy the chocolate steps for the sponge cake, they will fail.
The Finding: For reasoning, the "steps" (the procedure) matter more than the "topic" (the surface similarity). Two problems can look alike but require completely different thinking paths. If the AI tries to copy the wrong path, it fails.

3. The "Curriculum" Approach

The authors realized that for an AI to truly "learn" from a long list of examples, the list needs to act like a school curriculum, not just a pile of books.

Principle 1: Understandability. The examples must be written in a way the specific AI can understand. If the AI is "weak," showing it examples written by a "genius" AI might be too confusing. It learns best from examples that match its own current level of thinking.
Principle 2: Smooth Transitions. The examples need to be ordered so that the concepts flow gently from one to the next. You can't jump from "addition" to "calculus" without the steps in between.

4. The Solution: "Curvilinear Demonstration Selection" (CDS)

To fix the ordering problem, the researchers invented a method called CDS.

The Metaphor: Imagine the examples are points on a map. If you connect them in a random order, you might have to make sharp, dizzying turns (high "curvature"). If you arrange them so the path is smooth and gentle (low "curvature"), the student can walk the path easily.
The Result: By arranging the hundreds of examples so the "thinking path" is smooth and gradual, they improved the AI's performance on geometry problems by about 5.4% compared to random ordering.

Summary

The paper argues that giving an AI hundreds of reasoning examples isn't just about "more data." It's about how that data is presented.

Don't just grab similar-looking examples.
Do arrange them like a smooth lesson plan, moving gently from easy concepts to harder ones.
Do make sure the examples are written in a language the specific AI can actually understand.

By treating the long list of examples as a structured lesson rather than a retrieval buffer, the AI can actually "learn" the reasoning process in real-time, rather than just guessing based on patterns.

Technical Summary: Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

1. Problem Statement

While In-Context Learning (ICL) has been shown to adapt Large Language Models (LLMs) to new tasks without parameter updates, current understanding of its scaling behavior is largely derived from non-reasoning tasks (e.g., classification). With the advent of long-context models, "many-shot" ICL (using dozens to hundreds of examples) has emerged, achieving performance comparable to fine-tuning. However, it remains unclear whether the established principles of many-shot ICL—specifically that demonstration order diminishes in importance and that similarity-based retrieval is effective—transfer to Chain-of-Thought (CoT) ICL for reasoning tasks.

The paper investigates whether providing more reasoning demonstrations leads to reliable improvement or introduces new instabilities. It questions whether many-shot CoT-ICL is merely large-scale pattern matching or a form of genuine in-context learning that follows pedagogical principles.

2. Methodology

The authors establish an experimental framework spanning three dimensions:

Task Types: Non-reasoning tasks (classification, NLU, banking intent detection) vs. Reasoning tasks (mathematical reasoning on GSM8K/MATH, narrative reasoning on DetectiveQA).
Model Types: Non-reasoning LLMs (e.g., LLaMA 3.1/3.3, Qwen 2.5) vs. Reasoning-oriented LLMs (e.g., Qwen 3, QwQ, DeepSeek-R1) which expose explicit reasoning segments (e.g., <thought> tokens).
ICL Configurations: Traditional ICL vs. CoT-ICL (prompts containing input, reasoning chain, and output).

The study evaluates scaling effects by varying the number of demonstrations ( $n$ ) from few-shot to many-shot (up to $n=128$ ). Key experimental analyses include:

Scaling Disparity: Comparing performance trends as $n$ increases across different model and task combinations.
Similarity-Based Retrieval: Constructing "most-similar" and "most-dissimilar" demonstration sets based on question-level embedding similarity to test the retrieval hypothesis.
Order Sensitivity: Measuring the standard deviation of accuracy across random permutations of the same demonstration set.
Procedural Ablation: Replacing specific CoT rationales with static chains to isolate whether models learn procedures or merely match input-output pairs.
Demonstration Selection Principles: Testing the impact of "Ease of Understanding" (using self-generated vs. dataset-provided CoTs) and "Smoothness of Information Flow" (ordering demonstrations to minimize conceptual curvature).

3. Key Findings

A. Setting-Dependent Scaling

Non-Reasoning Tasks: Many-shot ICL scales reliably; performance improves steadily as the number of demonstrations increases, and sensitivity to demonstration order diminishes.
Reasoning Tasks (Non-Reasoning LLMs): Many-shot CoT-ICL is unstable. Increasing demonstrations often leads to performance degradation or negative gains for standard instruction-tuned models, even at large parameter scales (e.g., LLaMA 3.3 70B).
Reasoning Tasks (Reasoning-Oriented LLMs): Models with explicit reasoning capabilities (e.g., Qwen 3, DeepSeek-R1) show consistent, near-monotonic performance improvements with more CoT demonstrations. This suggests that benefiting from long CoT contexts requires mechanisms to interpret reasoning signals (e.g., thinking tokens) rather than shallow pattern matching.

B. Failure of Similarity-Based Retrieval

Non-Reasoning: Semantic similarity between questions is a reliable proxy for label similarity; retrieving similar examples improves performance.
Reasoning: Question-level semantic similarity fails to predict procedural compatibility. Two mathematically similar questions may require different solution strategies. Consequently, retrieving the "most-similar" CoT demonstrations often yields worse performance than using dissimilar or uncurated sets, as it can induce conflicting intermediate steps.

C. Order-Scaling Effect

Contrary to non-reasoning tasks where order sensitivity fades with scale, reasoning tasks exhibit an order-scaling effect: performance variance increases as the number of CoT demonstrations grows. This indicates that the sequence of demonstrations critically shapes the induced reasoning trajectory.

D. Mechanism: In-Context Test-Time Learning

The paper reframes many-shot CoT-ICL not as pattern matching, but as in-context test-time learning. Evidence shows that reasoning models absorb specific procedures from demonstrations rather than just memorizing labels. When rationales are corrupted (replaced with static chains), performance drops significantly at high shot counts, confirming that models internalize the procedural steps.

4. Proposed Principles and Method (CDS)

Based on the finding that CoT-ICL acts as a learning process, the authors propose two principles for effective demonstration design:

Ease of Understanding: Demonstrations must align with the model's current knowledge and generation distribution. Self-generated CoTs (even with some errors) often outperform ground-truth CoTs for weaker models because they are more "understandable" to the target model.
Smoothness of Knowledge Progression: Demonstrations should be ordered to create a smooth conceptual transition, avoiding abrupt jumps in reasoning strategies.

To operationalize the second principle, the authors introduce Curvilinear Demonstration Selection (CDS).

Method: CDS treats the sequence of demonstrations as a trajectory in embedding space. It orders demonstrations to minimize the total conceptual curvature (the sum of turning angles between consecutive embedding vectors).
Implementation: It uses a TSP-based heuristic (nearest-neighbor + 2-opt) with a cost function balancing Euclidean proximity and curvature minimization.
Results: CDS yields consistent gains, achieving up to a 5.42 percentage-point improvement on geometry tasks with 64 demonstrations. It outperforms random ordering and high-curvature baselines across math and narrative reasoning tasks.

5. Significance and Claims

The paper claims to reframe the understanding of long-context windows from a simple "retrieval buffer" into a "structured curriculum" for in-context test-time learning.

Theoretical Contribution: It demonstrates that the rules governing many-shot ICL for classification do not transfer to reasoning. It argues that effective reasoning ICL relies on the model's ability to interpret and internalize procedural signals, not just surface pattern matching.
Practical Contribution: It provides a simple, low-cost, parameter-free method (CDS) to improve reasoning performance by optimizing the order of demonstrations based on conceptual smoothness.
Limitations: The authors note that the benefits of CDS are most pronounced when the baseline ordering is suboptimal (high curvature) and that gains may be smaller in tasks where baseline accuracy is already very high (e.g., number theory).

The work suggests that for reasoning-capable LLMs, the long context window serves as a dynamic learning environment where the structure of the prompt (curriculum) is as critical as the content of the examples.