💬 NLP

Not the Example, but the Process: How Self-Generated Examples Enhance LLM Reasoning

This paper demonstrates that the reasoning performance gains in Large Language Models from self-generated few-shot examples stem primarily from the cognitive process of creating the problems within a unified prompt, rather than from the utility of the generated examples themselves.

Original authors: Daehoon Gwak, Minseo Jung, Junwoo Park, Minho Park, ChaeHun Park, Junha Hyung, Jaegul Choo

Published 2026-02-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daehoon Gwak, Minseo Jung, Junwoo Park, Minho Park, ChaeHun Park, Junha Hyung, Jaegul Choo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly confused, robot how to solve a tricky math problem. You have three ways to do it. This paper is about figuring out which way actually works best and why.

Here is the breakdown in simple terms, using some everyday analogies.

The Big Question: Is it the "Answer Key" or the "Study Session"?

Scientists noticed that when they ask Large Language Models (LLMs) to solve a hard problem, the robot gets better if it first generates its own practice examples.

But there was a mystery: Does the robot get smarter because it has the practice examples (the answer key), or does it get smarter because it went through the process of making them (the study session)?

To find out, the researchers set up a test with three different "teaching styles":

1. The "Zero-Shot" Approach (The Cold Start)

The Analogy: You hand the robot a math test and say, "Solve this." No hints, no practice, no warm-up.
The Result: The robot tries its best, but it often stumbles because it hasn't warmed up its brain.

2. The "Decoupled" Approach (The Borrowed Notes)

The Analogy: Imagine the robot first writes a practice problem and solves it on a piece of paper. Then, you take that paper away, copy the final answer onto a clean sheet, and hand that clean sheet to the robot as a "hint" before it takes the real test.
The Catch: The robot sees the example, but it didn't do the work of creating it in the current conversation. It's like reading someone else's notes right before an exam.
The Result: This is slightly better than doing nothing, but not by much. The robot still feels a bit cold.

3. The "Integrated" Approach (The Live Workshop)

The Analogy: You tell the robot: "First, create a similar practice problem and solve it right here, out loud. Then, use that thinking to solve the real test question."
The Catch: The robot has to build the bridge while it is walking across it. The act of creating the example happens in the same "moment" as solving the real problem.
The Result: This wins every time. The robot performs significantly better.

The Big Discovery: It's About the Process, Not the Product

The paper's main conclusion is surprising: The magic isn't in the example itself; it's in the act of making it.

Think of it like cooking:

Decoupled is like being given a perfect recipe card. You can follow it, but you haven't really learned the feel of the ingredients.
Integrated is like being told, "Go chop some onions and figure out how to sauté them, then use that skill to cook the main dish."
Even if the onions you chop for the practice dish are slightly different from the main dish, the act of chopping and heating them warms up your hands and your brain. You are "in the zone."

The researchers found that when the robot just reads an example (Decoupled), it barely uses it. But when it creates the example (Integrated), it gets deeply engaged. It's like the difference between reading a map and actually driving the route; driving teaches you the turns in a way reading never could.

The "Brain Scan" Evidence (Attention Analysis)

To prove this, the researchers looked inside the robot's "brain" (its attention mechanisms). They found two cool things:

Deep Focus: When the robot is in "Integrated" mode, it pays much more attention to the example it just created. It's like a student who just solved a problem is hyper-focused on their own logic.
The Whole Journey: In the "Integrated" mode, the robot uses different parts of its brain at different times. First, it uses the "beginning" parts of its brain to understand the context of creating the problem. Then, it uses the "advanced" parts to solve the final answer.
- In the "Decoupled" mode, the robot skips the "creation" phase, so it misses out on that initial warm-up. It's like trying to run a marathon without a warm-up jog; your muscles (the reasoning layers) aren't ready.

Why This Matters

This paper tells us that if we want AI to be smarter at reasoning, we shouldn't just feed it a library of pre-written examples. Instead, we should design prompts that force the AI to think out loud and generate its own examples right before solving the problem.

The Takeaway:
Don't just give the AI the answer key. Ask it to write the practice test first. The struggle of creating the example is what actually makes the AI smarter.

1. Problem Statement

Recent advancements have shown that Large Language Models (LLMs) can improve reasoning performance by generating their own few-shot examples (self-generation) rather than relying on manually curated ones. However, the underlying mechanism driving this improvement remains unclear. It is ambiguous whether the performance gain stems from:

The quality/content of the generated examples themselves.
The cognitive process of generating those examples (i.e., the act of creation).

This uncertainty makes it difficult to design optimal prompting strategies or determine when self-generation is effective. The paper aims to isolate these factors to understand the true source of reasoning enhancement.

2. Methodology

The authors conducted systematic experiments across five diverse LLM architectures (GPT-4.1, GPT-4O, GEMINI-1.5-FLASH, LLAMA-3.3-70B-INSTRUCT, and LLAMA-3.1-8B-INSTRUCT) on two standard mathematical reasoning benchmarks: MATH (competition-level problems) and GSM8K (grade-school word problems).

They compared three distinct prompting strategies to isolate the variable of "problem creation":

Zero-shot Prompting: The model solves the test question directly without any context or examples.
Integrated Prompting: The model is instructed to create a related problem and solve it, and then immediately solve the original test question, all within a single unified prompt. This forces the model to engage in the creation process immediately before solving.
Decoupled Prompting: The model first generates a problem and solution (in a separate interaction or step). This generated example is then provided to the model as a static in-context example to solve the test question, excluding the conversational history of how the example was created.

Key Experimental Controls:

Multi-shot Analysis: Tested 1-shot, 2-shot, and 4-shot configurations to see if the advantage persists with more examples.
Attention Analysis: Used the open-source LLAMA-3.1-8B-INSTRUCT to analyze internal attention patterns during token generation.
Ablation Studies:
- Dummy Examples: Replaced generated math problems with random text to test if mere text length drives gains.
- Example Repetition: Explicitly repeated Decoupled examples to test if location alone matters.
- Quality Control: Reused high-quality examples generated by Integrated prompting in a Decoupled setup to test if example quality was the differentiator.
- Chain-of-Thought (CoT): Evaluated strategies with and without explicit CoT reasoning.
- Intention Statements: Added prompts where the model explained why it generated an example.

3. Key Contributions

Mechanism Identification: The paper establishes that the primary driver of performance improvement in self-generation is not the generated examples themselves, but the process of creating them.
Unified vs. Decoupled Superiority: The authors demonstrate that "Integrated" prompting (creation + solving in one context) consistently outperforms "Decoupled" prompting (using the example as a static guide), even when the examples are identical.
Attention Pattern Insights: Through layer-wise attention analysis, the paper reveals that Integrated prompting alters how the model attends to context. Specifically, it increases attention to self-generated examples in both early layers (contextual encoding) and deep layers (reasoning/decision-making), whereas Decoupled prompting shifts attention heavily toward the test question tokens, neglecting the generated context.
Robustness: The findings hold across different model sizes, architectures, difficulty levels, and even when Chain-of-Thought reasoning is employed.

4. Results

Performance Gains: Integrated prompting consistently achieved the highest accuracy.
- Example (MATH, GPT-4O): Integrated (37.86%) > Zero-shot (33.33%) > Decoupled (32.65%).
- Example (GSM8K, LLAMA-3.3-70B): Integrated (35.19%) > Zero-shot (32.67%) > Decoupled (33.59%).
Decoupled Limitations: Decoupled prompting offered only marginal gains over Zero-shot and, in some cases (e.g., LLAMA-3.1-8B on GSM8K), performed slightly worse than Zero-shot.
Ablation Findings:
- Text Length: "Dummy" Integrated prompts (random text) showed slight improvement over Zero-shot but significantly less than real Integrated prompts, suggesting the content of the reasoning process matters, not just the act of generating text.
- Example Quality: Reusing high-quality examples from Integrated prompts in a Decoupled setup did not match the performance of the Integrated setup, proving that the generation process is the critical factor, not just the example quality.
- CoT: The advantage of Integrated prompting persisted even when CoT was used, though the gap narrowed slightly, suggesting CoT partially compensates for the lack of a generation process.
Attention Analysis:
- Integrated: The model allocates significantly more attention to the self-generated example tokens ( $p < 10^{-10}$ ) compared to Decoupled.
- Layer-wise: Integrated prompting shows higher attention scores in early layers (0–12) for contextual grounding and upper layers (23–31) for final decision-making. Decoupled prompting shows a "transitional" behavior where induction heads (middle layers) function similarly, but the model fails to leverage the context effectively in the final reasoning layers.

5. Significance and Implications

Redefining In-Context Learning: The study challenges the assumption that the value of few-shot learning lies solely in the examples provided. Instead, it suggests that the cognitive engagement required to generate examples primes the model's internal reasoning mechanisms.
Prompt Engineering Strategy: For complex reasoning tasks, prompting strategies should prioritize unified, integrated workflows where the model "thinks" about creating a similar problem before solving the target. Simply providing pre-generated examples (even if self-generated) without the creation context is suboptimal.
Interpretability: The attention analysis provides a mechanistic explanation for why integrated prompting works: it ensures the generated context is deeply embedded in the model's representation from the initial encoding phase through to the final decision layer.
Future Directions: The authors suggest that future systems could benefit from iterative refinement, self-verification, and explicitly stating the "intention" behind generated examples to further enhance the creation-and-solving loop.

In conclusion, the paper argues that the act of problem creation is a form of reasoning rehearsal that activates the model's capabilities more effectively than passively consuming examples, offering a new paradigm for designing robust LLM prompting strategies.