🤖 machine learning

Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuning

This paper systematically demonstrates that the optimal data difficulty for supervised fine-tuning of large language models depends on the dataset size, revealing a trade-off between generalization and extrapolation where larger data budgets benefit from harder examples.

Original authors: Siyuan Liu (IIIS, Tsinghua University), Tinghong Chen (College of AI, Tsinghua University,Shanghai Qi Zhi Institute), Xinghan Li (IIIS, Tsinghua University), Yifei Wang (Amazon AGI SF Lab), Jingzhao Z

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyuan Liu (IIIS, Tsinghua University), Tinghong Chen (College of AI, Tsinghua University,Shanghai Qi Zhi Institute), Xinghan Li (IIIS, Tsinghua University), Yifei Wang (Amazon AGI SF Lab), Jingzhao Zhang (IIIS, Tsinghua University,Shanghai Qi Zhi Institute)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly rigid, robot how to solve math problems. You have a huge library of practice questions, ranging from "What is 2+2?" to "Solve this complex calculus equation."

The big question researchers asked is: Should you feed the robot only the easy questions, only the hard ones, or a mix?

For a long time, experts argued about this. Some said, "Throw out the easy stuff; it's boring and doesn't teach anything!" Others said, "Stick to the easy stuff; if you go too hard, the robot will get confused and forget what it already knows."

This paper says: You were both right, but only half the time. The answer depends entirely on how much time (or data) you have to train the robot.

Here is the simple breakdown of their findings:

1. The "Goldilocks" Zone Changes Size

The researchers found that there is no single "perfect" difficulty level for all situations. Instead, the perfect difficulty level shifts depending on how much data you have.

If you have a tiny amount of data: You should stick to easier questions.
- Analogy: Imagine you only have 10 minutes to study for a test. If you try to read a dense, 500-page textbook (hard data), you won't finish, and you'll be confused. It's better to read a clear, simple summary (easy data) so you actually learn the basics without getting overwhelmed.
If you have a massive amount of data: You should switch to harder questions.
- Analogy: Now imagine you have a whole semester to study. If you only read the simple summaries, you'll get bored and stop improving. You need to tackle the tough textbook chapters to really master the subject.

2. The Two "Gaps" (The Why)

Why does this happen? The paper explains it using two invisible "gaps" that the robot has to bridge.

Gap A: The "Confusion Gap" (Extrapolation Gap)
- This happens when the training data is too easy. The robot learns the easy stuff perfectly but gets totally lost when it sees a hard question on the real test. It's like learning to ride a bike on a flat sidewalk and then being thrown onto a mountain trail.
- Solution: You need harder training data to bridge this gap.
Gap B: The "Overwhelm Gap" (Generalization Gap)
- This happens when the training data is too hard and you don't have enough of it. The robot tries to memorize the hard answers but gets confused, forgets the basics, and fails even the easy questions. It's like trying to learn advanced physics before you know how to count.
- Solution: You need more data (or easier data) to fix this.

The Magic Balance:

Small Data Budget: The "Overwhelm Gap" is the biggest danger. So, you pick easier data to make sure the robot actually learns something without breaking.
Large Data Budget: You have so much data that the robot can handle the hard stuff without getting overwhelmed. Now, the "Confusion Gap" becomes the bigger problem. So, you switch to harder data to prepare the robot for the toughest challenges.

3. The Robot's Starting Level Matters

The paper also notes that "difficulty" is relative. A question that is "hard" for a beginner robot might be "easy" for a super-smart robot.

If your robot is already very smart, you can feed it harder data sooner.
If your robot is weaker, you need to stick to easier data for longer.

4. The "Smart Filter" (Dynamic Fine-Tuning)

The researchers also looked at a method called "Dynamic Fine-Tuning" (DFT), which is like a teacher who automatically highlights the parts of a sentence the robot already understands and focuses on the tricky parts.

The Result: This method is great when you have very little data because it helps the robot avoid getting overwhelmed.
The Catch: If you have a huge amount of data, this method actually holds the robot back. It keeps the robot too comfortable, preventing it from tackling the really hard problems that a standard training method would eventually solve.

The Bottom Line

There is no "one-size-fits-all" rule for picking training data.

Small dataset? Stick to easier, clearer examples.
Huge dataset? Challenge the model with harder examples.

The key is to match the difficulty of the training material to the size of your dataset and the current ability of the model, rather than just blindly picking the "hardest" or "easiest" stuff available.

Technical Summary: Data Difficulty and the Generalization–Extrapolation Tradeoff in LLM Fine-Tuning

Problem Statement

Supervised fine-tuning (SFT) is a critical step in adapting Large Language Models (LLMs) to specific tasks, yet the role of data difficulty in this process remains controversial. Existing literature presents conflicting findings: some studies suggest that removing easy, uninformative data and prioritizing harder examples improves performance, while others argue that staying close to the base model's training distribution (favoring easier data) yields better generalization. A third view posits that intermediate difficulty is optimal. These inconsistencies suggest that data difficulty cannot be analyzed in isolation; its effectiveness likely depends on other factors, such as the available training data size. This paper addresses the open question: Under what conditions should SFT prioritize easy, hard, or intermediate-difficulty data?

Methodology

The authors employ a multi-faceted approach combining empirical analysis, controlled synthetic experiments, and theoretical derivation:

Empirical Analysis on Real Datasets:
- The authors conducted preliminary experiments on datasets like OpenR1-Math, OpenMath, and OpenScience, splitting data into easy, medium, and hard subsets based on Chain-of-Thought (CoT) length.
- They evaluated fine-tuned models (e.g., Qwen2.5-Math) on standard benchmarks (Math500, MMLU) to observe how performance varies with data difficulty and dataset size.
- They observed that removing hard data helps in small-data regimes, while removing easy data becomes beneficial in large-data regimes.
Controlled Synthetic Experiments (iGSM):
- To isolate variables, the authors utilized the iGSM framework, which generates grade-school math reasoning problems with precise control over difficulty (measured by the number of arithmetic operations, or "ops").
- They performed systematic two-dimensional studies varying both data size (from 1k to 128k examples) and data difficulty (ops count) on base models (Qwen2.5-Math-1.5B and 7B).
- They decomposed test performance by difficulty levels to analyze failure modes (e.g., poor extrapolation vs. poor in-distribution generalization).
Theoretical Analysis:
- The authors provided a theoretical explanation using PAC-Bayesian generalization bounds.
- They formalized the error decomposition into two components: the in-distribution (IID) generalization gap and the extrapolation gap.
- They analyzed how these gaps interact with data size and difficulty to derive the observed trends.
Extension to Token-Level Selection:
- The study extended its analysis to Dynamic Fine-Tuning (DFT), a token-level reweighting method that implicitly favors easy tokens, to compare its behavior against standard SFT under varying data conditions.

Key Findings and Results

1. Non-Monotonic Relationship and Optimal Difficulty

The experiments reveal that SFT performance does not change monotonically with data difficulty. Instead, there exists an optimal data difficulty level for a fixed data budget.

Too Easy: Models fit the training distribution well but fail to generalize to harder test cases (large extrapolation gap).
Too Hard: Models struggle to learn effectively from the training data itself, leading to poor in-distribution generalization and forgetting of base capabilities (large generalization gap).
Intermediate: An optimal "sweet spot" exists where the trade-off between these two gaps is minimized.

2. The Data Size–Difficulty Tradeoff

A central finding is that the optimal difficulty shifts toward harder data as the data budget increases.

Small Data Regime: Removing hard examples improves performance because the model cannot generalize from difficult data with limited samples; easier data ensures better in-distribution learning.
Large Data Regime: Removing easy examples improves performance because the model has sufficient data to learn from harder examples, thereby reducing the extrapolation gap to challenging test cases.

3. Relative Difficulty

The effectiveness of data difficulty is relative to the base model's capability. A stronger base model can successfully learn from a broader and harder range of data compared to a weaker model, shifting the optimal difficulty curve toward higher difficulty levels.

4. Mechanism: Generalization vs. Extrapolation

The authors identify the underlying mechanism as a trade-off between two gaps:

Extrapolation Gap: The performance degradation caused by the mismatch between the training distribution and harder, out-of-distribution test cases. This gap decreases as training data becomes harder (better covering the test distribution).
Generalization Gap: The performance degradation due to imperfect learning within the training distribution. This gap increases as data becomes harder (requiring larger model adaptations relative to the prior).
Data Size Effect: Increasing data size reduces the generalization gap (scaling as $O(1/\sqrt{n})$ ) but leaves the extrapolation gap largely unchanged. This reduction allows the model to tolerate harder training data, shifting the optimum.

5. Comparison with DFT

In the context of Dynamic Fine-Tuning (DFT), which favors easy tokens:

DFT exhibits a smaller generalization gap and is less sensitive to data difficulty, outperforming SFT when data is too difficult for the available size.
However, DFT saturates faster and is eventually outperformed by standard SFT when sufficient data is available to learn from harder examples.

Significance and Claims

The paper claims to provide a principled explanation for previously conflicting observations in the literature regarding data selection for SFT. By connecting empirical phenomena with theoretical analysis, the authors establish that:

There is no universally optimal difficulty level; effectiveness is jointly determined by data size and base model capability.
The trade-off between the in-distribution generalization gap and the extrapolation gap is the fundamental driver of these behaviors.
Effective data selection strategies should adapt data difficulty to the available data scale rather than relying on fixed heuristics (e.g., always removing easy or hard data).

The authors note that while they establish the existence of an optimal difficulty, efficiently identifying this optimum in complex, real-world data distributions remains an open challenge. They suggest that adaptive search strategies over data difficulty could be a practical direction for optimizing SFT performance under fixed data budgets.