🤖 AI

Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics

This paper introduces Relative Transfer as a principled measure to explain the varying effectiveness of curriculum learning in large language models and proposes Transfer-aware Dynamic Curriculum Sampling (TDCS), a method that dynamically adjusts data sampling based on cross-difficulty transfer relationships to consistently outperform existing scheduling strategies across diverse reasoning tasks and model scales.

Original authors: Zhikai Ding, Ziyi Ye

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhikai Ding, Ziyi Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers are constantly trying to teach massive computer programs, known as large language models, to solve complex problems like math puzzles, coding challenges, and logical riddles. To do this, they feed the models vast amounts of training data. For years, a popular idea has been to organize this data like a school curriculum: start with the easiest examples and gradually introduce harder ones. This approach, called curriculum learning, assumes that just as a human student learns better by mastering simple concepts before tackling difficult ones, a machine should follow the same path. However, when scientists applied this method to different types of reasoning tasks, the results were confusing. Sometimes the "easy-to-hard" approach worked wonders; other times, it performed no better than random guessing, or even worse. This inconsistency raised a fundamental question: why does a strategy that works so well in one situation fail completely in another?

A team of researchers at Fudan University set out to solve this mystery by looking not at the schedule itself, but at what happens inside the model's brain during the learning process. They discovered that the success of a curriculum depends entirely on how the model transfers what it learns from one type of problem to another. Imagine a student learning to ride a bicycle; if mastering the bike helps them immediately learn to ride a motorcycle, the order of lessons matters less. But if learning the bike actually confuses them when they try the motorcycle, the order becomes critical. The researchers found that for some tasks, like solving Sudoku puzzles, learning the hardest examples automatically improved the model's ability to solve the easiest ones. For other tasks, like writing code or solving specific math word problems, learning the hardest examples did little to help with the easier ones, and sometimes even interfered.

To understand this phenomenon, the team developed a way to measure the "transfer" of knowledge between different levels of difficulty. They observed that when a model practices a difficult task, it generates a specific pattern of changes in its internal settings. The researchers measured how much this pattern helped or hurt the model's performance on easier tasks. They found that on Sudoku, the hardest problems provided a strong, positive boost to the easier ones, meaning the model could safely focus only on the difficult examples. In contrast, on tasks like code generation, the hardest problems offered very little help to the easier ones, and focusing solely on them caused the model to forget how to handle the simpler cases. This explained why a fixed "easy-to-hard" schedule failed on some tasks: it forced the model to ignore the easier examples that it still needed to practice, because the difficult examples weren't doing the heavy lifting for them.

Armed with this understanding, the researchers created a new method called Transfer-aware Dynamic Curriculum Sampling. Instead of following a rigid plan that moves from easy to hard, this new system acts like a smart coach that watches the model in real time. It constantly checks how well the current difficult examples are helping the model with the easier ones. If the difficult examples are providing a strong boost, the system lets the model focus on them. If the boost is weak or negative, the system automatically mixes in more practice on the easier examples to keep those skills sharp. This approach allows the training strategy to adapt to the specific nature of the task, rather than forcing every task into the same mold.

The results of this new approach were striking. When tested on three different reasoning benchmarks—Sudoku, a code generation dataset, and a mathematical reasoning dataset—the new method consistently outperformed all the standard strategies. On the Sudoku task, it improved accuracy by a significant margin over the best fixed schedule. On the code generation task, it also achieved the highest scores, proving that the model benefited from revisiting easier examples rather than pushing straight to the hardest ones. Perhaps most importantly, the method worked across different sizes of models, from smaller ones with 1.5 billion parameters to larger ones with 7 billion, and even when the model was teaching itself using its own generated answers.

The study suggests that there is no single "best" way to organize training data for all artificial intelligence tasks. The effectiveness of a curriculum is not a universal rule but a specific property of how knowledge flows between different levels of difficulty within a particular task. By measuring this flow and adjusting the training mix accordingly, the researchers were able to create a more robust and efficient learning process. This work moves the field beyond simple trial and error, offering a clear, measurable reason why certain training schedules succeed or fail. It provides a unified explanation for the varied performance of curriculum learning and offers a practical tool for building better, more adaptable artificial intelligence systems.

Technical Summary: Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics

Problem Statement
Curriculum Learning (CL), which organizes training data from easy to hard, is a widely adopted paradigm in the post-training of Large Language Models (LLMs). However, its effectiveness is inconsistent across different reasoning tasks; while it may outperform random sampling in some scenarios, it can be ineffective or even detrimental in others. Existing literature primarily offers empirical observations regarding difficulty metrics and fixed schedules but fails to answer a fundamental question: what determines whether curriculum learning is effective? Without understanding the underlying optimization mechanism, designing robust curriculum strategies remains reliant on empirical trial and error.

Methodology
The authors propose a shift from viewing curriculum learning as a predefined heuristic to analyzing the optimization dynamics induced by different schedules. The core methodology involves three stages:

Empirical Observation: Preliminary experiments on three reasoning benchmarks (Sudoku, KodCode, and iGSM) reveal that no single fixed scheduling strategy (Curriculum, Random, or Mix) consistently dominates across all tasks. The optimal strategy varies by task and changes dynamically during training.
Theoretical Analysis (Relative Transfer): The paper formalizes the interaction between difficulty levels as knowledge transfer. Using first-order optimization analysis, the authors derive a metric called Relative Transfer ($Re(i, j)$). This metric quantifies how optimizing a difficulty level $j$ $j$ influences the loss of another difficulty level $i$ $i$ , normalized by the self-optimization capability of $i$ $i$ .
- Mathematically, $Re(i, j) = \frac{g_i^T g_j}{g_i^T g_i}$ , where $g$ represents the gradient.
- A value $>0$ indicates positive transfer (optimizing $j$ helps $i$ ), while negative values indicate optimization conflicts.
- The analysis shows that tasks with strong positive transfer from hard to easy levels (e.g., Sudoku) benefit from pure curriculum schedules, whereas tasks with weak transfer (e.g., iGSM, KodCode) require replay of easier examples.
Algorithm Development (TDCS): Based on the transfer analysis, the authors introduce Transfer-aware Dynamic Curriculum Sampling (TDCS). This adaptive framework dynamically adjusts the sampling distribution based on the estimated Relative Transfer matrix. Key components include:
- Current Difficulty Ratio Mapping: Uses a sigmoid function to map the average transfer capability of the current difficulty to the sampling probability of that difficulty. High transfer capability leads to higher sampling of the current (harder) level; low transfer triggers more replay.
- Difficulty Allocation: For difficulty levels requiring replay, sampling probabilities are allocated via reverse exponential weighting, prioritizing levels that receive insufficient transfer.
- Harder Difficulty Adjustment: If a harder difficulty level provides exceptionally strong positive transfer to the current level, a portion of the sampling probability is reassigned to that harder level.

Key Contributions

Systematic Empirical Study: Demonstrated that no fixed scheduling strategy is universally optimal for LLM reasoning tasks, highlighting the task-dependent nature of curriculum effectiveness.
Theoretical Explanation: Provided a unified optimization-based explanation for curriculum learning success or failure through the lens of cross-difficulty knowledge transfer, formalized as Relative Transfer.
Adaptive Framework: Derived TDCS, a dynamic sampling strategy that outperforms fixed schedules by adapting to the estimated transfer relationships throughout training.

Experimental Results
Extensive experiments were conducted on multiple reasoning benchmarks (Sudoku, KodCode, iGSM), various model scales (Qwen2.5-1.5B, 3B, 7B, and Llama3.2-3B), and training paradigms (Supervised Fine-Tuning and Self-Improvement).

Performance: TDCS consistently achieved the best performance across all tasks and model scales, outperforming the strongest fixed baseline by 2.6% to 20.8% depending on the task and model.
Training Dynamics: Visualization of sampling distributions showed that TDCS adapts its behavior to the specific transfer characteristics of each task (e.g., shifting rapidly to harder difficulties for Sudoku while maintaining moderate replay for iGSM).
Ablation: Removing either the "Harder Adjustment" or "Exponential Difficulty Allocation" components resulted in performance degradation, confirming the necessity of both mechanisms for exploiting cross-difficulty transfer.

Significance
The paper claims to provide a unified optimization-based explanation of curriculum learning, moving beyond empirical observations of difficulty metrics to a principled understanding of cross-difficulty transfer. By characterizing the transfer relationship between difficulty levels, the work offers a theoretical foundation for when and why curriculum learning succeeds. The proposed TDCS framework demonstrates that dynamically adjusting sampling distributions based on these transfer dynamics leads to more stable and effective optimization across diverse LLM post-training scenarios. The authors suggest this transfer-based perspective serves as a foundation for future research into broader optimization settings and general data scheduling methods.

Technical Summary: Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics

More like this