🤖 machine learning

TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents

To address the issue of "Trajectory-Level KL Instability" caused by compounding errors in multi-turn agent training, this paper proposes **TCOD (Temporal Curriculum On-Policy Distillation)**, a framework that stabilizes training and improves performance by progressively increasing the trajectory depth exposed to the student model through a curriculum schedule.

Original authors: Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, James Cheng

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, James Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a toddler how to navigate a complex, multi-room obstacle course.

If you simply throw the toddler into the middle of the course and yell, "Follow the professional athlete's exact footsteps!" every time they make a tiny mistake, they will quickly get lost, frustrated, and eventually just sit in a corner and give up. This is exactly the problem the researchers found with current AI "agents."

Here is a breakdown of the paper, TCOD, using that analogy.

1. The Problem: The "Snowball of Mistakes"

Current AI training uses a method called On-Policy Distillation (OPD). Think of this as a "Pro Athlete" (the Teacher AI) watching a "Novice" (the Student AI) try to run the course. The Pro watches the Novice and says, "You should have moved your left foot like this."

The problem is that in a multi-turn task (like a long conversation or a complex game), mistakes compound.

The Snowball Effect: If the student takes one slightly wrong step at the beginning, they end up in a part of the room the Pro has never been in.
The Confusion: Because the student is now in a "weird" spot, the Pro doesn't know how to help anymore. The Pro’s advice becomes confusing or irrelevant.
The Collapse: In the paper, the researchers noticed that as the student makes more mistakes, the "gap" between the student and the teacher (called KL Divergence) explodes, and the student’s success rate crashes to zero. The student essentially "breaks."

2. The Solution: TCOD (The "Training Wheels" Approach)

The researchers proposed TCOD (Temporal Curriculum On-Policy Distillation). Instead of throwing the student into the deep end immediately, they use a Curriculum—a structured learning plan that gets harder only when the student is ready.

They created two clever ways to do this:

Method A: TCOD-F2B (The "Short Sprints" Method)

The Analogy: Instead of making the toddler run the whole 10-lap course, you first ask them to just walk from the start to the first cone. Once they master that, you ask them to go to the second cone. Slowly, you lengthen the distance.
How it works: The AI starts by only practicing the first few steps of a task. As it gets better, the "horizon" expands. It masters the "beginning" before it ever has to worry about the "end."

Method B: TCOD-B2F (The "Guided Finish" Method)

The Analogy: Imagine the toddler is struggling to finish the race. You pick them up, carry them through the messy, confusing middle part of the course, and place them right at the finish line. You say, "Now, you try to take the final three steps to win." Once they can do that, you move them back a little further and let them try the last four steps.
How it works: The "Teacher" AI acts as a navigator. It handles the difficult early parts of the task to get the student into a "success zone." The student then practices finishing the task from that safe spot. Gradually, the teacher steps back more and more until the student can do the whole thing alone.

3. The Results: A Smarter Student

The researchers tested this on several difficult digital environments (like virtual kitchens and shopping websites), and the results were impressive:

Stability: Unlike the old method, where the AI would "break" and stop succeeding, TCOD kept the training smooth and steady.
Efficiency: It actually saved time! Because the student wasn't wasting time wandering aimlessly in "error zones," the training finished up to 32% faster.
Surpassing the Teacher: Most amazingly, in some cases, the student actually became better than the teacher. By learning the "logic" of the task through this curriculum, the student developed a more robust way of solving problems than the expert it was imitating.

Summary in one sentence:

Instead of forcing a student to mimic an expert's entire long journey perfectly from day one, TCOD teaches them in small, manageable chunks—starting with short bursts or guided finishes—so they don't get lost in their own mistakes.

Technical Summary: TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents

1. Problem: Trajectory-Level KL Instability

The paper addresses a critical failure mode in On-Policy Distillation (OPD) when applied to multi-turn autonomous agents. While OPD is effective for single-turn reasoning (like math or QA), it struggles in long-horizon, interactive environments.

The authors identify a phenomenon they term Trajectory-Level KL Instability. In multi-turn settings, as a student model interacts with an environment, its small errors in action selection accumulate over time. This compounding error pushes the agent into states that are "out-of-distribution" (OOD) relative to the teacher's training or effective support. Consequently:

KL Divergence Escalates: The teacher assigns increasingly low probabilities to the student's actions as the trajectory progresses.
Success Rate Collapse: The training signal becomes unreliable, leading to a sharp drop in the agent's ability to complete tasks.
Training Instability: Even if the model eventually converges, it often starts with prohibitively high KL divergence, making the optimization process highly unstable.

2. Methodology: Temporal Curriculum On-Policy Distillation (TCOD)

To mitigate error accumulation, the authors propose TCOD, a framework that uses curriculum learning to control the "trajectory depth" (the number of interaction steps) exposed to the student. Instead of training on full, long-horizon trajectories from the start, TCOD progressively expands the horizon.

They introduce two practical variants:

Forward-to-Backward (TCOD-F2B): A "shallow-to-deep" approach. The student is restricted to a maximum of $k$ interaction steps. Initially, $k$ is small, forcing the student to master early-turn reasoning and immediate actions. As training progresses, $k$ is increased via a linear pacing schedule until the student can handle full-length trajectories.
Backward-to-Forward (TCOD-B2F): A "navigator" approach. The teacher model is used to "navigate" the agent to a near-terminal state (an intermediate successful state). The student then takes over from that state to complete the task. The curriculum works by gradually reducing the teacher's "prefix" (the number of steps the teacher performs) until the student must perform the entire trajectory from the initial state. This ensures the student always learns from "successful" starting points, bypassing early-turn errors.

3. Key Contributions

Identification of a New Failure Mode: The formal definition and empirical demonstration of "Trajectory-Level KL Instability" in multi-turn agent distillation.
Novel Curriculum Framework: The introduction of TCOD, providing two distinct ways (F2B and B2F) to manage temporal complexity in reinforcement learning/distillation.
Lightweight Implementation: The method requires minimal code modifications to existing on-policy distillation frameworks and does not require external difficulty metrics; difficulty is defined inherently by trajectory depth.
Efficiency Gains: The curriculum approach reduces total training time by up to 32% because early training stages involve shorter, faster trajectories.

4. Experimental Results

The authors evaluated TCOD across three benchmarks: ALFWorld (embodied), WebShop (e-commerce), and ScienceWorld (scientific reasoning), using various student-teacher pairs (e.g., Qwen series).

Performance Boost: TCOD improved agent performance by up to 18 points over vanilla OPD. It successfully recovered small models (e.g., Qwen3-1.7B) from near-zero success rates.
Stability: TCOD significantly mitigated KL divergence spikes and maintained smoother policy gradient losses compared to the erratic behavior of vanilla OPD.
Generalization (Surpassing the Teacher): Remarkably, TCOD-B2F demonstrated the ability to surpass the teacher's performance. In "Hard" task splits where the teacher failed under sampling, the student achieved success rates up to 14 points higher than the teacher, proving it learns robust policies rather than just mimicking errors.
Robustness: The method proved robust to the choice of the curriculum growth rate ( $\eta$ ).

5. Significance

This work is significant because it shifts the focus of LLM distillation from token-level alignment to temporal-level alignment. It provides a blueprint for training highly capable, small-scale autonomous agents that can handle long-horizon tasks. By solving the compounding error problem through a temporal curriculum, TCOD enables smaller models to achieve complex reasoning and execution capabilities that were previously thought to be the sole domain of much larger frontier models.