🤖 machine learning

Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

The paper proposes Near-Policy Distillation (NPD), an asynchronous framework that accelerates on-policy knowledge distillation by decoupling generation from training and employing $\Delta$ -IFD filtering to mitigate policy delays, thereby achieving an 8.1-fold speedup and superior performance compared to standard SFT and larger models.

Original authors: Miao Rang, Zhenni Bi, Hang Zhou, Kai Han, Xuechun Wang, An Xiao, Xinghao Chen, Yunhe Wang, Hanting Chen

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Miao Rang, Zhenni Bi, Hang Zhou, Kai Han, Xuechun Wang, An Xiao, Xinghao Chen, Yunhe Wang, Hanting Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Teacher-Student" Discrepancy

Imagine you are teaching a student (a small AI model) to write essays by having them copy a master teacher (a massive AI model).

The Old Way (Standard Distillation): You give the student a list of essays the teacher has already written. The student memorizes them. However, when the student takes an exam, they must write from scratch. Since they have only memorized the teacher's perfect answers, they become confused when they must generate their own sentences. It is like a student who can recite a script but freezes when asked to improvise.
The "On-Policy" Way (The Expensive Solution): To fix this, you have the student first write their own essays and then show them to the teacher for correction. This works brilliantly because the student learns from their own mistakes. However, it is incredibly slow. Every time the student writes a sentence, the teacher must stop, read it, evaluate it, and provide feedback before the student is allowed to write the next sentence. It is like a tutor who refuses to let the student write a single word until they have evaluated the previous sentence.

The Solution: Near-Policy Distillation (NPD)

The authors propose a new method called Near-Policy Distillation. Imagine this as a high-speed assembly line that retains the benefits of the "On-Policy" method without the slowness.

Here is how it works, broken down into three parts:

1. The Asynchronous Assembly Line (Decoupling)

Instead of the teacher and student working in a slow, one-to-one conversation, NPD separates them.

The Student's Task: The student model runs quickly on a computer and generates thousands of essays (responses) simultaneously, like a factory producing goods. It does not wait for the teacher.
The Teacher's Task: Once the student has a large stack of essays, the teacher model reviews them all at once. Since the teacher is not waiting for the student to type, they can process them in "batches" (like reading an entire book chapter at once, rather than page by page).
The Result: This is 8.1 times faster than the old slow method because the computer is not idly waiting for feedback.

2. The "Safety Filter" (The ∆-IFD Mechanism)

There is a catch. Since the student generates essays before the teacher evaluates them, the student could become a little "drunk" on their own ideas. They might start writing nonsense or things that are completely wrong because their "policy" (their way of thinking) has drifted away from the teacher's.

To fix this, the paper introduces an intelligent filter called ∆-IFD.

The Analogy: Imagine the student is a novice cook and the teacher is a Michelin-star chef. The student cooks a series of dishes.
- Zone 1 (Degenerate): The student cooks something so simple and strange (like a bowl of plain water) that even the teacher thinks it is too simple, but the student is confused. The filter discards this.
- Zone 2 (Cognitive Decoupling): The student cooks a dish where they are very confident, but the Michelin chef considers it garbage. This is dangerous because the student is stubbornly wrong. The filter discards this.
- Zone 3 (The Proximal Learning Zone): The student cooks something just above their skill level, and the teacher agrees that it is good and helpful. This is the only zone from which the student learns.

This filter ensures the student learns only from "Goldilocks" examples—not too easy, not too hard, and not dangerously wrong. This keeps training stable even though the student and teacher are not communicating in real time.

3. The "Sparse Update" (The Occasional Reset)

Normally, with these fast assembly lines, the student's ideas could drift too far from the teacher's style over time.

The Solution: Instead of stopping the entire factory to synchronize every single second (which is slow), NPD stops the line only occasionally to reset the student's brain to the teacher's current style.
The Result: The paper found that such a "reset" is needed only once or twice during the entire training process to keep everything on track, saving enormous amounts of time.

The Grand Finale: A Super-Student

The paper shows that this method not only trains the student faster but also makes them smarter.

The Result: They took a small 1-billion-parameter model (a "tiny" AI) and trained it with this method.
The Comparison: This tiny model achieved 68.73% on a difficult reasoning test after training with NPD and some additional Reinforcement Learning.
The Shock: It beat a much larger, famous model (Qwen3-1.7B) with 1.7 billion parameters, which achieved 63.69%.

Summary

Near-Policy Distillation is like setting up a high-speed factory where a student learns from a teacher's feedback without having to wait in line. It uses an intelligent filter to discard bad examples and occasional resets to keep the student on the right path. The result is a tiny AI that learns faster, cheaper, and ends up smarter than much larger models.

Technical Summary: Near-Policy Distillation (NPD)

1. Problem Statement

Standard Knowledge Distillation (KD) for autoregressive Large Language Models (LLMs) faces a fundamental distributional imbalance: student models are trained on teacher-forced sequences but generate autoregressive sequences during inference. While On-Policy methods (e.g., GKD, DistiLLM) mitigate this by training on student-generated outputs (SGOs), they rely on computationally intensive, reinforcement-learning-like (RL) frameworks. These frameworks require synchronous inference between teacher and student, which prevents the use of Sequence Packing, leading to low token utilization rates and slow training throughput. Conversely, standard Supervised Fine-Tuning (SFT) supports Sequence Packing but fails to correct the distributional shift at inference time, resulting in exposure bias and error accumulation.

2. Methodology: Near-Policy Distillation (NPD)

The authors propose Near-Policy Distillation (NPD), an asynchronous framework that decouples student generation from the training loop. This reformulation allows the system to approximate the distribution-matching benefits of on-policy distillation while preserving the high throughput of off-policy SFT.

Core Architecture

The NPD pipeline consists of three asynchronous phases:

Student-Driven Generation: The current student model generates responses for a prompt dataset. These sequences are generated in high-throughput batches (e.g., using vLLM).
Teacher Annotation via Packed Parallel Prefill: Unlike on-policy methods requiring synchronized autoregressive decoding, NPD concatenates multiple student responses into fixed sequences. The teacher model processes these via a parallel Prefill step to generate logits. Only the top- $k$ logits are recorded to balance memory requirements and supervision quality.
Combined Optimization: The student model is updated using a standard SFT architecture with a combined loss function:
$L_{total} = (1 - \lambda) \cdot L_{CE} + \lambda \cdot L_{KD}$
where $L_{KD}$ is a top- $k$ KL divergence calculated only over the teacher's high-confidence predictions.

Addressing Policy Lag: $\Delta$ -IFD and Sparse Updates

The asynchronous nature of NPD leads to Policy Lag, where the data generation strategy deviates from the current training strategy, potentially introducing noise. NPD counteracts this with two mechanisms:

$\Delta$ -IFD Filtering: A heuristic mechanism for sample selection based on the Instruction-Following Difficulty (IFD) metric. It calculates the differential difficulty: $\Delta IFD = IFD_{Teacher} - IFD_{Student}$ $Δ I F D = I F D_{T e a c h er} - I F D_{S t u d e n t}$ .
- Degenerate Zone ( $\Delta IFD < 0$ ): Samples where the teacher is more confident than the student (often trivial or meaningless patterns) are discarded.
- Cognitive Disconnect Zone ( $\Delta IFD > \tau$ ): Samples where the student is confident but the teacher is highly uncertain (indicating severe errors) are discarded.
- Proximal Learning Zone ( $0 \le \Delta IFD \le \tau$ ): Only samples where the teacher confirms baseline quality and the difficulty gap is moderate are retained. This ensures updates remain within a "safe proximal learning zone."
Sparse Student Updates: To prevent the generation strategy from deviating significantly, the student model is updated periodically (e.g., once every 10 epochs) rather than continuously. This minimizes the computational cost of policy synchronization while maintaining the "near-policy" state.

Synergy with Reinforcement Learning (RL)

NPD serves as a superior initialization for subsequent RL (e.g., GRPO). By training exclusively on high-quality, filtered data, the student's probability mass concentrates on a manifold of logically correct trajectories. This effectively narrows the exploration space for the subsequent RL phase, allowing it to bypass inefficient stochastic exploration and focus on fine-grained optimization.

3. Key Contributions

NPD Framework: An asynchronous distillation framework that decouples generation from training, enables Sequence Packing, and achieves an 8.1x speedup compared to on-policy baselines while maintaining their performance advantages.
Stabilization Mechanisms: The integration of sparse updates and the $\Delta$ -IFD filtering mechanism effectively mitigates policy lag and sample noise. This prevents noise from dominating gradients and empirically stabilizes the optimization trajectory, keeping KL divergence below 0.10.
RL Initialization: The method demonstrates that NPD-trained models provide a significantly better starting point for RL, narrowing the search space and enabling higher asymptotic performance.

4. Experimental Results

Experiments were conducted using openPangu-Embedded-7B as the teacher and openPangu-Embedded-1B as the student, supplemented by comparisons with Qwen2.5-Math models.

Distillation Performance: NPD outperformed standard SFT by 8.09% and the standard SeqKD approach by 4.83% on a large-scale industrial dataset (1.2 million samples).
Efficiency: Compared to the state-of-the-art on-policy method DistiLLM-2, NPD achieved superior accuracy on the GSM8K and MATH-500 benchmarks with an 8.1x reduction in training time (1.48 NPU-hours vs. 12.00 NPU-hours per epoch).
RL Synergy: As an initialization for GRPO, the NPD model achieved an average endpoint score of 68.73% across 11 benchmarks.
Comparison with State-of-the-Art: The final openPangu-Embedded-1B-RL model (1 billion parameters) outperformed the significantly larger Qwen3-1.7B (63.69%) with a score of 68.73%. It also achieved leading scores on GSM8K (87.60%) and MATH-500 (84.76%).

5. Significance and Claims

The work claims that NPD offers a scalable path toward ubiquitous intelligence on edge devices by aligning high-performance AI with edge constraints. By effectively narrowing the exploration space for subsequent RL and enabling efficient Sequence Packing, the framework demonstrates that advanced training and alignment methods can be more effective than mere model size scaling. The authors emphasize that their approach reduces the costs of training and deploying better-aligned smaller models, making powerful LLMs more accessible for real-world scenarios requiring low latency and data privacy.

The work concludes that while NPD is constrained by the teacher's distribution, its ability to filter systemic errors via $\Delta$ -IFD and stabilize training through sparse updates makes it a robust solution for efficient knowledge transfer in autoregressive models.