🤖 machine learning

Co-Evolving Policy Distillation

This paper proposes Co-Evolving Policy Distillation (CoPD), a novel training paradigm that integrates parallel expert training with bidirectional online policy distillation to overcome capability loss and behavioral gaps, achieving superior all-in-one text, image, and video reasoning performance compared to existing RLVR and distillation methods.

Original authors: Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin, Dingyu Yao, Peng Fu, Zheng Lin, Weiping Wang, Nan Duan, Jiaqi Wang

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin, Dingyu Yao, Peng Fu, Zheng Lin, Weiping Wang, Nan Duan, Jiaqi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to train a single, super-smart student to become an expert in three very different fields: Math, Art, and Video Analysis.

In the past, researchers tried two main ways to do this, but both had a major flaw. This paper introduces a new method called Co-Evolving Policy Distillation (CoPD) that fixes those flaws by having the student learn in a unique, "dance-like" partnership.

Here is how the paper explains the problem and the solution, using simple analogies:

The Problem: The "Tug-of-War" and the "Too-Far-Apart" Gap

1. The "Tug-of-War" (Mixed RLVR)
Imagine trying to teach your student Math and Art at the exact same time, in the same classroom, with the same teacher.

The Issue: Math requires strict logic and rules, while Art requires creativity and breaking rules. When you mix them, the student gets confused. The paper calls this "capability divergence." It's like a tug-of-war where the Math lesson pulls the student one way, and the Art lesson pulls them the other. The student ends up mediocre at both because the lessons cancel each other out.

2. The "Too-Far-Apart" Gap (Static OPD)
So, researchers tried a different approach: "Let's train a Math expert first, then train an Art expert separately, and then have them teach a new student."

The Issue: By the time the Math expert is finished training, they have changed so much that they are no longer speaking the same "language" as the new student. The student is a beginner; the expert is a grandmaster.
The Analogy: Imagine a grandmaster chess player trying to teach a toddler. The grandmaster makes moves that are too complex for the toddler to understand. The toddler (the student) looks at the grandmaster's (the teacher's) moves and thinks, "I have no idea what you are doing." The knowledge is lost because they are too far apart in their thinking patterns. The paper calls this a "behavioral pattern gap."

The Solution: The "Co-Evolving Dance" (CoPD)

The authors propose CoPD, which changes the training schedule entirely. Instead of training one expert first and then teaching later, they train two students simultaneously who are constantly teaching and learning from each other.

Here is how the dance works:

Step 1: The Solo Practice (RLVR Phase)

The "Math Student" practices only Math problems.
The "Art Student" practices only Art problems.
Why? This allows them to get really good at their specific skills and discover new, advanced techniques. They start to drift apart in their thinking, which is good because it means they have new things to teach each other.

Step 2: The Dance Partner Swap (Mutual OPD Phase)

Before they drift too far apart, they stop and swap roles.
The Math Student tries to solve Art problems, and the Art Student watches and gives feedback.
The Art Student tries to solve Math problems, and the Math Student watches and gives feedback.
Why? Because they just practiced together, they are still close enough in their thinking patterns to understand each other. The Math Student can say, "I see you're trying to do X, but here is a better way," and the Art Student actually understands it because they are still on the same page.

Step 3: Repeat

They go back to Step 1 to learn even more advanced tricks, then swap again in Step 2.
This cycle repeats constantly.

Why This Works Better

The paper claims this method is superior for three main reasons:

No Tug-of-War: Because they practice their specific skills separately for a while, they don't get confused by mixing Math and Art rules.
No "Too-Far-Apart" Gap: Because they swap roles during training (not after), they never drift so far apart that they can't understand each other. They stay in a "sweet spot" where the teacher is advanced enough to teach, but close enough to be understood.
Mutual Growth: They aren't just a teacher and a student; they are co-evolving. They grow together. The paper found that the final result is a single model that is actually better at both Math and Art than the individual "experts" were on their own.

The Results

The researchers tested this on a model that needed to handle text (Math), images (Art), and videos.

Old Methods: The model was either confused (mixed training) or lost knowledge (static teaching).
CoPD: The model mastered all three. In fact, the final "all-in-one" model performed better than the specialized experts it was built from.

Summary

Think of CoPD not as a school where you graduate one subject before starting the next, but as a gym where two athletes train together. They do their own drills to get stronger, then immediately spot each other to share tips while they are still warm and in sync. By the time they are done, they are both stronger than they would have been training alone or training in isolation.

1. Problem Statement

The paper addresses the challenge of consolidating multiple specialized capabilities (e.g., text reasoning, image reasoning, video understanding) into a single unified model. Current post-training paradigms face two primary limitations:

Mixed-Data RLVR (Reinforcement Learning with Verifiable Rewards): Training a single model on mixed data from different domains often leads to capability divergence. Because different capabilities favor conflicting optimization directions, joint training results in gradient conflicts, causing a "seesaw effect" where gains in one domain come at the expense of another.
Static On-Policy Distillation (OPD/MOPD): The standard alternative is a two-stage pipeline: first, train separate "expert" models on specific domains until convergence; second, distill these experts into a single student model. The authors identify a critical flaw in this approach: by the time distillation begins, the experts have drifted significantly in their behavioral patterns (token distributions) from the student. This large behavioral distance makes the teacher's supervision difficult for the student to absorb, leading to suboptimal knowledge transfer and failure to fully retain the experts' capabilities.

2. Methodology: Co-Evolving Policy Distillation (CoPD)

CoPD proposes a unified framework that interleaves capability-specific exploration with cross-domain consolidation, allowing experts to co-evolve as both teachers and students simultaneously.

Core Mechanism

Instead of training experts to convergence before distillation, CoPD maintains parallel training branches (one per capability) that alternate between two phases in every training cycle:

Branch-Specific RLVR Phase:
- Each branch independently performs Group Relative Policy Optimization (GRPO) on its specific domain data.
- Goal: Deepen domain-specific expertise and intentionally increase the "behavioral gap" (knowledge divergence) between branches. This ensures that subsequent distillation has new, informative knowledge to transfer.
Mutual OPD Phase:
- Branches generate rollouts on each other's data.
- Each branch acts as a teacher for the others, providing token-level supervision (via KL divergence minimization) on the student's own trajectories.
- Goal: Close the behavioral gap while the knowledge is still fresh and the models are behaviorally aligned. This ensures high absorption efficiency.

Key Design Features

Bidirectional Co-Evolution: Unlike static pipelines where teachers are fixed, CoPD uses a dynamic topology where every branch is simultaneously a teacher and a student.
Behavioral Consistency: The method relies on the hypothesis that distillation efficiency ( $\eta$ ) is positively correlated with the top- $k$ token overlap between teacher and student. CoPD actively maintains this overlap above a critical threshold (e.g., >0.90) by alternating phases, preventing the models from drifting too far apart.
Scalability: The framework supports $K$ branches. For $K > 2$ , it employs a "hub-and-spoke" topology to avoid full pairwise complexity (e.g., in a 3-branch setting, text reasoning acts as a hub for image and video branches).
Final Integration: Since all branches start from a shared base and remain tightly coupled, the final unified model is obtained via simple parameter merging.

3. Key Contributions

Theoretical Insight: The authors identify that the effectiveness of OPD is inversely related to the behavioral distance between teacher and student. They demonstrate via pilot studies that static pipelines operate in a "low-absorption" regime because experts drift too far before distillation begins.
CoPD Framework: They propose a novel training paradigm that interleaves RLVR and mutual OPD. This solves the trade-off between needing knowledge divergence (for informative distillation) and behavioral proximity (for absorbable distillation).
Breaking the "Expert Ceiling": CoPD demonstrates that a unified model can surpass the performance of individual domain-specific experts, effectively turning cross-domain trade-offs into mutual gains.
Scalability Validation: The method is validated in both 2-branch (Text + Image) and 3-branch (Text + Image + Video) settings, proving its efficacy in complex multi-modal scenarios.

4. Experimental Results

The paper evaluates CoPD on Qwen3-VL-4B-Instruct across text, image, and video reasoning benchmarks.

Text & Image Reasoning (2-Branch):
- CoPD achieved the highest overall average accuracy (57.71%), outperforming Mixed RLVR, Static OPD (both directions), and even the individual Text-Expert and Image-Expert models.
- Notably, CoPD improved text reasoning to 58.76% (vs. 57.89 for the Text-Expert) and image reasoning to 56.97% (vs. 55.76 for the Image-Expert), proving the "mutual gain" effect.
Text, Image, & Video Reasoning (3-Branch):
- CoPD achieved an overall average of 58.12%, surpassing the Multi-Teacher OPD (MOPD) baseline (56.99%) and the Video-Expert (58.75) in video reasoning while maintaining high performance in text and image tasks.
- Mixed RLVR showed the highest video average among baselines but suffered significant degradation in text reasoning, confirming the capability divergence problem.
Ablation Studies:
- Removing mutual distillation in either direction caused performance drops, confirming the necessity of bidirectional co-evolution.
- Even without parameter merging, individual branches in CoPD outperformed static OPD baselines, indicating that the co-evolution process itself creates robust, well-rounded capabilities.
- The optimal ratio of RLVR steps to OPD steps was found to be 1.5:1, balancing exploration and consolidation.

5. Significance

New Training Paradigm: CoPD challenges the conventional "train-then-distill" workflow, suggesting that parallel training with continuous mutual distillation is a more effective scaling strategy for multi-capability models.
Solving the "All-in-One" Problem: It provides a solution to the long-standing issue of integrating diverse modalities without sacrificing performance in any single domain.
Self-Taught RLVR Series: This paper is the third installment in the authors' "Self-Taught RLVR" series, exploring the concept of "parallel selves" (mutual teaching) complementing previous work on "informed selves" and "temporal selves."
Practical Impact: The method offers a scalable path for building general-purpose AI agents that can reason across text, images, and video simultaneously, potentially reducing the need for maintaining multiple specialized models.