Technical Summary: Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models

1. Problem Statement

Current Reinforcement Learning (RL) post-training for Large Language Models (LLMs) operates primarily as a single-policy endeavor. In this siloed workflow, exploration and improvement are coupled to a single policy, and complementary experiences discovered by concurrently trained peers are discarded. This isolation is particularly inefficient in sparse-reward reasoning domains where heterogeneous policies (varying in architecture, scale, or tokenizer) exhibit diverse inductive biases and traverse distinct regions of the solution space.

Two primary barriers prevent the sharing of these discoveries:

1. Tokenizer Heterogeneity: Policies often use disparate tokenizers, making direct exchange of token-level traces (log-probabilities) impossible without translation.
2. Distribution Shift: Directly reusing peer rollouts within on-policy surrogates (like PPO or GRPO) introduces significant distribution shift, leading to instability and variance issues common in off-policy learning.

The paper asks: Within the standard regime of single-step verifiable post-training for open-weights LLMs, which experience-sharing regime is appropriate, and what trade-offs do alternative regimes induce?

2. Methodology

2.1 The Mutual RL Framework

The authors introduce a framework for Mutual Reinforcement Learning where heterogeneous LLM policies exchange typed experience while maintaining separate parameters, objectives, and tokenizers. The system extends the VERL framework with three core components:

• Shared Experience Exchange (SEE): A typed dataflow mechanism where distinct policies publish and subscribe to structured experience records (rollouts, rewards, advantages, and traces) while preserving native per-policy objectives.
• Multi-Worker Resource Allocation (MWRA): Manages the assignment of workers and compute budgets across different policy workers.
• Tokenizer Heterogeneity Layer (THL): A critical interface that bridges incompatible vocabularies. It performs two operations:
  1. Text Adaptation (Retokenization): Decodes peer responses into text and re-encodes them using the learner's tokenizer.
  2. Policy-Distribution Alignment: Constructs word-level trace alignments. It aggregates source log-probabilities within each textual word and redistributes this mass evenly across the target tokens for that word. This preserves word-level probability mass despite tokenization mismatches, with bounded residual errors analyzed theoretically.

2.2 Three Controlled Probes

To investigate the stability-support trade-off, the authors instantiate three distinct experience-sharing regimes on top of the same Group-Relative Policy Optimization (GRPO) base update:

1. Regime 1: Data-Level Sharing (Peer Rollout Pooling - PRP)

   • Mechanism: The learner pools peer rollouts directly into its own batch, treating them as supplementary training trajectories.
   • Challenge: Requires handling the density-ratio variance between the learner's policy and the peer's behavior policy. The authors test two denominator choices: a "learner-snapshot" denominator (ignoring the peer's actual generation probabilities) and a "THL-aligned peer denominator" (using the peer's recorded likelihood trace aligned to the learner's token space).
   • Theoretical Position: High coupling; pays density-ratio variance costs and THL residual costs.

2. Regime 2: Value-Level Sharing (Cross-Policy GRPO Advantage Sharing - XGRPO)

   • Mechanism: The learner samples its own rollouts (strictly on-policy) but shapes the advantages using reward statistics aggregated across the entire policy pool.
   • Challenge: None regarding tokenization, as only scalar rewards are exchanged.
   • Theoretical Position: Preserves on-policy actor support while changing scalar baselines. It reduces gradient variance without introducing off-policy trajectory signals.

3. Regime 3: Outcome-Level Sharing (Success-Gated Transfer - SGT)

   • Mechanism: Treats peer outputs as sparse outcome certificates. The transfer is gated: it activates only when the learner's own rollout group contains no correct solution, but at least one peer has a verified success.
   • Process: A verified peer trajectory is injected as a sparse positive sample (via a negative log-likelihood loss) alongside the learner's on-policy GRPO update.
   • Theoretical Position: Supplies a "rescue-set" score direction. It is peer-conditioned, failure-triggered, and interleaved with the base RL objective.

3. Key Contributions

1. Framework Substrate: The introduction of a reproducible framework (SEE, MWRA, THL) enabling mutual RL across heterogeneous LLMs with incompatible tokenizers. The THL provides a bounded-alignment interface for text and token-level trace exchange.
2. Controlled Probes: The instantiation and comparison of PRP, XGRPO, and SGT under matched compute and hyperparameters within a single VERL-based stack. This populates the design space at three distinct coupling levels (data, value, outcome).
3. Theoretical Characterization: A contextual-bandit analysis proving the structural differences between the probes:
   • PRP: Exposes sensitivity to density-ratio variance and THL residuals.
   • XGRPO: Preserves on-policy support but cannot directly introduce probability mass for peer-only successes (hard-support boundary).
   • SGT: Provides a positive-gradient direction toward verified peer successes specifically on the rescue subset.

4. Experimental Results

The evaluation focuses on mathematical reasoning (MATH dataset) using a heterogeneous pool of Qwen models (Qwen2.5-Math-1.5B and Qwen3-1.7B-Base).

• Outcome-Level Sharing (SGT) is Optimal: SGT occupies the favorable point of the stability-support trade-off. It achieves the strongest average performance and faster early-to-mid training progress compared to standalone GRPO. It stays at or above the baseline curve at virtually every checkpoint.
  • Why: It supplies a missing positive trajectory only when the learner fails, preserving the on-policy update for other prompts.
• Value-Level Sharing (XGRPO) is Stable: XGRPO tracks closely with standalone GRPO, preserving the trust region and reducing baseline variance. It serves as a stable, low-coupling baseline but does not significantly expand the learner's sampled support.
• Data-Level Sharing (PRP) is Unstable: Direct rollout reuse exposes the divergence sensitivity of the learner. While importance correction (via THL) mitigates some issues, PRP imposes a larger optimization burden on weaker policies and reduces average accuracy compared to SGT. It requires significant distribution control to remain stable.
• Optimization Signatures:
  • SGT maintains entropy comparable to or higher than GRPO while keeping KL low.
  • PRP drives entropy lower and increases KL deviations, indicating strong distribution coupling.
  • XGRPO remains nearly indistinguishable from the single-policy baseline.
• Generalization: SGT's benefits extend to heterogeneous pools (e.g., Llama-Mist, Qwen-Phi4) and broader commonsense/scientific QA benchmarks, showing gains of +10 to +30 percentage points in some heterogeneous configurations.

5. Significance and Claims

The paper claims that Mutual RL transforms peer discoveries into typed training signals for heterogeneous LLM policies without merging weights or replacing standard on-policy pipelines.

• Structural Insight: The work establishes that in the evaluated regime, outcome-level sharing (SGT) is the superior abstraction. It balances stability (by remaining largely on-policy) with support (by injecting verified trajectories only when necessary).
• Feasibility of Heterogeneity: The authors demonstrate that with the THL substrate, policies with different tokenizers can effectively exchange signals, making the question of "which experience-sharing regime is right" well-posed and answerable.
• Efficiency: Unlike complex off-policy algorithms that replace standard pipelines, this approach integrates peer experiences into the standard on-policy surrogates (GRPO/PPO) used by contemporary LLM RL, offering a principled channel for policies to learn from one another during training.

The authors conclude that while data-level sharing (PRP) is theoretically possible with correction, it is fragile; value-level sharing (XGRPO) is stable but limited in support expansion; and outcome-level sharing (SGT) offers the most favorable trade-off by providing targeted, verified support exactly where the learner is deficient.