Technical Summary: F-TIS – Harnessing Diverse Models in Collaborative GRPO

Problem Statement

Group Relative Policy Optimization (GRPO) has emerged as a dominant method for post-training Large Language Models (LLMs), particularly for enhancing reasoning capabilities. Unlike Proximal Policy Optimization (PPO), GRPO eliminates the need for a separate value model by computing advantages relative to a group of completions generated for a single prompt. However, the auto-regressive nature of LLM generation creates a significant computational bottleneck during the inference phase.

To address this, prior work has explored distributing the generation step across multiple nodes. Existing distributed approaches largely assume homogeneous models (identical architectures and parameters) to maintain "on-policy" conditions. This assumption is impractical for decentralized training scenarios where participants possess heterogeneous resources and preferences. In such environments, three types of heterogeneity arise:

1. Model Size: Nodes may run models of vastly different parameter counts (e.g., 1B vs. 3B).
2. Model Expertise: Nodes may possess models of similar size but different specializations (e.g., generalist vs. coder).
3. Trainable Parameters: Nodes may employ different fine-tuning strategies (e.g., full fine-tuning vs. Parameter-Efficient Fine-Tuning like LoRA).

When heterogeneous models collaborate without adjustment, the training data becomes highly off-policy. The divergence between the generating policy ($\pi_{gen}$) and the updating policy ($\pi_{\theta}$) introduces destructive noise, leading to policy collapse or significantly degraded performance compared to training in isolation.

Methodology: Filtered Truncated Importance Sampling (F-TIS)

The authors propose F-TIS, a unified framework designed to enable stable collaborative GRPO training across heterogeneous models while minimizing communication overhead. The methodology combines two key mechanisms: Truncated Importance Sampling (TIS) and Sample Filtering.

1. Truncated Importance Sampling (TIS)

Standard GRPO assumes on-policy data. To handle off-policy samples, the authors adapt the loss function using importance sampling. They compare three approaches:

• NoIS (No Importance Sampling): Treats all samples as on-policy (baseline for failure).
• VIS (Vanilla Importance Sampling): Applies the importance ratio $\frac{\pi_{\theta}}{\pi_{gen}}$ to every token.
• TIS (Truncated Importance Sampling): Truncates the importance ratio at a constant $C$ (e.g., $C=2$) to prevent extreme weights from destabilizing training. The ratio is moved outside the inner summation over tokens.

The TIS loss formulation is:
$$\mathcal{L}_{GRPO} = \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|a_i|} \sum_{t=1}^{|a_i|} \min \left( \frac{\pi_{\theta}(a_{i,t}|p \circ a_{i,<t})}{\pi_{gen}(a_{i,t}|p \circ a_{i,<t})}, C \right) \cdot \min \left( R_{i,\theta} \hat{A}_i, \text{clip}(R_{i,\theta} \hat{A}_i, 1-\epsilon, 1+\epsilon) \right)$$
Where $R_{i,\theta}$ is the detached ratio and $\hat{A}_i$ is the group advantage.

2. Sample Filtering

The authors observe that off-policy samples with negative advantages ($\hat{A}_i < 0$) can amplify unlikely tokens, leading to gibberish completions. They introduce a filtering mechanism based on:

• Advantage: Samples with $\hat{A}_i < 0$ are discarded for the update step.
• KL-Divergence: Samples where the KL-divergence between the current policy and the generating policy exceeds a threshold $g$ ($D_{KL}(\pi_{\theta} \parallel \pi_{gen}) > g$) are also filtered.

Filtered samples are still used to compute the group advantage (to maintain the relative ranking of the group) but are excluded from the gradient update.

3. The F-TIS Framework

F-TIS integrates TIS and filtering. The advantage term $\hat{A}_{t,i}$ in the loss function is set to 0 if the sample is filtered (i.e., if $\hat{A}_i \leq 0$ OR $D_{KL} > g$).

• Communication Efficiency: The framework requires minimal communication, transmitting only tokens and log-probabilities (approx. 8 bytes per token), making it suitable for decentralized networks.

Key Contributions

1. Identification of Heterogeneity Challenges: The paper empirically demonstrates that standard GRPO fails in decentralized heterogeneous settings (NoIS), leading to performance degradation compared to isolated training.
2. F-TIS Algorithm: A novel training paradigm that successfully leverages off-policy samples from diverse models (different sizes, expertise, and PEFT configurations) without collapsing the policy.
3. Communication Efficiency: The approach maintains low communication overhead, requiring only log-probabilities and tokens, unlike methods requiring full model synchronization or heavy value-model exchanges.
4. Ablation Studies: The authors systematically analyze the impact of the filtering threshold $g$ and compare F-TIS against F-VIS (Filtered Vanilla Importance Sampling), demonstrating the superiority of truncation in later training stages.

Experimental Results

The authors evaluated F-TIS on the GSM8k dataset for training and the MATH-500 dataset for out-of-distribution (OOD) testing, using Qwen2.5 models (1.5B and 3B, Base and Coder variants).

• Convergence: F-TIS achieves final model convergence identical to purely on-policy (homogeneous) training across all heterogeneous setups.
• Generalization (OOD): In several configurations, F-TIS outperformed isolated on-policy training on OOD tasks. Notably:
  • The 3B Coder model paired with a Base model improved its MATH-500 performance by 5.2%.
  • The 3B Coder model paired with a smaller Coder model improved by 12%.
  • The 3B PEFT model improved by 7% when trained collaboratively.
• Filtering Threshold ($g$): Ablation studies revealed that smaller $g$ values benefit smaller models (1.5B) initially by removing random noise, while larger models (3B) benefit from higher $g$ (e.g., 50) later in training, allowing for greater exploration of the space.
• Vertical vs. Horizontal: The paper finds that vertical collaboration (one node generates the full group for a prompt) is superior to horizontal collaboration (nodes generate subsets). Horizontal collaboration introduces bias in advantage calculation relative to the swarm mean, degrading performance.

Significance and Claims

The paper positions F-TIS as a foundational step toward decentralized collaborative RL for LLMs. The authors claim that their work is among the first to successfully study and enable collaborative RL across heterogeneous models.

Key claims include:

• Feasibility: It is possible to train diverse models together in a single GRPO run without the performance collapse typically associated with off-policy data.
• Efficiency: The method is communication-efficient, making it viable for decentralized systems with varying network constraints.
• Enhanced Reasoning: Beyond matching on-policy performance, the collaborative use of off-policy samples can act as a regularizer or exploration booster, leading to better generalization on out-of-distribution tasks (up to 12% improvement) compared to training models in isolation.
• Practicality: The framework accommodates real-world constraints where users have different hardware capabilities (model sizes) and specialized needs (expert models or PEFT), allowing them to contribute to a shared learning objective without requiring identical infrastructure.