💬 NLP

Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs

This paper demonstrates that reinforcement learning (RL) significantly outperforms supervised fine-tuning in model merging by mitigating task conflicts through smaller gradient updates, a convergent optimization objective that minimizes conflicting parameter changes, and the joint optimization of positive and negative examples to ensure robust, unbiased performance.

Original authors: Zixuan Ren, Jinliang Lu, Junhong Wu, Yang Zhao, Dai Dai, Hua Wu, Haifeng Wang, Chengqing Zong

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zixuan Ren, Jinliang Lu, Junhong Wu, Yang Zhao, Dai Dai, Hua Wu, Haifeng Wang, Chengqing Zong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a giant library where every book is a super-smart robot brain, known as a Large Language Model (LLM). These brains are incredible, but they are often trained to be experts at just one thing: one loves math, another is a coding wizard, and a third is a master of telling jokes. Sometimes, we want to combine these brains into one "super-brain" that can do everything at once. This process is called model merging. Think of it like trying to mix two different smoothies—one strawberry and one spinach—into a single cup. Usually, when you mix them, the flavors clash, and you end up with a gross, unrecognizable sludge where neither the strawberry nor the spinach tastes right. In the world of AI, this "sludge" is called a task conflict, where the knowledge from one task messes up the other.

For a long time, scientists have been trying to figure out how to mix these brains without ruining the flavor. They usually take two models that have been taught using a method called Supervised Fine-Tuning (SFT). You can think of SFT as a strict teacher who gives a student a textbook with the right answers and says, "Memorize this exactly." But recently, a new method called Reinforcement Learning (RL) has become popular. RL is more like a video game: the AI tries things, gets points for good moves, and learns by trial and error to maximize its score. The big question is: if we train our expert brains using this "video game" style (RL) instead of the "strict textbook" style (SFT), will they mix together better?

This paper, titled "Enough Is as Good as a Feast," dives deep into that question. The researchers took models trained with both methods and tried to merge them in various ways, testing them on five different skills: math, coding, following instructions, solving logic puzzles, and ranking items. They found something surprising: RL-trained models are much better at merging. While the SFT models turned into that "gross sludge" when mixed, losing up to 65% of their ability on certain tasks, the RL models stayed surprisingly sharp, losing very little performance.

Why does this happen? The authors suggest three main reasons, using some clever analogies of their own. First, RL uses on-policy data, meaning the AI learns from its own recent attempts rather than a fixed textbook. This keeps the "learning steps" small and gentle, so the AI doesn't accidentally overwrite the knowledge it already has from other tasks. Second, RL has a natural "stop button." As the AI gets really good at a task, the updates it makes to its brain get smaller and smaller. The paper calls this the idea that "enough is as good as a feast." Once the AI has learned enough to do the job well, it stops making huge, disruptive changes. In contrast, SFT keeps making big changes even when the model is already perfect, which causes a mess when you try to mix it with another model. Finally, RL learns from both good examples and bad examples (positive and negative samples). This helps the AI understand not just what to do, but what not to do, creating a cleaner, more balanced set of instructions that doesn't clash with other tasks.

In short, the paper suggests that if you want to build a versatile, multi-talented AI by mixing different experts together, training them with Reinforcement Learning is like using a gentle, precise hand to blend the smoothie, whereas the traditional method is like throwing the ingredients into a blender on high speed. The result? A much tastier, more capable super-brain.

Technical Summary: Enough is as Good as a Feast

Problem Statement

Model merging has emerged as a critical technique for consolidating multiple specialized Large Language Models (LLMs) into a single unified model without requiring access to original training data or expensive retraining. However, a central challenge in this process is task conflict: when parameters from independently fine-tuned models are fused, interference between them often leads to significant performance degradation.

While recent research has focused on developing better merging strategies (e.g., Task Arithmetic, TIES-Merging, DARE), the impact of the training paradigm used to create the source models has been largely overlooked. Most existing works assume models are trained via Supervised Fine-Tuning (SFT). This paper investigates whether the choice between SFT and Reinforcement Learning (RL) as a post-training paradigm influences the susceptibility of models to task conflicts during merging.

Methodology

The authors conducted a comprehensive empirical and theoretical analysis comparing models fine-tuned with SFT against those fine-tuned with RL (specifically PPO, GRPO, and REINFORCE++).

Experimental Setup

Base Models: Llama-3.2-3B, Llama-3.1-8B, and Mistral-Small-3-24B.
Tasks: Five representative tasks allowing automatic verification: Mathematical Reasoning, Code Generation, Instruction Following, Logical Puzzles, and Ranking.
Merging Strategies: The study evaluated four merging methods: Model Averaging, TIES-Merging, Task-Arithmetic, and DARE+TIES.
Protocol: Pairwise merging was performed between models fine-tuned on different tasks to measure performance preservation.

Theoretical and Empirical Analysis

To explain the observed differences, the authors analyzed three specific mechanisms distinguishing RL from SFT:

On-Policy Data: Comparing the gradient norms of updates derived from on-policy data (RL) versus fixed offline datasets (SFT).
Intrinsic Optimization Dynamics: Theoretical analysis of how RL objectives (specifically the advantage function) behave as training converges compared to the fixed-intensity updates of SFT.
Positive vs. Negative Samples: A controlled experiment isolating the effect of negative samples by creating an "RL-Pos" variant where negative sample advantages are zeroed out.

Key Results

1. Superior Performance Preservation in RL Models

Across all five tasks, four merging strategies, and three base model architectures, RL-trained models consistently exhibited significantly less performance degradation after merging compared to SFT-trained models.

Magnitude of Improvement: In the "Puzzle" task, SFT models suffered up to a 65% performance drop with certain merging methods, whereas RL models limited losses to under 10% in most cases.
Generality: The advantage of RL held regardless of the specific RL algorithm used (PPO, GRPO, REINFORCE++) or the base model size.

2. Mechanisms Mitigating Task Conflicts

The paper identifies three primary factors contributing to RL's robustness:

Smaller Gradient Magnitudes via On-Policy Data: RL relies on data sampled from the current model, which aligns closely with the model's distribution. This results in significantly smaller parameter update norms compared to SFT (e.g., in Math tasks, SFT update norms were ~6.5 vs. ~0.78 for RL). Smaller updates reduce the risk of overwriting knowledge acquired from other tasks.
Adaptive Convergence ("Enough is as Good as a Feast"): Theoretical analysis (Theorem 1 & 2) demonstrates that as RL training converges, the expected advantage ( $A$ ) approaches zero. Consequently, the magnitude of parameter updates naturally diminishes. In contrast, SFT applies updates with fixed intensity regardless of convergence. This intrinsic property of RL progressively reduces the number and magnitude of conflicting parameter updates.
Joint Optimization of Positive and Negative Samples: RL optimizes over both positive (high reward) and negative (low reward) examples. Experiments comparing full RL against "RL-Pos" (positive samples only) showed that the inclusion of negative samples leads to more unbiased updates, higher convergence stability, and further reduction in cross-task conflicts.

3. Performance Landscape Visualization

Visualizing the loss landscape around merged parameters revealed that while both SFT and RL models are robust to random noise, SFT models suffer severe degradation when perturbed in the direction of task-induced updates (merging with another task). RL models, however, maintain stable performance in these directions, indicating that their learned parameters are more task-orthogonal and less entangled.

Key Contributions

Paradigm Discovery: The paper establishes that RL-trained models are inherently more suitable for model merging than SFT-trained models, a finding that holds across diverse merging methods, algorithms, and base models.
Mechanistic Explanation: Through empirical and theoretical analysis, the authors attribute this superiority to three factors: the stabilizing effect of on-policy data, the self-attenuating nature of RL updates during convergence, and the conflict-reducing effect of joint optimization over positive and negative samples.
Strategic Insight: The work suggests that RL is not merely an alternative to SFT for alignment but constitutes a fundamentally more robust paradigm for multi-task post-training in foundation models, offering a scalable path to building generalist models without retraining from scratch.

Significance

The paper argues that the choice of post-training paradigm is a critical, yet previously underappreciated, factor in the success of model merging. By demonstrating that RL naturally mitigates task conflicts, the authors provide a new perspective on how to build robust, generalist LLMs. This finding challenges the prevailing focus on merging strategies alone and highlights the importance of the training dynamics that precede merging. The results suggest that for applications requiring the integration of multiple specialized capabilities, utilizing RL-trained models as the source for merging offers a superior foundation for maintaining performance across diverse tasks.