💬 NLP

Multi-task Code LLMs: Data Mix or Model Merge?

This paper compares data mixing and model merging strategies for creating efficient multi-task code LLMs, finding that model merging outperforms data mixing at larger scales (7B) by retaining or even surpassing specialized performance, while data mixing is preferred at smaller scales (2B).

Original authors: Mingzhi Zhu, Boris Sobolev, Rahul Krishna, Raju Pavuluri, Stacy Patterson, Michele Merler

Published 2026-01-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingzhi Zhu, Boris Sobolev, Rahul Krishna, Raju Pavuluri, Stacy Patterson, Michele Merler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but small robot assistant that needs to learn two very different skills: writing code (like a programmer) and explaining code (like a teacher). You want this robot to be good at both, but you don't have the budget or time to train two separate robots. You have two main ways to teach it:

The "Mix-and-Match" Class (Data Mixture): You throw all the programming problems and all the explanation lessons into one big bucket and teach the robot everything at once.
The "Specialist Swap" (Model Merging): You first train one robot to be a master programmer and another to be a master teacher. Then, you take their brains (their "weights") and carefully blend them together into one super-robot.

This paper asks a simple question: Which method works better? And the answer depends entirely on how big the robot is.

The Big Discovery: Size Matters

The researchers tested this on robots of different sizes (small ones with about 2 billion "brain cells" and larger ones with 7 billion). Here is what they found:

1. The Small Robots (2 Billion Parameters): The "Mix-and-Match" Wins

For the smaller robots, trying to blend two expert brains together was a disaster. It was like trying to mix oil and water; the two skills fought each other, and the robot got confused, forgetting how to do either job well.

The Analogy: Imagine a small student trying to learn calculus and poetry simultaneously in a single class. If you try to force them to be a "math-poet" by just blending two separate lessons, they might get overwhelmed and learn neither well.
The Solution: For small robots, it's better to just put all the math and poetry books in one pile and study them together. The "Mix-and-Match" approach (Data Mixture) kept the small robot competent at both tasks without it getting confused.

2. The Big Robots (7 Billion Parameters): The "Specialist Swap" Wins

For the larger robots, the story flipped completely. Blending the brains of two experts worked beautifully. In fact, the merged robot was sometimes better than the individual experts.

The Analogy: Imagine a genius professor who is already a master of math and a master of literature. If you take their "math brain" and "literature brain" and combine them, they don't get confused. Instead, they find a way to use their vast knowledge to solve problems in a way neither specialist could do alone.
The Result: The large merged robot retained 96% of the performance of the specialized experts. In some cases, the merged robot even scored higher on coding tests than the robot trained specifically for coding!

Why Does This Happen? (The "Brain Scan")

The researchers didn't just look at the test scores; they looked inside the robots' brains to see how they changed during training.

Small Robots: When small robots tried to learn two things at once, their brains changed in very similar ways for both tasks. It was like they were using the exact same neurons for math and poetry. When you tried to merge two small robots, those neurons were fighting over who got to do what, causing a "traffic jam" in the brain.
Big Robots: Large robots have enough "brain space" to use different parts of their brain for math and different parts for poetry. They are like having two separate rooms in a house. When you merge them, the rooms don't collide; they just sit side-by-side, allowing the robot to be a master of both without interference.

The Takeaway

If you are building a small, efficient AI system (perhaps for a device with limited battery or memory), don't try to merge experts. Just train one model on a mix of all the data you need.

However, if you have a larger, more powerful model, go ahead and merge experts. You can train a coding specialist and a summarizing specialist separately, then combine them to get a versatile, high-performing model that saves you the cost of training a massive multi-task model from scratch.

In short:

Small Model? Mix the data.
Big Model? Merge the experts.

The paper provides a clear rulebook for developers to choose the right strategy based on the size of their model, ensuring they get the best performance without wasting resources.

Technical Summary: Multi-task Code LLMs: Data Mix or Model Merge?

Problem Statement

The rise of agentic frameworks has created a demand for smaller, specialized code Large Language Models (LLMs) that can operate alongside frontier models to balance performance, cost, and inference constraints. However, practical deployment often requires these small models to handle multiple coding tasks simultaneously (e.g., code generation and code summarization). Two primary strategies exist for creating such multi-task models:

Data Mixture: Fine-tuning a base model once on a combined dataset of all desired tasks.
Model Merging: Training separate "specialist" models for each task and merging their weights post-hoc.

While both data mixing and model merging are active research areas, their comparative effectiveness specifically for code LLMs remains understudied. It is unclear which strategy yields better multi-task performance, how model scale influences this choice, and whether weight-space analysis can predict the optimal strategy.

Methodology

The authors conducted extensive experiments comparing data-mixture Supervised Fine-Tuning (SFT) against four state-of-the-art model merging techniques across two model families (Qwen2.5-Coder and DeepSeekCoder) at two scales (approx. 1.5B/1.3B and 7B parameters).

Tasks: The study focused on two distinct coding tasks: Code Generation (using KodCode, HumanEval, MBPP) and Code Summarization (using CodeXGLUE Code-to-Text).
Strategies:
- Data Mixture: SFT on a shuffled combination of generation and summarization datasets.
- Model Merging: Training separate specialists for generation and summarization, then merging them using Linear, TIES (Trim, Elect Sign & Merge), DARE (Drop And Rescale), and DELLA (Deep Linear Learning Alignment).
Analysis: The authors introduced a weight-space diagnostic approach, measuring layer-wise $L_2$ distance from the base model and computing Pearson correlations between task-specific weight updates to understand interference and representational overlap.

Key Contributions

Scale-Swept Comparison: The paper provides a comprehensive empirical comparison of data mixing versus model merging for code LLMs at 2B and 7B scales.
Weight-Space Diagnostics: The authors propose using layer-wise weight shifts and inter-task update correlations as a mechanistic diagnostic. They found that smaller models exhibit high correlation between task updates (indicating interference), while larger models show low correlation (indicating orthogonal representations).
Practical Guidelines: The study establishes a clear rule of thumb based on model scale:
- Small Models (~2B): Data-mixture SFT is the preferred strategy, as merging specialists leads to significant performance degradation due to capacity constraints.
- Large Models (7B): Model merging is superior, often matching or surpassing individual specialists with minimal average degradation (<2.1%).

Results

The evaluation on HumanEval, MBPP, and CodeXGlue benchmarks revealed a distinct scale-dependent divergence in strategy effectiveness:

Small Models (1.3B/1.5B):
- Data-mixture SFT consistently outperformed merging strategies. For example, on Qwen 1.5B, data mixing achieved only a -1.26% average degradation compared to specialists, whereas merging methods (Linear, TIES, DELLA) suffered losses ranging from -7% to -27%.
- Weight analysis showed high Pearson correlation between generation and summarization updates in small models, suggesting they modify similar parameters, leading to destructive interference when merged.
Large Models (7B):
- Model merging significantly outperformed data mixing. On Qwen 7B, the DELLA merge achieved the best overall performance (-2.08% average loss), slightly outperforming data-mixture SFT (-2.57%).
- On DeepSeek 7B, DARE merging achieved near-perfect performance retention (-0.01% average loss), even surpassing specialists on specific metrics (e.g., MBPP pass@1 improved from 0.775 to 0.799).
- Weight analysis revealed low correlation between task updates in 7B models, indicating they utilize distinct representational pathways, allowing for successful combination without conflict.
Architecture Specificity: The optimal merging algorithm varied by architecture; DELLA performed best for Qwen-Coder 7B, while DARE excelled for DeepSeek-Coder 7B.

Significance and Claims

The paper claims to provide actionable insights for practitioners deploying resource-constrained agentic systems. The primary significance lies in the finding that the optimal multi-task learning strategy is fundamentally determined by model scale rather than task characteristics alone.

For Small Models: Investing in careful data mixture strategies during training yields superior results compared to post-hoc merging, avoiding the "catastrophic interference" caused by limited parameter capacity.
For Large Models: Merging task-specific specialists offers compelling advantages, enabling modular development where teams can optimize models independently before combining them. This approach reduces training costs by avoiding redundant multi-task training and allows for the addition of new capabilities without retraining from scratch.

The authors conclude that with careful strategy selection, small agentic systems can enjoy multi-task code intelligence without wasteful over-training, and that weight-space analysis can serve as a predictive tool for selecting the appropriate strategy before deployment.