Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

In the rapidly evolving world of artificial intelligence, researchers are constantly trying to build machines that can do many things at once. Imagine a student who has spent years mastering mathematics, another who has become a coding wizard, and a third who is an expert in following complex instructions. In the past, to use all these skills, a computer would need to run three separate programs, each dedicated to one subject. This is inefficient, requiring massive amounts of computer memory and energy. A more elegant solution is to combine these separate experts into a single, versatile model. This process, known as model merging, attempts to blend the "knowledge" of different specialized programs into one unified brain without needing to retrain it from scratch. However, this blending is notoriously difficult. When you mix two distinct sets of instructions, they often clash. The computer gets confused, and the resulting model performs poorly on all tasks, as if the different skills are fighting over the same space in the machine's memory.

The core of this problem lies in how these artificial brains store information. Inside a neural network, different skills often share the same physical pathways, a phenomenon researchers call superposition. It is as if multiple distinct ideas are trying to occupy the same narrow hallway at the same time. When researchers try to simply add the weights of two models together, these overlapping ideas interfere with one another, creating a tangled mess where useful information is lost. Previous attempts to fix this have involved trying to untangle the mess within the original, crowded space, often by cutting away parts of the model or rescaling numbers. But when the interference is severe, these methods are like trying to sort a pile of mixed-up wires by only looking at the knots; they cannot fully separate the individual strands without damaging the circuit.

A team of researchers has proposed a new way to solve this by changing the perspective entirely. Instead of trying to untangle the wires in the crowded hallway, they suggest moving the wires into a much larger, empty room where there is plenty of space to lay them out separately. They call this approach High-Dimensional Sparse Disentanglement Merging. The researchers used a tool known as a Sparse Autoencoder, which acts like a translator. It takes the tangled instructions from a specific task and converts them into a long list of simple, distinct features. Because this new list is much longer than the original set of instructions, each feature gets its own dedicated space, allowing the computer to see exactly which parts of the knowledge belong to math, which belong to coding, and which belong to following instructions. Once the information is separated in this spacious, high-dimensional room, the researchers can carefully combine the relevant parts without the different skills stepping on each other's toes. After the combination is complete, the translator converts the clean, separated list back into the original format so the model can use it.

However, translating every single layer of a massive model into this new format would be incredibly slow and expensive. To solve this, the researchers introduced a smart selection method called a Group-Ranked Zeroth-Order Optimizer. This tool acts as a scout, quickly testing which parts of the model are most critical for a specific task. It does this by making tiny, random adjustments to different sections of the model and seeing how much the performance changes, without needing to calculate complex gradients. By identifying only the most important layers, the system applies the expensive translation and separation process only where it is truly needed, leaving the rest of the model to be merged using simpler, faster methods. This ensures that the system remains efficient while still gaining the benefits of deep separation.

The researchers tested this new framework using the Qwen2.5 series of language models, which are powerful artificial intelligence systems. They created expert models for tasks like mathematical reasoning, code generation, and following complex instructions. When they merged these experts using traditional methods, the results were often disappointing, with performance dropping significantly on difficult tasks. In contrast, their new method consistently outperformed existing techniques. On a large 7-billion parameter model, their approach achieved an average score of 68.49 across math, coding, and instruction-following tasks, beating the previous best method by a clear margin. The improvement was even more dramatic in highly conflicting scenarios. When they tried to merge four different tasks, including safety alignment, on a smaller 1.5-billion parameter model, traditional methods saw their average scores collapse to as low as 15.45. The new method maintained a score of 36.48, effectively preserving the model's ability to do math and code while also adhering to strict safety guidelines.

The study suggests that the key to successful model merging is not just in how we mix the numbers, but in where we mix them. By projecting the task information into a higher-dimensional space where features can be clearly distinguished, the researchers were able to reduce the interference that usually plagues these systems. They found that their improved autoencoder, which uses a specific technique to keep the most important features active while ignoring the rest, was crucial for this success. Furthermore, their layer-selection tool proved that it is not necessary to apply this complex separation to every part of the model; focusing on the critical layers was enough to capture the benefits. The results indicate that this approach offers a robust way to build generalist models that can handle multiple, conflicting tasks without the need for additional training data or expensive retraining. While the method does require some initial computational effort to set up the translation tools, the payoff is a model that retains its specialized skills much better than those created with older techniques. This work provides a concrete path forward for creating more capable and versatile artificial intelligence systems that can navigate the complex landscape of multiple human tasks without losing their way.