🤖 machine learning

When Model Merging Rivals Joint Multi-Task Reinforcement Learning: A Task-Vector Geometry Analysis

This paper demonstrates that merging independently trained reinforcement learning agents achieves performance statistically indistinguishable from joint multi-task training, a result attributed to the near-orthogonality of task vectors which renders complex merging strategies equivalent to simple averaging.

Original authors: S. Aaron McClendon

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: S. Aaron McClendon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to do chores. You have two options. The first is to hire one super-smart teacher and show them every chore at once, hoping they learn to juggle everything perfectly. This is called "joint training." The second option is to hire two different specialists: one who only learns to wash dishes and another who only learns to fold laundry. Once they are experts, you try to "merge" their brains into one robot that can do both. This is called "model merging."

In the world of artificial intelligence, merging is like a magic trick. It promises to combine the skills of separate experts without needing to retrain them together, which is great if you can't share the data or if the experts were trained by different people. But there's a big question hanging over this magic trick: Does merging actually work as well as training one robot to do everything from the start? Most people assume merging is a clever workaround, but nobody has really tested it fairly against the "gold standard" of joint training in the same environment. This paper steps into that gap to see if the magic trick is real or just a lucky guess.

The researchers set up a controlled experiment using a digital playground called AppWorld, where an AI agent has to complete various tasks, like booking a flight or ordering food. They took a base AI model and trained two "specialists" on it: one focused only on easy tasks (Difficulty-1) and another on slightly harder ones (Difficulty-2). They also trained a "joint" model on both sets of tasks at the same time. Then, they took the two specialists and mashed them together using two different merging recipes (called TIES and RAM+). Finally, they pitted the merged models against the joint model to see who actually got the job done.

The results were surprisingly flat, but in a very specific way. When they looked at the main score—how many tasks were completed perfectly—all the models tied. The merged specialists performed just as well as the joint model. In fact, the differences between them were so tiny that they fell into the "noise wall," meaning they were statistically indistinguishable. It didn't matter if they used a fancy merging recipe or just averaged the brains together; the result was the same. The paper suggests that this happened because the two specialists learned in such different ways that their "thought patterns" (mathematically known as task vectors) were almost at right angles to each other. Imagine two people walking in completely different directions; when you try to combine their paths, you don't get a messy collision, you just get a gentle average that covers both bases.

The authors also looked deeper to see if they missed something. They found that while the merged models got the same number of perfect scores, they sometimes did slightly worse on the "partial credit" scale—meaning they might have gotten 90% of a task right instead of 100%, whereas the joint model might have gotten 95%. However, the paper argues this isn't a sign that merging failed. Instead, it's just a quirk of how you measure success. The merged models didn't discover any new, super-human abilities that neither specialist had; they just averaged out the existing skills.

Crucially, the paper rules out the idea that a smarter merging method could fix this. They tested whether picking a specific moment in training to stop and merge would help, but they found that in this specific setup, the "interference" between the models was already so low that there was nothing left to fix. The paper concludes that for this type of AI setup, merging is a perfectly valid substitute for joint training, but it doesn't offer a magical boost. It's a solid, reliable backup plan that works just as well as the hard way, but it doesn't break the laws of physics to do it.

Technical Summary: When Model Merging Rivals Joint Multi-Task Reinforcement Learning

Problem Statement

Model merging is frequently proposed as a substitute for joint multi-task training, particularly in reinforcement learning (RL) settings where independent agents are released and their training data cannot be pooled. However, the claim that merging matches joint training has rarely been empirically tested against the baseline it seeks to replace, as joint models are typically unavailable in the cross-domain scenarios where merging is applied. This paper addresses this gap by constructing a controlled comparison: training RL specialists on disjoint task subsets, merging them, and pitting the result against a jointly trained model on the same data.

Methodology

The study utilizes the AppWorld benchmark, an interactive digital-agent environment with tasks annotated by difficulty. The experimental setup involves the following components:

Base Model & Architecture: All models are fine-tuned from Qwen3-8B using LoRA (Low-Rank Adaptation) with rank $r=16$ on attention and MLP projection matrices.
Training Regime: Specialists are trained using LOOP (Leave-One-Out Policy Optimization), a critic-free policy-gradient method combining RLOO advantages with PPO-style clipping.
- Difficulty-1 Specialist: Trained only on difficulty-1 tasks.
- Difficulty-2 Specialist: Trained only on difficulty-2 tasks.
- Joint Model: Trained on the union of difficulty-1 and difficulty-2 tasks.
Merging Strategies: The specialists' task vectors (parameter offsets from the base) are merged using:
- TIES: Trims low-magnitude coordinates and resolves sign conflicts via magnitude voting.
- RAM+: Partitions coordinates by support co-occurrence (shared vs. unique) and amplifies unique directions.
Evaluation: Models are evaluated on a held-out test set (168 tasks) using Task-Goal Completion (TGC) (binary success) and a continuous partial-credit metric (fraction of unit tests passed).
Geometric Analysis: The paper introduces a calibrated geometric diagnostic to measure the relationship between specialists' task vectors. This includes global cosine similarity, sign agreement, and support overlap. Crucially, the authors calibrate these measurements against a floor (random rank-16 adapters) and a ceiling (same-run checkpoints) to distinguish learning signals from low-rank parameterization artifacts.

Key Contributions

Empirical Null Hypothesis: The paper establishes that, within this controlled setting, model merging is statistically indistinguishable from joint multi-task RL on the primary TGC metric. Furthermore, behavior-aware merging methods (RAM+, TIES) are indistinguishable from naive averaging.
Geometric Explanation: The authors demonstrate that the lack of interference requiring complex merging is due to the near-orthogonality of the specialists' task vectors. Despite a 65% support overlap (a consequence of identical LoRA target modules), the directional alignment (cosine similarity) remains low (0.06–0.10). This small shared component develops over training but remains distinct from the low-rank initialization floor.
Decoupling of Support and Direction: The study reveals that in LoRA-based RL specialists, support co-occurrence (the basis for RAM) does not imply directional alignment. Because the vectors are near-orthogonal, support-based merging strategies collapse toward simple averaging, rendering complex sign or subspace resolution unnecessary in this regime.
Metric Sensitivity Analysis: While the continuous partial-credit metric shows merged models scoring lower than joint models, the authors argue this does not indicate a failure of merging as a substitute for joint training. The metric reversal (merges outperforming joint on TGC but underperforming on continuous scores) and the lack of emergent cross-task capability (joint performance does not exceed the best specialist) suggest the difference is a mechanical consequence of averaging near-orthogonal vectors rather than a loss of capability.

Results

Performance Parity: On the primary metric (TGC), the Difficulty-1 specialist, Difficulty-2 specialist, Joint model, TIES merge, and RAM+ merge all fall within a statistically overlapping band (12.5% – 16.1% aggregate TGC). No pairwise contrast among RL models is significant ( $p \geq 0.56$ ).
Geometric Findings:
- Cosine Similarity: The global cosine between specialists is 0.06–0.10, rising monotonically from near-zero during training but saturating well below the ceiling (0.78–0.81 for same-run checkpoints).
- Sign Agreement: Sign agreement on shared support is ~52%, consistent with chance.
- Support Overlap: High (~65%) due to shared LoRA modules, yet this does not translate to directional interference.
Merging Dynamics: Because the vectors are near-orthogonal, TIES and RAM+ effectively reduce to uniform averaging. The "unique" sets in RAM are too small to drive significant amplification, and sign conflicts in TIES are resolved arbitrarily due to lack of consensus.
Checkpoint Selection: An analysis of "merge-aware" early stopping (selecting checkpoints to minimize interference while maximizing reward) reveals no leverage in this regime. Interference and capability grow together; the optimal point for merging coincides with the peak capability checkpoint, leaving no high-capability, low-interference alternative.

Significance and Claims

The paper frames its findings as a calibrated null: a carefully bounded characterization of where model merging suffices.

For Practitioners: In settings where data can be pooled, merging offers no statistical advantage over joint RL at this scale (8B parameters, ~10 iterations). However, for practitioners who cannot pool data, the results provide assurance that merging does not obviously sacrifice the performance that joint RL would have delivered.
Mechanistic Insight: The study clarifies that the success of merging in this context is not due to the sophistication of the merging algorithm (e.g., RAM or TIES) but rather the inherent geometry of the learned task vectors. The near-orthogonality of LoRA-based RL specialists minimizes interference, making complex disentanglement unnecessary.
Limitations and Scope: The authors are modest about the scope, noting that these results are specific to the within-benchmark setting, the LoRA parameterization, and the specific difficulty splits used. They acknowledge that larger models, different training regimes, or full fine-tuning might yield different geometric properties where merging methods could have more leverage. The paper explicitly avoids claiming that merging is universally superior or that the observed "null" applies to all multi-task scenarios, focusing instead on the specific conditions under which merging matches joint training.