💬 NLP

MeRoTune: RoPE-Safe Merging with a Tunable Dial

MeRoTune introduces a model merging technique that resolves attention subspace misalignment in RoPE-equipped models by learning constrained, RoPE-commuting correction matrices, enabling post-hoc tunable blending of fine-tuned models without modifying base weights.

Original authors: Salman Faroz

Published 2026-09-09✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Salman Faroz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, researchers often face a dilemma: they have a powerful, general-purpose computer brain, and they have trained separate, specialized versions of it to excel at specific tasks, like writing code in one language or answering questions in another. The goal is to combine these specialists into a single, versatile model without the cost of running multiple programs at once. The standard way to do this is to simply mix the mathematical weights of the two models together, like blending two batches of paint. However, this simple mixing assumes that the internal logic of both models is perfectly aligned. If one model thinks about a concept using a slightly different internal coordinate system than the other, blending them directly can create a confused result that is worse than either original. This is particularly tricky in modern models that use a specific mechanism to understand the order of words, a system that rotates information based on position. If the blending process ignores this rotation, it silently breaks the model's ability to understand context, often in ways that are invisible during training.

A researcher has developed a new method called MeRoTune to solve this problem. Instead of blindly averaging the two models, they first teach each model to adjust its own internal coordinate system so that they can agree on how to handle word order before they are mixed. They discovered that for this adjustment to work without breaking the model, the changes must follow a very specific, narrow set of rules. They proved mathematically that the only safe way to correct these models is to apply a specific type of rotation that respects the unique way the model handles position. They built a system where two specialized models learn these precise adjustments on their own, and then they can be combined in any proportion the user desires, acting like a tunable dial rather than a fixed mixture.

They tested this approach on two distinct versions of a language model: one trained to be an expert in Indonesian and coding, and another trained to be an expert in Japanese. Before attempting to merge them, they applied a strict mathematical check to ensure the two models were actually different enough to warrant the complex process. They verified that each model was genuinely better at its own specialty and worse at the other's, confirming that a real conflict existed that needed solving. They rejected several other candidate pairs that did not meet this criteria, ensuring they were only working on a genuine problem. Once they confirmed the pair was suitable, they trained the models to learn their own unique correction factors. These factors were designed to be simple rotations, ensuring the models remained stable and did not become distorted during the process.

When they combined the models using their new method, the results were superior to existing techniques. They tested the merged model across a wide range of blending ratios, from mostly Indonesian to mostly Japanese, and found that their approach never performed worse than the standard methods used by other researchers. In fact, on most tests, their method cleared a high bar for statistical reliability, while a competing method actually performed worse than the original, unmerged base model on at least one task. They also explored whether the blending ratio could be split into two independent controls, allowing each model to contribute more freely. They found that doing so caused the performance to collapse, proving that keeping the blend ratio tied together was essential for the method to work.

The study also revealed interesting details about what the models actually learned during the process. Most of the adjustments the models made were very small, tiny rotations of just a few degrees, suggesting that the models were mostly aligned to begin with. However, a small number of specific parts of the models required much larger changes, up to eighty-six degrees, and these parts also shrank in magnitude. This indicates that while the general structure was similar, there were specific, deep disagreements in how the models handled certain concepts that required significant realignment. They noted that this method is not a magic fix for every possible pair of models; it only works when the models are truly specialized in conflicting ways and when the internal structure of the model includes the specific position-handling mechanism they addressed.

Ultimately, this work demonstrates that merging AI models is not just a matter of averaging numbers. It requires understanding the hidden geometry of how the model processes information. By respecting the specific rules of how these models handle word order, they created a tool that allows users to smoothly dial between different expert capabilities without losing the quality of either. The method provides a reliable way to combine specialized AI brains, ensuring that the final result is a coherent, high-performing model rather than a confused mixture. The code and data for this work are publicly available, allowing others to verify the findings and apply the same careful, mathematically grounded approach to their own model combinations.

Technical Summary: MEROTUNE: ROPE-SAFE MERGING WITH A TUNABLE DIAL

1. Problem Statement

Model merging aims to combine multiple fine-tuned models into a single set of weights, preserving the specialized capabilities of each without the cost of serving multiple models. Standard methods (e.g., weight averaging, TIES, DARE) operate directly on raw weight tensors, implicitly assuming that the internal coordinate systems (specifically the attention subspaces) of the models being merged are already aligned.

Recent work [13] attempted to correct misalignments by learning an invertible correction matrix, $M$ , for query ( $W_q$ ) and key ( $W_k$ ) projections. Theoretically, applying $M$ to $W_q$ and $M^{-\top}$ to $W_k$ is a "no-op" in isolation because $(Mq)^\top (M^{-\top}k) = q^\top k$ . However, this cancellation relies on the assumption that nothing intervenes between the projection and the dot product.

The Core Conflict: Almost all modern open-weight language models (e.g., Qwen, Llama 3) utilize Rotary Position Embeddings (RoPE) exactly in this intervening space. RoPE applies a position-dependent rotation to the query and key vectors. If the learned correction matrix $M$ does not commute with the RoPE rotation, the "no-op" property is broken. The correction inadvertently distorts the relative-position dependencies that RoPE is designed to encode, a distortion that is invisible to standard training losses that do not explicitly re-derive the RoPE interaction.

2. Methodology

2.1 Theoretical Derivation: The RoPE-Commutant Class

The paper derives the specific class of matrices $M$ for which the cancellation $(Mq)^\top R(\Delta\theta) (M^{-\top}k) = q^\top R(\Delta\theta) k$ holds exactly in the presence of RoPE.

Condition: $M$ must commute with the RoPE rotation matrix $R(\Delta\theta)$ for all relative positions $\Delta$ .
Result: The valid matrices form a strict, low-dimensional subset of invertible matrices. Specifically, for each RoPE frequency pair (a 2D subspace), $M$ must be a scaled rotation acting independently within that pair.
Structure: $M$ is a block-diagonal matrix where each $2\times2$ block takes the form $aI + bJ$ (where $J$ is the rotation by $\pi/2$ ). This reduces the search space from $d^2$ parameters per head to $d$ parameters (one scale and one rotation angle per frequency pair).
Properties:
- Safety: Applying such a transformation to a single model is an exact no-op (Corollary 1).
- Stability: These matrices always have a condition number of exactly 1, preventing the ill-conditioning issues common in unconstrained matrix learning (Corollary 2).

2.2 The MeRoTune Algorithm

Building on this class, the author proposes MeRoTune (Merging + RoPE + Tunable):

Dual-Sided Alignment: Two fine-tuned models ( $A$ and $B$ ) each learn their own RoPE-compliant correction matrices ( $M_a$ and $M_b$ ).
Merging Operator: The merged weights at a blend ratio $\alpha$ $α$ are computed as:
- $W_k^{merge}(\alpha) = (1-\alpha) M_a^{-\top} W_k^A + \alpha M_b^{-\top} W_k^B$
- $W_q^{merge}(\alpha) = (1-\alpha) M_a W_q^A + \alpha M_b W_q^B$
- Other weights (MLP, embeddings, etc.) are linearly interpolated.
Tunable Dial: The method allows the final blend ratio $\alpha$ to be adjusted post-hoc. The training objective minimizes loss over a chosen range of $\alpha$ (defaulting to a fixed $\alpha=0.5$ , similar to LoRA's fixed scaling), enabling the merged model to be tuned to different operating points without retraining.
Pre-Training Validation: The paper introduces a mathematical condition (Eq. 7) to verify if a pair of fine-tunes is actually worth aligning. A pair is a valid target only if each model improves upon the base in its own specialty ( $\Delta > 0$ ) but performs worse or equal on the other's specialty ( $\Gamma \le 0$ ). If this "genuine bidirectional capability gap" does not exist, the alignment process adds noise without benefit.

3. Key Contributions

Proof of RoPE-Safe Corrections: A self-contained proof identifying the exact class of matrices (scaled rotations within frequency pairs) that preserve RoPE's relative-position structure during merging.
MeRoTune Method: A dual-sided merging algorithm that trains these constrained corrections to act as a runtime dial, allowing post-hoc adjustment of the blend ratio.
Validation Condition: A closed-form mathematical condition to empirically verify the existence of a genuine capability gap between two fine-tunes before committing to training, preventing wasted computation on invalid pairs.
Comprehensive Evaluation: An evaluation against TIES and DARE-TIES on a verified pair of Qwen2.5-1.5B-Instruct fine-tunes (Indonesian/Code vs. Japanese), demonstrating robustness across blend ratios.

4. Experimental Results

The method was tested on a pair of Qwen2.5-1.5B-Instruct models: one specialized in Indonesian and code, the other in Japanese.

Performance vs. Baselines:
- MeRoTune never lost to the official mergekit implementations of TIES or DARE-TIES at any tested blend ratio.
- DARE-TIES performed significantly worse than the unmerged base model on at least one benchmark axis.
- MeRoTune cleared the statistical noise floor on most benchmarks (MMLU, Indonesian, Japanese), whereas TIES fell below the base model on three of four benchmarks.
Blend Ratio Behavior:
- The blend ratio $\alpha$ functioned effectively as a dial. Metrics generally trended monotonically toward the stronger model for that specific task as $\alpha$ shifted.
- Training with a fixed blend ratio ( $\alpha=0.5$ ) generally outperformed or matched a variant where $\alpha$ was resampled randomly at every step.
Decoupling Ablation:
- The paper tested relaxing the tied blend ratio constraint ( $\alpha$ ) into two independent scaling factors ( $s_a, s_b$ ).
- Result: Decoupling caused a reliable degradation of performance across all benchmarks. Pushing both factors high simultaneously compounded uncontrolled deltas, leading to performance worse than random guessing in some cases. This confirms the tied convex constraint is necessary for stability.
Learned Transforms:
- The learned corrections were predominantly small rotations (median $\approx 10^\circ$ ) with scales close to 1.
- A small tail of blocks exhibited larger rotations ( $>45^\circ$ ) accompanied by significant scaling (shrinking), suggesting that while the method is theoretically safe, the optimizer occasionally utilized the capacity to reshape behavior beyond pure geometric realignment.

5. Significance and Claims

The paper claims that Rotary Position Embeddings are not a minor detail but a fundamental constraint that dictates which merging corrections are mathematically safe. By deriving the "RoPE-commutant" class, MeRoTune provides a method that:

Guarantees Safety: Unlike unconstrained methods, it cannot silently corrupt the relative-position logic of the model.
Offers Flexibility: It produces a model that can be tuned post-hoc like a dial, rather than locking into a single fixed merge.
Prevents Waste: The pre-training condition ensures resources are only spent on pairs with genuine, conflicting specializations.

The author emphasizes modesty in their claims: the results are specific to the Qwen2.5-1.5B architecture and the verified pair tested. They note that while the method is robust, the specific performance margins depend on the statistical noise floor of the benchmarks used, and the "alignment tax" (where fine-tunes regress below the base) remains a critical filter for applicability. The code, data, and scripts are publicly available.