Technical Summary: RotMoLE – Enhancing Mixture of Low-Rank Experts through Rotational Gating

Problem Statement

While Large Language Models (LLMs) utilize Mixture-of-Experts (MoE) architectures and Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA to handle diverse tasks, existing Mixture of Low-rank Experts (MoE-LoRA) frameworks face limitations in complex, multi-task, or multilingual scenarios.

The core issue identified is the limitation of conventional gating mechanisms in MoE. Standard routers apply only a scalar reweighing (scaling) to selected experts. While this modulates the magnitude of an expert's contribution, it lacks the capacity to perform complex spatial transformations on the expert outputs. Consequently, the solution space formed by the weighted aggregation of experts is restricted to a linear combination along fixed spatial directions. This limitation is particularly pronounced when the number of candidate experts is limited relative to the number of tasks (e.g., in resource-constrained scenarios where experts must be shared across multiple domains), as the router cannot sufficiently distinguish or specialize the shared experts beyond simple magnitude scaling.

Furthermore, implementing complex transformations (such as full linear transformations) via gating modules typically requires a prohibitive number of trainable parameters (e.g., $d \times d$ matrices for hidden dimension $d$), which negates the parameter efficiency of PEFT.

Methodology: RotMoLE

The authors propose RotMoLE (Rotatable Mixture of Low-rank Experts), a specialized MoE-LoRA framework that introduces a rotational gating mechanism alongside the traditional scaling gate.

Theoretical Foundation

The method is grounded in the observation that any transformation between two non-zero vectors can be decomposed into a pure scaling operation and a pure rotation within the 2D sub-plane spanned by those vectors. By leveraging the low-rank structure of LoRA (where rank $r \ll d$), RotMoLE applies these transformations within the $r$-dimensional space of the expert adapters rather than the full hidden dimension.

Architecture Components

The weight update for the model is formulated as:
$$W = W_0 + \sum_{i \in TopK(x)} g_i B_i R_i A_i$$
Where:

• $W_0$ is the pre-trained weight matrix.
• $A_i$ and $B_i$ are the low-rank matrices for the $i$-th expert.
• $g_i$ is the scaling gate value (standard scalar weight).
• $R_i$ is the rotation matrix specific to the $i$-th expert.

The framework operates in two modes based on the LoRA rank $r$:

1. Rank $r = 2$ (2D Rotation):

   • The rotation is modeled using a single learnable angle $\theta_i$.
   • The rotation matrix $R_i$ is a standard 2D rotation matrix:
     $$R_i = \begin{pmatrix} \cos \theta_i & -\sin \theta_i \\ \sin \theta_i & \cos \theta_i \end{pmatrix}$$
   • The angle $\theta_i$ is generated by a rotation gate $\Theta(x)$, which outputs values in $[-\pi, \pi]$ using a Sigmoid-based normalization to ensure stability and periodicity.

2. Rank $r > 2$ (High-Dimensional Rotation):

   • Since a single angle cannot define a rotation in dimensions $>2$, the method constructs a specific rotation plane $P_i$ for each expert.
   • The plane is defined by two basis vectors:
     1. The normalized input embedding to the expert ($e_1 = A_i x / |A_i x|$).
     2. A learnable cluster center vector $q_i$ specific to the expert, which represents the aggregated center of inputs routed to that expert. The second basis vector $e_2$ is derived from the component of $q_i$ orthogonal to $e_1$.
   • The rotation matrix $R_i$ is constructed by projecting the 2D rotation (defined by $\theta_i$) onto this $r$-dimensional plane spanned by $e_1$ and $e_2$.

Parameter Efficiency

RotMoLE maintains high parameter efficiency. The additional parameters introduced are:

• For $r=2$: $d \times n$ parameters (for the rotation gate weights).
• For $r>2$: $(d + r) \times n$ parameters (for the rotation gate and the cluster center vectors $q$).
  This is significantly smaller than the $d^2 \times n$ parameters required for full linear transformations, keeping the overhead comparable to the expert modules themselves.

Key Contributions

1. Identification of Limitations: The paper highlights that existing MoE routers are limited to norm scaling, lacking the ability to perform complex spatial transformations necessary for diverse scenarios with limited expert counts.
2. RotMoLE Framework: The proposal of a specialized MoE-LoRA framework that integrates a rotation gate, enabling 2D rotations within the low-rank space of experts to expand the solution space of weighted expert combinations.
3. Empirical Validation: Extensive experiments across multi-task and multilingual joint learning scenarios demonstrating that RotMoLE outperforms conventional MoE-LoRA baselines and other recent variants.

Experimental Results

The authors evaluated RotMoLE on Llama-3.2-3B, Qwen-2.5-3B, and GLM-4-9B across several benchmarks:

• Multi-Task Joint Learning (QA & GLUE):
  • On a mixture of three QA tasks (CommonsenseQA, OpenBookQA, SocialIQA), RotMoLE consistently outperformed the MoE-LoRA baseline, achieving average accuracy improvements ranging from 1% to 10% (average ~5%).
  • Improvements were most pronounced when the number of experts was limited (e.g., $n=2$), validating the hypothesis that rotation helps distinguish shared experts.
  • Similar gains were observed on GLUE tasks (SST-2, MRPC, QNLI).
• Multilingual Joint Learning:
  • On Multilingual Text Generation (MTG) and XNLI (Natural Language Inference), RotMoLE achieved higher average and per-language scores compared to baselines.
  • Convergence analysis indicated that RotMoLE exhibits better anti-overfitting properties, converging more slowly initially but achieving better final performance without the early saturation seen in baselines.
• Comparison with Variants:
  • RotMoLE outperformed other MoE-LoRA variants focusing on forward routing (MoCLE, MoLA, AdaMoLE) in terms of overall and per-task accuracy.
• Ablation Studies:
  • Experiments confirmed that the performance gains are due to the rotational structure rather than simply the increase in parameter count. A size-equivalent baseline using an MLP gate performed worse, suggesting the rotation mechanism is more effective at utilizing extra parameters for diversity.

Significance and Claims

The paper claims that RotMoLE provides a more powerful transformation function for low-rank experts beyond simple scaling. By introducing a rotation gate, the framework significantly expands the global solution space of MoE, allowing for better exploitation and specialization of experts, particularly in scenarios where experts are shared across multiple tasks or languages.

The authors emphasize that this approach addresses a specific gap in current PEFT-MoE research: the inability to model diverse exploitation strategies for candidate experts when the number of experts is limited. The results suggest that RotMoLE offers a robust, parameter-efficient method to enhance the representational capacity of LLMs in complex, multi-domain applications without incurring the computational costs of full-rank transformations.