🤖 AI

Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

This paper proposes Hard-Routed MoR-LoRA, a two-stage framework that composes frozen, independently trained reasoning LoRA experts by using a lightweight router to select exactly one expert per token via hard top-1 routing, thereby preserving the original unit-scale additive updates of the experts while requiring significantly fewer trainable parameters than soft-routing baselines.

Original authors: Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of five brilliant, specialized chefs. One is a master of Italian pasta, another a genius at Japanese sushi, a third at Mexican tacos, and so on. Each chef was trained separately in their own kitchen with their own secret recipes.

Now, imagine you want to open a single restaurant that can cook any of these dishes perfectly, but you can't bring all the chefs together to retrain them as a team. You also can't mix their recipes into one giant "super-recipe" because that would ruin the delicate balance of their individual skills.

This is the problem the paper "Learning to Select, Not Relearn" solves for Artificial Intelligence (AI).

Here is the simple breakdown of their solution:

The Problem: The "Soft Mix" Mistake

In the past, when AI researchers tried to combine these specialized "chefs" (called LoRA experts), they used a method called Soft Routing.

Think of Soft Routing like a blender. If you want to make a dish, the blender takes a little bit of the Italian chef, a little bit of the Sushi chef, and a little bit of the Taco chef, and blends them all together.

The Issue: The Italian chef was trained to cook a full plate of pasta on their own. If you only give them 10% of the ingredients (because the blender diluted them), their dish tastes terrible. The math doesn't add up. To fix this, previous methods had to send the chefs back to school to relearn how to cook with "diluted" ingredients, which is expensive and slow.

The Solution: The "Hard-Routed" Waiter

The authors propose a new system called Hard-Routed MoR-LoRA. Instead of a blender, they use a super-efficient waiter.

The Chefs Stay Frozen: The specialized chefs (the AI experts) are trained individually to be the best at their specific task (like math or medical questions). Once they are done, they are "frozen." They don't change. They keep their original, perfect recipes.
The Smart Waiter: The system trains a tiny, lightweight "waiter" (a router). This waiter's only job is to look at the customer's order (the input question) and decide: "Okay, this is a math problem. Send it to the Math Chef. This is a medical question. Send it to the Doctor Chef."
One Chef, One Dish: The waiter sends the entire order to exactly one chef. The chef cooks the dish using their full, original recipe (100% scale). No blending, no dilution.
The Magic Trick: Since the waiter has to make a sudden, discrete choice (Chef A OR Chef B), it's hard to teach the waiter how to get better using standard math. The paper uses a clever trick called a Straight-Through Estimator (STE). Imagine the waiter practicing by "pretending" to split the order (to learn the math) but actually serving the full dish to one chef (to keep the quality high). This allows the waiter to learn without breaking the chefs.

Why This is a Big Deal

The paper tested this on five different types of tasks (math, common sense, medicine, reading, and grammar) using different sizes of AI models.

Better Results with Less Work: Their "Hard-Routed" system performed just as well (or better) than the old "blender" methods, but it required far fewer trainable parameters. It's like hiring one smart waiter instead of retraining five chefs.
Preserving Reasoning: The paper found that when you use a special training method called RLVF (Reinforcement Learning from Verifiable Feedback) to train the chefs first, they get much better at "thinking" (reasoning step-by-step). The Hard-Routed system preserves this thinking ability perfectly because it doesn't dilute the chef's work.
The "Blender" was lying: The authors analyzed the old "blender" methods and found that even when they tried to blend two chefs, the blender usually ended up relying on just one chef anyway (about 70% of the time). So, the "blender" was doing extra work for no reason. The "Hard-Routed" waiter just skips the middleman and picks the right chef immediately.

The Bottom Line

This paper teaches us that when combining specialized AI experts, you don't need to retrain them or mix their brains together. You just need a smart, lightweight system to select the right expert for the job and let them do their thing exactly as they were trained to do.

It's the difference between forcing a team to compromise on a mediocre group project versus letting the right specialist handle the task perfectly, one by one.

Technical Summary: Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

Problem Statement

The paper addresses the challenge of composing independently trained Low-Rank Adaptation (LoRA) adapters into a single Large Language Model (LLM) for multi-domain adaptation, particularly when original training data cannot be shared due to privacy or regulatory constraints.

A critical issue arises when applying standard Mixture-of-Experts (MoE) soft routing to frozen, independently trained LoRA adapters. LoRA modules are trained under the assumption of a unit-scale additive update ( $W_{updated} = W_{pretrained} + AB$ ). However, standard soft MoE routing combines experts using weighted sums ( $W_{updated} = W_{pretrained} + \sum \omega_i A_i B_i$ ), where routing weights $\omega_i < 1$ . This creates a scale mismatch: the effective update applied during integration differs from the unit-scale update under which the expert was originally trained.

Existing solutions, such as LoRA-Mixer, attempt to resolve this by retraining the adapters with auxiliary preservation losses. This approach increases the number of trainable parameters, requires more data, and risks altering the knowledge encoded in the original experts. Furthermore, this challenge is particularly pronounced for instruction-tuned models with Chain-of-Thought (CoT) reasoning, where conventional fine-tuning can degrade internal reasoning behaviors, and existing composition methods often fail to preserve these specialized reasoning traces.

Methodology

The authors propose Hard-Routed MoR-LoRA, a two-stage framework designed to compose frozen reasoning LoRA experts through unit-scale hard selection, thereby avoiding the need to retrain the experts.

Stage I: Training Reasoning Experts

Domain-specific LoRA adapters are trained independently using Reinforcement Learning from Verifiable Feedback (RLVF). Unlike supervised fine-tuning (SFT) which encourages imitation of reference outputs, RLVF optimizes for correctness using automatically verifiable rewards (e.g., checking the final answer). This paradigm encourages the emergence of structured CoT reasoning without overriding the base model's aligned reasoning patterns. The resulting experts are then frozen.

Stage II: Hard-Routed Integration

The frozen experts are integrated into a shared model using a Hard-Routed Mixture.

Hard Top-1 Routing: For each token, the router selects exactly one expert. This ensures the selected expert is applied with unit scale, preserving the original LoRA assumption ( $W_{updated} = W_{pretrained} + 1 \cdot A_j B_j$ ).
Straight-Through Estimator (STE): Since discrete selection (argmax) is non-differentiable, the authors employ STE. In the forward pass, the hard decision is used; in the backward pass, gradients are propagated through the soft probabilities. This allows for end-to-end training of the router while maintaining the discrete, unit-scale behavior during inference.
Parameter Efficiency: Only a lightweight shared linear router and a small attention LoRA module are trained. The pretrained experts remain frozen.
Distillation: The router is trained via Supervised Fine-Tuning (SFT) on a small dataset of reasoning traces distilled from the frozen experts. The objective is to learn which expert to select, not to relearn the reasoning capability itself.

Key Contributions

Identification of Scale Mismatch: The paper identifies that applying soft MoE routing to independently trained frozen LoRA adapters introduces a scale mismatch that alters the effective function of the adapters.
Hard Unit-Scale Routing: The authors propose using hard top-1 routing with a Straight-Through Estimator to preserve the unit-scale application of experts, enabling modular composition without retraining.
Two-Stage Framework: A novel pipeline that separates reasoning acquisition (via RLVF) from expert selection (via hard routing), keeping experts frozen during integration.
Empirical Validation: The study demonstrates that this approach preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing baselines.

Experimental Results

Experiments were conducted across five benchmarks (GSM8K, ARC-Challenge, MedQA, BoolQ, CoLA) and multiple model scales (1B, 3B, 8B) using LLaMA-3, Gemma-3, and Qwen-2.5 families.

RLVF Superiority: RLVF-trained experts significantly outperformed SFT-trained experts on reasoning-heavy tasks for capable models (3B and 8B), whereas SFT sometimes degraded reasoning performance in large instruction-tuned models.
Performance vs. Parameters: Hard-Routed MoR-LoRA achieved comparable or better performance than soft-routing baselines (LoRAMixer, MoLE) while training substantially fewer parameters. For example, on the 3B model with RLVF experts, the proposed method trained ~73M parameters to reach 72.07 average accuracy, whereas LoRAMixer required ~606M parameters to reach 65.67.
Routing Behavior Analysis: Analysis of soft top-2 routing revealed that it often collapses to near top-1 selection (average dominant weight ~0.71), suggesting that hard routing captures the intended behavior more directly and efficiently.
Data Efficiency: The integration stage (Stage II) required only ~1,000 samples per dataset to achieve performance close to training on the full dataset, indicating that the router learns selection patterns rather than domain knowledge.
Mixed-Domain Capability: In mixed-domain evaluations (e.g., a prompt containing both math and reading comprehension), token-level hard routing outperformed prompt-level routing, demonstrating flexibility in selecting different experts within a single input.

Significance and Claims

The paper claims that modular adaptation can be achieved by learning when to use each expert rather than modifying what each expert knows.

The authors argue that for frozen pretrained LoRA experts, hard unit-scale routing provides a simple and efficient abstraction. By treating integration primarily as a routing problem, the method avoids the computational cost and potential knowledge degradation associated with retraining experts. The results suggest that the primary benefit of composition comes from selecting the appropriate frozen expert for the context, rather than extensively modifying the experts' parameters.

The work highlights that while prompt-level routing is sufficient for clean single-domain inputs, token-level hard routing offers necessary flexibility for complex, mixed-domain inputs. The authors modestly note that the method is not entirely training-free, as Stage II requires some labeled data to learn the routing policy, and it currently does not support true zero-shot integration of new domains without additional routing data.