Technical Summary: Gradient-Guided Optimizer Routing for LLM Adaptation (MoLF)

1. Problem Statement

The fine-tuning of Large Language Models (LLMs) faces a fundamental structural trade-off between Full Fine-Tuning (FFT) and Low-Rank Adaptation (LoRA).

• FFT offers high representational plasticity, essential for injecting high-entropy knowledge (e.g., factual updates), but risks catastrophic forgetting of pre-trained reasoning capabilities due to aggressive optimization.
• LoRA acts as an implicit regularizer, preserving pre-trained logic and preventing overfitting, but suffers from a capacity bottleneck when tasks require high-rank updates (e.g., memorizing complex factual associations).

Empirical evidence suggests that neither static architecture is universally optimal; the best choice depends on the intrinsic dimensionality of the task and the specific model. Existing solutions, such as adaptive-rank LoRA or Mixture-of-Experts (MoE) that route tokens, fail to resolve this tension because they either restrict the search space to low-rank subspaces or fragment gradient statistics by routing inputs, leading to unstable training dynamics and cold-start issues when ranks change.

2. Methodology: MoLF and MoLF-Efficient

The authors propose Mixture of LoRA and Full (MoLF) Fine-Tuning, a framework that unifies FFT and LoRA within a single training regime without altering the forward pass structure.

Core Architecture

MoLF treats the base model weights ($W_{base}$) as an "FFT expert" and injects multiple independent LoRA adapters as "LoRA experts." Unlike traditional MoE, all experts participate in every forward and backward pass. The forward pass is an unconditional superposition:
$$y = W_{base}x + \sum_{i=1}^{N} \alpha_i \frac{1}{\sqrt{r_i}} B_i(A_i(\text{Dropout}(x)))$$
where $\alpha_i$ is a Rank-Stabilized LoRA (RS-LoRA) scaling factor.

Gradient-Guided Optimizer Routing

The novelty of MoLF lies in sparsifying the parameter update step rather than the forward pass. The framework employs a custom Sparse AdamW optimizer with a three-phase routing mechanism:

1. Universal Momentum Tracking: All experts (FFT and LoRA) receive full-batch gradients and update their Adam first ($m_t$) and second ($v_t$) moment estimates. This ensures every expert maintains a mature, debiased momentum state, avoiding the "cold-start" failure mode seen in adaptive-rank methods.
2. Expert Scoring (Expected Preconditioned Descent - EPD): Instead of routing based on raw gradient norms, MoLF calculates an EPD score ($S_t^{(i)}$) for each expert. This score estimates the expected loss reduction per parameter if that expert were updated, normalized by the number of parameters ($N_{params}^{(i)}$):
   $$S_t^{(i)} = \frac{\eta_t^{(i)}}{N_{params}^{(i)}} \sum_{\theta \in \Theta_i} \frac{(m_t^{(i)})^2}{\sqrt{v_t^{(i)} + \epsilon}}$$
   This metric accounts for both the optimizer's intended step size and the local loss topology, allowing fair comparison between the massive FFT backbone and lightweight LoRA adapters.
3. Top-K Sparse Update: Only the top-$K$ experts (typically $K=1$) with the highest EPD scores receive physical weight updates via the standard AdamW step. Losing experts retain their previous weights but continue accumulating gradient statistics.

MoLF-Efficient (MoLF-E)

For memory-constrained environments, MoLF-E removes the FFT expert entirely. It routes updates among a pair of LoRA experts with varying ranks (e.g., $r$ and $128$). This variant freezes the base weights and non-linear parameters, reducing memory overhead while retaining the benefit of dynamic subspace search.

Inference

Post-training, MoLF allows for zero-overhead inference. All trained LoRA experts are mathematically projected directly into the base weights ($W_{final} = W_{base} + \sum \Delta W_i$), collapsing the multi-expert graph into a standard dense model identical to the base LLM architecture.

3. Key Contributions

1. Empirical Characterization of Fine-Tuning Regimes: The authors systematically evaluate FFT and LoRA across three datasets (CounterFact, MedMCQA, Text-to-SQL) and three models (Gemma-3-1B, Qwen2.5-1.5B, Qwen2.5-3B). They identify three distinct regimes:
   • Fact (Capacity Bottleneck): FFT dominates; LoRA fails to capture high-entropy, heavy-tailed singular value spectra.
   • Med (Spectral Regularization): High-rank LoRA outperforms FFT; the low-rank constraint acts as a regularizer preventing catastrophic forgetting of reasoning.
   • SQL (Low Intrinsic Dimension): Both perform similarly; LoRA captures the concentrated singular spectrum efficiently.
2. MoLF Framework: A unified method that dynamically navigates between FFT and LoRA at the optimizer level, ensuring stable training dynamics by keeping all experts active in gradient accumulation.
3. MoLF-E: A memory-efficient variant that outperforms existing adaptive-rank methods (AdaLoRA, AdaMix) by maintaining parallel LoRA experts of different ranks without an FFT path.

4. Experimental Results

• MoLF vs. Static Baselines: Across 9 settings (3 models $\times$ 3 tasks), MoLF consistently matches or exceeds the best static baseline (FFT or optimal-rank LoRA). It stays within 1.5% of the best baseline in all cases and achieves the outright best score in 3 configurations. Notably, it recovers FFT-level capacity on Fact tasks while surpassing FFT by up to 3.25% on Med tasks.
• MoLF-E vs. Adaptive PEFT: MoLF-E outperforms AdaLoRA and AdaMix in 8 out of 9 settings. The improvement is most significant on Fact tasks, where MoLF-E achieves up to 20% higher accuracy than the lowest-performing baseline (AdaMix) by effectively allocating updates to higher-rank subspaces when capacity is the bottleneck.
• Ablation Studies:
  • Sparse Updates: Updating all experts simultaneously (dense update) causes severe performance degradation on Med tasks due to optimization oscillations between competing representational capacities.
  • Routing Heuristic: The EPD score outperforms the Preconditioned Frobenius Norm (PFN) on Med tasks, demonstrating that incorporating the learning rate and step size is crucial for balancing experts of different scales.
  • Router Stability: The router converges rapidly (within ~500 steps) to a stable structural assignment, where specific modules commit definitively to either FFT or LoRA pathways rather than oscillating.

5. Significance and Claims

The paper claims that relying on a single static architecture is structurally limited for LLM adaptation. MoLF resolves the tension between representational plasticity and implicit regularization by decoupling the forward pass from the update mechanism.

The authors assert that MoLF provides a "continuous navigation" between training regimes, allowing the model to leverage the high capacity of FFT for knowledge injection and the regularization of LoRA for reasoning preservation without manual hyperparameter tuning or exhaustive rank searches. Furthermore, the ability to collapse the model post-training ensures that these gains do not incur inference latency penalties, making the approach practical for deployment. The work suggests that maintaining an expressive trainable architecture while sparsifying only expert-level updates is an effective strategy for capturing the benefits of diverse expert types.