💬 NLP

Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning

This paper introduces Maximal-Update Adaptation ( $\mu$ A), a theoretical framework that defines how optimal learning rates scale with LoRA rank and initialization, enabling practitioners to transfer tuned learning rates from efficient LoRA experiments to full finetuning across diverse tasks.

Original authors: Nan Chen, Soledad Villar, Soufiane Hayou

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nan Chen, Soledad Villar, Soufiane Hayou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly talented chef (a large AI model) who has already learned to cook thousands of dishes. Now, you want to teach them a very specific new recipe, like "how to make the perfect sushi."

You have two ways to do this:

Full Finetuning (FFT): You hand the chef a blank notebook and tell them to rewrite their entire cookbook from scratch, keeping only the sushi parts. This is powerful but requires a huge kitchen, a lot of time, and a massive budget.
LoRA (Low-Rank Adaptation): Instead of rewriting the whole book, you give the chef a small, sticky note pad (the "adapter") to stick on top of the existing pages. You only write the new sushi instructions on these notes. This is cheap, fast, and uses very little memory.

The problem? The sticky notes come in different sizes (called Ranks). Sometimes you use a tiny 4x4 inch note; other times, a huge 1024x1024 inch note. The paper asks a simple but tricky question: If you change the size of the sticky note, do you need to change how fast the chef writes?

In the world of AI, "how fast the chef writes" is the Learning Rate. If you write too fast, the notes get messy and the chef gets confused (the training crashes). If you write too slow, nothing gets done.

The Big Discovery: "The Golden Rule of Sticky Notes"

The authors of this paper realized that for years, people had to guess the right writing speed every time they changed the size of the sticky note. If they switched from a small note to a big one, they often had to start over and guess the speed again.

They developed a new theory called Maximal-Update Adaptation (µA). Think of this as a "Universal Instruction Manual" that tells you exactly how to set the writing speed based on the note size and how you started writing.

They found there are two main scenarios (or "regimes") depending on how you set up your sticky notes:

Scenario 1: The "Shrinking Speed" Rule

How it works: You start by writing the notes randomly on one side, leaving the other side blank.
The Problem: If you make the sticky note bigger (increase the Rank), you must slow down your writing speed significantly.
The Analogy: Imagine you are painting a wall. If you use a tiny brush (small rank), you can paint quickly. If you switch to a giant roller (large rank), you have to move much slower, or you'll splatter paint everywhere.
The Result: Every time you double the size of the note, you have to cut your speed in half. This makes tuning annoying because you have to re-calculate the speed every time you change the note size.

Scenario 2: The "Magic Constant" Rule (The Breakthrough)

How it works: You start by writing the notes on the other side (a different initialization method) and use a specific scaling factor.
The Discovery: In this setup, the writing speed does not change no matter how big the sticky note is. Whether you use a tiny 4x4 note or a giant 1024x1024 note, the perfect speed is exactly the same.
The Analogy: It's like having a self-adjusting pen. No matter how big the paper is, the pen automatically finds the perfect flow. You don't need to guess; the speed is "rank-invariant."

The Superpower: Transferring Speed from Notes to the Whole Book

The most exciting part of the paper is what happens with Scenario 2.

The authors discovered that the "Magic Constant" speed for the small sticky notes (LoRA) is exactly the same as the perfect speed for rewriting the entire cookbook (Full Finetuning).

Why is this a big deal?
Usually, tuning a Full Finetuning (rewriting the whole book) is expensive and slow because you need a massive computer to do it.

Old Way: You try to tune the speed on the big computer, fail, try again, and waste money.
New Way (µA): You use a small, cheap computer to tune the speed on the tiny sticky notes (LoRA). Once you find the perfect speed there, you can copy-paste that exact speed to the expensive Full Finetuning job, and it will work perfectly.

Summary of the Paper's Claims

LoRA is tricky: Changing the size of the adapter (Rank) usually breaks your learning speed settings, forcing you to re-tune everything.
There is a fix: By choosing the right way to start (Initialization) and scale the notes, you can find a "sweet spot" where the learning speed stays the same regardless of the adapter size.
The Transfer: This specific setup allows you to find the perfect learning speed on a small, cheap LoRA experiment and immediately apply it to a massive, expensive Full Finetuning project without needing to re-tune.
Proof: They tested this on many different types of AI tasks (writing, seeing images, solving math, and generating art) and found that their "Universal Instruction Manual" worked every time.

In short, the paper gives us a reliable map to navigate the confusing world of AI tuning, saving time and money by letting us test on small models and confidently apply the results to giant ones.

Technical Summary: Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning

Problem Statement

Low-Rank Adaptation (LoRA) is a standard method for parameter-efficient finetuning (PEFT) of large models, adding a low-rank factorization $\alpha BA$ to frozen pretrained weights. However, the training dynamics of LoRA are highly sensitive to hyperparameter interactions, specifically the initialization scheme (which factor is randomized vs. zeroed), the rank-dependent scaling multiplier $\alpha$ , and the learning rate $\eta$ .

Practitioners face a significant challenge: the optimal learning rate is not invariant to changes in the LoRA rank $r$ . Changing the rank or switching between standard design choices (e.g., $Init[A]$ vs. $Init[B]$, or $\alpha=1$ vs. $\alpha=r^{-1}$ ) often renders a previously optimal learning rate suboptimal, necessitating costly re-tuning. Furthermore, it remains unclear how to transfer learning rates from LoRA configurations to Full Finetuning (FFT), a process that is computationally expensive. The core question addressed is: How does the optimal learning rate $\eta$ depend on the adapter rank $r$ and model width $n$ under different LoRA configurations?

Methodology

The authors introduce Maximal-Update Adaptation ( $\mu A$ ), a theoretical framework inspired by Maximal-Update Parametrization ( $\mu P$ ) used in pretraining. The analysis is conducted in the joint limit where both the model width $n$ and the LoRA rank $r$ approach infinity ( $n, r \to \infty$ ).

Theoretical Framework

Stability and Non-Vanishing Updates: The framework defines "stable feature learning" as a regime where feature updates remain bounded ( $O(1)$ ) to prevent explosion, yet non-vanishing ( $\Theta(1)$ ) to ensure learning does not stall.
Decomposition of Updates: The feature update $\Delta Z_B$ is decomposed into three terms ( $\delta_1, \delta_2, \delta_3$ ) representing linear updates of one factor while the other is fixed, and multiplicative updates of both. The theory analyzes the asymptotic scaling of these terms under different initialization schemes.
Assumptions: The analysis assumes bounded forward/backward signals and utilizes a SignSGD optimizer abstraction (a tractable proxy for Adam) to derive closed-form scaling rules.
Initialization Schemes:
- Init[A]: $A$ is initialized randomly ( $A \sim \mathcal{N}(0, 1/n)$ ), $B=0$ .
- Init[B]: $B$ is initialized randomly ( $B \sim \mathcal{N}(0, 1/r)$ ), $A=0$ .
Scaling Multipliers: The study examines common choices for the multiplier $\alpha$ , including constant values ( $\alpha=1$ ) and rank-dependent values ( $\alpha=r^{-1}$ ).

Derivation of Scaling Rules

By enforcing the condition $\Delta Z_B = \Theta(1)$ , the authors derive explicit scaling prescriptions for $\eta$ as a function of $n$ and $r$ :

For Init[A]: The optimal learning rate depends on both $n$ and $r$ . Specifically, if $\alpha=1$ , $\eta \propto n^{-1/2}r^{-1/2}$ . If $\alpha=r^{-1}$ , $\eta \propto n^{-1/2}$ (rank-invariant).
For Init[B]: With $\alpha=1$ , the optimal learning rate scales as $\eta \propto n^{-1}$ and is independent of rank $r$ . Crucially, this scaling matches the theoretical requirement for Full Finetuning of a linear layer under the same assumptions.

Key Contributions

$\mu A$ Framework: Introduction of a theoretical framework characterizing LoRA finetuning dynamics in the joint $n, r \to \infty$ limit, defining conditions for maximal (stable) adaptation.
Identification of Two Regimes:
- Rank-Dependent Regime: Under $Init[A]$ with constant $\alpha$ , the optimal learning rate scales inversely with rank (e.g., $\eta \propto r^{-1/2}$ ).
- Rank-Invariant Regime: Under $Init[B]$ with $\alpha=1$ , the optimal learning rate is invariant to rank changes.
LoRA-to-FFT Transfer: Identification of a specific configuration ($Init[B]$ with $\alpha=1$ ) where the optimal learning rate scaling ( $\eta \propto n^{-1}$ ) matches that of Full Finetuning. This provides a necessary condition for transferring learning rates from LoRA to FFT without re-tuning.
Empirical Validation: Extensive experiments across diverse tasks (language, vision, vision-language, image generation, reinforcement learning) and architectures (Llama, Qwen, RoBERTa, ViT, Stable Diffusion) validating the derived scaling rules.

Results

The experimental results consistently align with the theoretical predictions:

Init[A] with $\alpha=1$ : As rank $r$ increases, the optimal learning rate decreases. For a $4\times$ increase in rank, the optimal learning rate decreases by approximately $2\times$ (consistent with $\eta \propto r^{-1/2}$ ).
Init[A] with $\alpha=r^{-1}$ : The optimal learning rate is approximately rank-invariant, though the stable range is narrow.
Init[B] with $\alpha=1$ : The optimal learning rate is rank-invariant and aligns closely with the optimal learning rate for Full Finetuning across most models.
Transferability: Learning rates tuned on LoRA using the $Init[B], \alpha=1$ configuration transfer reliably to Full Finetuning, drastically reducing the cost of hyperparameter search for FFT.
RLVR Sensitivity: In Reinforcement Learning with Verifiable Rewards (RLVR), suboptimal learning rates not only degrade performance but significantly increase training time (by 2–4 $\times$ ) due to models getting stuck in exploratory phases. The $\mu A$ rules help identify stable regions to avoid this.

Significance and Claims

The paper claims to provide a principled approach to learning rate selection in LoRA, moving beyond heuristic rules. By characterizing the joint limit of width and rank, the authors explain why certain configurations require rank-dependent learning rates while others do not.

The primary significance lies in the transferability of hyperparameters. The authors demonstrate that by selecting the correct initialization ($Init[B]$) and scaling ( $\alpha=1$ ), practitioners can:

Avoid re-tuning learning rates when changing LoRA ranks.
Directly transfer the optimal learning rate from a low-rank LoRA experiment to a Full Finetuning setup, saving significant computational resources.

The authors modestly note limitations: the theory is asymptotic and may be sensitive to noise in finite $(n, r)$ settings, and the derived matching condition is necessary but not proven sufficient for all transfer scenarios. However, empirical evidence suggests strong agreement between theory and practice across diverse domains.