📊 statistics

When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

This paper mathematically demonstrates and empirically validates that excessive pre-training can computationally hinder LoRA fine-tuning on single-index models by inducing a prolonged search phase, revealing that strong pre-training does not always accelerate downstream optimization even when tasks are well-aligned.

Original authors: Gibbs Nwemadji, Bruno Loureiro, Jean Barbier

Published 2026-05-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gibbs Nwemadji, Bruno Loureiro, Jean Barbier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: When "Too Much Practice" Backfires

Imagine you are teaching a student to play a specific song on the piano. You have two options:

Start from scratch: The student knows nothing, so you teach them the song from the very first note.
Use a pre-trained student: You hire a student who has already practiced many other songs for years. You expect them to pick up your new song quickly because they already know how to play the piano.

Usually, we assume the second option is better. The paper argues that sometimes, it's actually worse. If the student has practiced their previous songs too perfectly, they might get "stuck" in their old habits and struggle to learn the new song, even if the new song is very similar.

The authors call this phenomenon "Catastrophic Overtraining." They show mathematically that if a model is pre-trained too strongly on a source task, it can actually slow down the process of fine-tuning it for a new task.

The Setup: The "LoRA" Shortcut

To understand why this happens, we need to look at how modern AI learns new tasks. Instead of retraining the whole brain of the AI (which is expensive and slow), researchers use a method called LoRA (Low-Rank Adaptation).

The Analogy:
Imagine the AI is a massive library of books (the pre-trained model).

Full Fine-Tuning: Rewriting every single book in the library to fit the new topic. (Too expensive).
LoRA: You keep all the original books exactly as they are. Instead, you add a small, sticky note pad (a low-rank update) to the front of the books. You only write on these sticky notes to adjust the library for the new topic.

The paper studies what happens when you try to write on these sticky notes when the books underneath are already very "set in their ways."

The Experiment: The Teacher and the Student

The authors created a simplified mathematical world to test this. They used a "Teacher-Student" setup:

The Teacher: A perfect model that knows the answer to a specific problem.
The Student: A model that starts with a "hint" from the Teacher (the pre-trained weights) but needs to learn the specific details.

They simulated the learning process using SGD (Stochastic Gradient Descent), which is like the student taking one step at a time, looking at one example, and trying to correct their mistake.

The Two Phases of Learning

The paper identifies two distinct phases the student goes through:

The Search Phase (The "Lost in the Fog" Phase):
At the beginning, the student is confused. They have a hint (the pre-trained weights), but they haven't figured out exactly how to use it. They are wandering around, trying to find the right direction. This is the hardest part.
- The Paper's Finding: If the hint is too strong (the pre-training was very intense), the student gets "over-confident" in the wrong direction. They get stuck in this fog for a very long time, taking thousands of extra steps just to realize they need to turn around.
The Descent Phase (The "Rolling Downhill" Phase):
Once the student finds the right direction, they zoom toward the solution very quickly.
- The Paper's Finding: This part is fast and easy. The problem is entirely in the first phase.

The Surprising Twist: Stronger isn't Always Faster

The authors tested different types of "activation functions" (the mathematical rules the AI uses to make decisions, like ReLU or Sigmoid).

The Linear Case: Imagine the student is trying to learn a straight line. If the pre-training hint is 90% perfect, the student actually learns slower than if the hint was only 50% perfect. Why? Because the strong hint creates a "flat" landscape where the student feels no pressure to move. They are stuck on a plateau.
The "Singular" Case: For some complex rules (like certain polynomial functions), the paper found a "trap." If the pre-training hint hits a specific sweet spot (e.g., 32.5% alignment), the student gets completely stuck. They cannot escape the search phase at all, no matter how long they train. It's like a car stuck in a hole where the engine revs but the wheels don't turn.

The Solution: Changing the Labels

The paper also found a clever trick to fix this. If the student gets stuck, you can change the "labels" (the answers) you give them during the early stages of training.

The Analogy:
Imagine the student is trying to learn a song, but the sheet music is too complex, and they keep getting stuck on the first measure.

The Trick: You tell them, "Don't worry about the exact notes yet. Just clap the rhythm." (This is "label squaring").
The Result: Once they get the rhythm (escape the search phase), you give them the real sheet music back, and they can finish the song quickly.

Mathematically, squaring the labels changes the shape of the "learning landscape," smoothing out the traps and allowing the student to escape the stuck phase.

Real-World Proof

The authors didn't just do math; they tested this on real AI models (Vision Transformers) using image datasets like EMNIST (handwritten letters) and FashionMNIST (clothing images).

The Result:
They took models that had been pre-trained for a long time (high source accuracy) and tried to fine-tune them on a new task.

Observation: The models that were "most trained" on the old task actually performed worse on the new task after fine-tuning compared to models that were stopped earlier in their pre-training.
Conclusion: The "perfect" pre-trained model was too rigid to adapt quickly.

Summary

Intuition: We think more pre-training = better fine-tuning.
Reality: Too much pre-training can make the model "stubborn," causing it to get stuck in a slow learning phase.
Key Insight: There is a trade-off. A model that is too good at the old task might be too slow to learn the new one.
Fix: Sometimes, changing how you present the data (like squaring labels) can help the model break out of this stuck state.

The paper provides a mathematical map showing exactly when and why this happens, proving that in the world of AI, sometimes less pre-training leads to faster adaptation.

Technical Summary: "When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models"

Problem Statement
While pre-training on source tasks is widely believed to facilitate fine-tuning on similar downstream problems, this work challenges the naive intuition that stronger pre-training always accelerates downstream adaptation. Specifically, the authors investigate whether excessive pre-training can computationally hinder the optimization process during Low-Rank Adaptation (LoRA) fine-tuning. The core question is how the initial alignment between pre-trained weights and the downstream task, combined with the non-linearity of the target function, shapes the learning dynamics and convergence rates of LoRA.

Methodology
The authors employ a mathematically tractable yet non-trivial framework based on single-index models trained under one-pass Stochastic Gradient Descent (SGD).

Teacher-Student Setup: The target function is a single-index model $f^\star(x) = \phi(\omega^\star \cdot x)$ , where $\omega^\star$ is an unknown unit vector. The student model mimics LoRA by updating a pre-trained direction $\tilde{\omega}$ with a rank-one correction $u\omega$ . The pre-trained weight is modeled as $\tilde{\omega} = \mu \omega^\star$ , where $\mu \in (0, 1)$ quantifies the strength of the pre-training signal (alignment).
Dynamics Analysis: The study focuses on the evolution of two order parameters: the scalar update magnitude $u$ and the overlap $m = \omega \cdot \omega^\star$ . By leveraging a summary statistics description, the high-dimensional dynamics are reduced to a system of coupled stochastic processes.
Hermite Expansion: The analysis utilizes a generalized Hermite expansion to characterize the population loss landscape. This allows the authors to derive the "information exponent" (IE) of the learning dynamics, which dictates the sample complexity and convergence speed.
Phases of Learning: The dynamics are analyzed in two distinct phases:
1. Correlated Search Phase: A regime where the model has finite initial correlation $\mu$ with the target but the order parameters $u$ and $m$ are small ( $\max(|u|, |m|) \ll \mu$ ). This phase corresponds to the time required to escape the initial "search" for the relevant direction.
2. Descent Phase: Once the parameters become comparable to the signal strength, the model enters a regime of exponential convergence to the global optimum.

Key Contributions

Quantification of the Pre-training Trade-off: The paper provides a precise characterization of the "escape time" ( $t_{exit}$ ) from the correlated search phase. It reveals a counter-intuitive trade-off: for many activation functions, stronger pre-training (higher $\mu$ ) can significantly slow down the escape from the search phase, thereby delaying convergence to the global optimum. This phenomenon is described as "catastrophic overtraining" in the context of LoRA.
Identification of Singular Regimes: The authors identify specific regimes where the escape from the search phase is provably impossible under original labels. For pure odd-degree Hermite activations, there exist specific values of pre-training alignment $\tilde{\mu}$ where the drift coefficient $A$ vanishes, causing the escape time to diverge.
Label Transformation as a Remedy: The paper demonstrates that label transformations, specifically label squaring, can alter the effective information exponent of the loss landscape. This transformation can remove singularities and restore efficient escape from the search phase, effectively smoothing the optimization landscape during the early stages of learning.
Empirical Validation: The theoretical findings are validated through controlled experiments on Vision Transformers (ViT) using LoRA. Experiments on transfer tasks (e.g., EMNIST $\to$ FashionMNIST, CIFAR-10 $\to$ SVHN) confirm that later pre-training checkpoints, despite having higher source accuracy, can lead to worse downstream performance and slower convergence under constrained LoRA updates compared to earlier checkpoints.

Results

Linear and IE=1 Activations: For linear, erf, sigmoid, and ReLU activations, the characteristic escape time $\tau(\mu)$ generally increases monotonically with the pre-training strength $\mu$ . As $\mu \to 1$ , the escape time diverges, implying that highly aligned pre-trained weights can trap the model in a prolonged search phase.
Hermite Activations (IE $\ge$ 2):
- Odd Degrees: For pure odd Hermite activations, the paper proves the existence of at least one singular alignment $\tilde{\mu} \in (0, 1)$ where the drift coefficient $A=0$ , causing the dynamics to fail to escape the search phase despite non-zero initial correlation.
- Even Degrees: For even Hermite activations, the presence of a non-zero mean in the activation function allows for finite escape times even near singularities, though the dynamics exhibit a distinct plateau behavior where the mean component is learned before higher-order features.
Effect of Label Squaring: Applying label squaring transforms the effective IE to at most 2. This eliminates the singular behavior observed in odd Hermite activations and reverses the trend for linear/ReLU activations, where stronger pre-training then leads to faster escape times.
Transformer Experiments: In real-world transfer tasks, later pre-training checkpoints (higher source accuracy) consistently resulted in slower downstream adaptation and lower final accuracy under LoRA, corroborating the theoretical prediction that excessive pre-training can hinder fine-tuning optimization.

Significance and Claims
The paper claims to provide a unified theoretical picture of how pre-training strength and task difficulty jointly shape the dynamics and limitations of LoRA fine-tuning. It challenges the assumption that pre-trained representations are always beneficial, showing that they can induce a prolonged search phase that computationally slows down optimization.

The authors position this work as a principled explanation for recent empirical observations of "catastrophic overtraining" in Large Language Models (LLMs). By isolating the mechanism in a solvable single-index model, the paper offers a physics-inspired interpretation (analogous to a damped nonlinear oscillator) where strong pre-training can act as a dissipative force that hinders the system's ability to escape local plateaus. The work suggests that the choice of activation function and potential label preprocessing (like squaring) are critical factors in determining whether pre-training aids or hinders the fine-tuning process. The authors explicitly state that their theory provides a "tractable model" for understanding these dynamics, while acknowledging that a full theory of attention-based LoRA dynamics in deep networks remains a subject for future work.