🤖 machine learning

The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

This paper demonstrates that Full Fine-Tuning causes severe negative transfer in sub-1B language models, establishing Parameter-Efficient Fine-Tuning (PEFT) as a critical stability requirement for mathematical reasoning tasks on edge devices.

Original authors: Rahul Nair, Chun Tao

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rahul Nair, Chun Tao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a tiny, incredibly smart pocket-sized robot (a "Small Language Model" or SLM) that has already read a library of books to learn how to speak, reason, and solve problems. Now, you want to teach it a specific new trick, like solving math homework.

This paper is a warning label and a user manual for that process. It says: "If your robot is too small, trying to reprogram its entire brain to learn the new trick will actually break its ability to think at all."

Here is the breakdown of their findings using simple analogies:

1. The "Full Brain Surgery" vs. The "Sticky Note"

The researchers tested two ways to teach these tiny robots:

Full Fine-Tuning (The "Full Brain Surgery"): This is like taking the robot apart and rewiring every single connection in its brain to fit the new math task.
- The Result: For tiny robots (under 300 million "connections" or parameters), this is a disaster. It's like trying to rearrange the furniture in a shoebox; you end up knocking everything over. The robot forgets how to speak, starts repeating itself in loops, or makes basic math errors (like saying 15 + 7 = 30). It actually performs worse than if you had just asked it to guess without any training.
PEFT / LoRA (The "Sticky Note"): This method freezes the robot's original brain and just adds a tiny, special "adapter" or a sticky note on top of it to handle the math.
- The Result: The robot keeps its original smarts and personality intact while learning the new trick. It works much better and doesn't crash.

2. The "Stability Cliff"

The authors discovered a specific size limit, which they call a "Stability Cliff."

Below 500 Million parameters: The robot is so packed with essential knowledge that there is no "extra room" to learn new things without overwriting old, important stuff. If you try a "Full Brain Surgery" here, the robot falls off the cliff and breaks.
Above 1 Billion parameters: The robot is big enough to have some "spare room." Here, you can do the full surgery, and it works fine.

3. The "Intruder" Dimensions

Why does the surgery fail? The paper suggests that when you try to rewire a tiny robot completely, the math forces the robot to explore "Intruder Dimensions."

Analogy: Imagine the robot's knowledge is a safe, flat highway. When you do a full rewrite, the robot gets pushed off the highway into a jagged, rocky canyon (high-loss regions). It gets lost in the rocks and can't find its way back.
The Fix: The "Sticky Note" method (PEFT) keeps the robot on the highway, only nudging it slightly to take the new turn.

4. The "Safety Bulldozer"

One of the scariest findings involves "aligned" robots (ones trained to be safe and helpful).

The Problem: When they tried to teach a safe robot (Qwen2.5) math using the "Full Surgery," the robot completely forgot how to be safe. It became useless on safety tests.
The Metaphor: It's like hiring a "Bulldozer" to paint a mural. The bulldozer (Full FT) is so powerful and indiscriminate that it smashes the delicate safety features along with the old paint.
The Solution: The "Sticky Note" (PEFT) is like a careful artist who paints over the safety features without destroying them.

5. The Speed vs. Safety Paradox

You might think, "If Full Surgery updates everything, shouldn't it be faster?"

Surprise: On powerful computers, the "Full Surgery" is actually twice as fast to run than the "Sticky Note" method.
Why use the slower method? Because the "Sticky Note" is the only thing that stops the robot from breaking. It's a trade-off: You accept a slower training time to get a robot that actually works.
Bonus: The "Sticky Note" method is so light that you can train these tiny robots on a regular laptop or gaming PC, whereas the "Full Surgery" requires massive, expensive supercomputers just to fit in memory.

The Bottom Line Recommendations

The paper gives three clear rules for anyone trying to teach these tiny AI models:

If the model is tiny (<500M): Never use "Full Surgery." It will ruin the model. Always use the "Sticky Note" (PEFT/LoRA/DoRA) method.
If the model is aligned (safe): Always use "Sticky Notes." Full Surgery will wipe out its safety training.
If the model is big (>1B): You can use "Full Surgery" if you want, because the model is big enough to handle it without breaking.

In short: For small AI models, less is more. Don't try to rebuild the whole engine; just add a new part, or the whole car will fall apart.

Technical Summary: The Fine-Tuning Trap in Sub-1B Mathematical Reasoning

Problem Statement

The deployment of Small Language Models (SLMs) on edge devices necessitates efficient adaptation strategies that do not degrade general capabilities. However, this paper identifies a critical vulnerability in models with fewer than 300 million parameters: Full Fine-Tuning (Full FT) often induces "negative transfer," actively harming performance and dropping accuracy below zero-shot baselines. This phenomenon, termed the "Fine-Tuning Trap," suggests that for sub-1B models, standard fine-tuning is not merely an efficiency choice but a stability requirement. The authors argue that SLMs operate near "capacity saturation," where dense pre-trained knowledge leaves no redundant parameters to absorb new task-specific gradients without overwriting core linguistic and reasoning capabilities.

Methodology

The study employs a systematic empirical framework to benchmark five decoder-only architectures across a parameter range of 135M to 2B.

Model Selection: The evaluation spans three regimes:
- Tiny Regime (<500M): SmolLM2-135M and SmolLM2-360M.
- Transition Regime (~500M): Qwen2.5-0.5B (highly aligned).
- Reference Regime (>1B): OLMo-1B and Gemma-2B.
Tasks: The primary focus is mathematical reasoning, utilizing a stratified subset of 10,000 examples from OrcaMath for training. Evaluation covers GSM8K (in-distribution reasoning), SVAMP (out-of-distribution robustness), and MATH (high-difficulty problems).
Fine-Tuning Strategies: The study compares:
- Full FT: Updating all parameters.
- PEFT (Parameter-Efficient Fine-Tuning): Specifically LoRA (Low-Rank Adaptation) and DoRA (Weight-Decomposed LoRA).
Training Protocol: Experiments were conducted on H100 nodes. The authors utilized distinct hyperparameters for stability, employing a higher learning rate for PEFT ( $2 \times 10^{-4}$ ) compared to Full FT ( $2 \times 10^{-5}$ ) to account for the zero-initialization of adapters versus the need to preserve priors in Full FT.
Analysis: The authors analyzed training dynamics via gradient norms, loss trajectories, and Hessian spectral analysis to understand optimization stability.

Key Contributions

Identification of a Stability Threshold: The paper establishes a "Phase Transition" at approximately 500M parameters. Below this threshold, Full FT consistently fails, often resulting in catastrophic forgetting. Above 1B parameters, Full FT regains efficacy as models possess sufficient "slack" capacity.
Theoretical Formalization of Instability: The authors model the failure of Full FT in tiny models through three lenses:
- Intruder Dimensions: Full FT unconstrained updates traverse orthogonal subspaces ( $W_{pre}^\perp$ ) characterized by high-curvature directions (large Hessian eigenvalues), leading to instability. PEFT constrains updates to the pre-trained manifold.
- Architectural Conditioning: Modern SLMs often use "Deep and Thin" architectures. This increases the condition number of the input-output Jacobian exponentially, causing extreme gradient variance that Full FT cannot balance, whereas LoRA decouples updates to mitigate this.
- Bounding Forgetting: PEFT methods strictly bound the displacement from the pre-trained weights ( $\delta_{PEFT}$ ), ensuring the model remains within a "trust region," unlike the unbounded displacement of Full FT.
Comparative Dynamics of PEFT: The study provides granular analysis showing that while LoRA and DoRA both stabilize training, their strengths differ. DoRA (which decouples magnitude and direction) excels in complex reasoning tasks (GSM8K), while LoRA dominates pattern matching (OrcaMath).
Hardware and Economic Trade-offs: The paper redefines the value of PEFT for SLMs. On high-end hardware (H100), Full FT is actually 2x faster in wall-clock time due to optimized dense matrix kernels, whereas PEFT incurs kernel overhead. However, PEFT is essential for accessibility, reducing VRAM requirements from ~16GB (Full FT) to <4GB, enabling training on consumer hardware (e.g., RTX 4090) where Full FT would exceed memory limits.

Empirical Results

The Optimization Cliff: For the 135M model (SmolLM2), Full FT loss plateaued early at ~0.82 with unstable gradient norms, while PEFT methods converged to ~0.74. Full FT accuracy on OrcaMath dropped to 4.80% (below the 6.00% zero-shot baseline).
Alignment Collapse: On the aligned Qwen2.5-0.5B model, Full FT achieved high in-distribution scores but resulted in 0.00% accuracy on the SVAMP robustness benchmark. This indicates that Full FT overwrote safety and robustness vectors with the narrow task distribution. PEFT methods preserved these capabilities.
Qualitative Failures: Full FT on tiny models led to "repetition loops" (infinite calculation steps) and "loss of arithmetic logic" (nonsensical intermediate math), suggesting damage to attention heads responsible for discourse structure.
Reference Regime Success: For models >1B (OLMo, Gemma), Full FT outperformed PEFT (e.g., Gemma-2B Full FT: 19.8% GSM8K vs. LoRA: 18.5%), confirming that sufficient capacity allows for full-rank updates without collapse.

Significance and Recommendations

The paper concludes that the "Stability-Capacity Trade-off" dictates a strict operational boundary for SLMs:

The <500M Danger Zone: For models like SmolLM2, Full FT is actively harmful. Practitioners should default to PEFT to prevent reasoning collapse.
The Alignment Mandate: For aligned models (e.g., Qwen2.5), Full FT destroys robustness and safety alignment. PEFT is mandatory to preserve the "safety subspace" of pre-trained weights, acting as a "safety sandbox."
The Return to Scale: Full FT becomes viable again only when parameter counts exceed 1B, where intrinsic capacity can absorb full-rank updates.

The authors recommend defaulting to PEFT for all aligned sub-1B models and cautioning against Full FT for any architecture smaller than 500M parameters. They emphasize that while Full FT may be faster on clusters, PEFT is the critical enabler for democratizing SLM training on edge devices and maintaining the safety and robustness required for regulated industries.