🤖 machine learning

Rotation-Preserving Supervised Fine-Tuning

This paper proposes Rotation-Preserving Supervised Fine-Tuning (RPSFT), an efficient method that penalizes changes in the projected top- $k$ singular-vector blocks of pretrained weights to improve out-of-domain generalization and preserve pretrained representations without the computational cost of Hessian or Fisher information.

Original authors: Hangzhan Jin, Tianwei Ni, Lu Li, Pierre-Luc Bacon, Mohammad Hamdaqa, Doina Precup

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Hangzhan Jin, Tianwei Ni, Lu Li, Pierre-Luc Bacon, Mohammad Hamdaqa, Doina Precup

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Specialized" Student

Imagine a Large Language Model (LLM) as a brilliant student who has read almost every book in the library during their "pre-training" phase. They are great at general knowledge, safety, and reasoning about the world.

Now, you want to teach this student to be a math expert. You give them a stack of math problems to study (this is called Supervised Fine-Tuning or SFT).

The Good News: The student gets really good at math.
The Bad News: In the process of memorizing math formulas, the student starts to forget how to write a polite email, how to answer safety questions, or how to reason about general topics. They have become so specialized in math that they've lost their "generalist" edge.

In the paper, the authors call this "forgetting." They discovered that when the model learns math, the internal "compass" of its brain (specifically, the most important directions in its mathematical structure) starts to rotate wildly. It's like the student's brain physically twists to fit the new math, and in doing so, it breaks the old connections that held their general knowledge together.

The Solution: RPSFT (The "Anchored" Study Method)

The authors propose a new way to study called Rotation-Preserving Supervised Fine-Tuning (RPSFT).

Think of the model's brain as a giant, complex 3D sculpture made of thousands of interlocking gears.

The "Dominant" Gears: Some of these gears are huge and do 90% of the heavy lifting for the model's general intelligence.
The "Free" Gears: Other smaller gears handle specific details.

Standard Training (SFT): When you train the model normally, you let all the gears spin and shift however they need to solve the math problems. The big gears rotate so much that the whole sculpture loses its original shape.

RPSFT Training: The authors say, "Let's put a lock on the big, important gears."

They identify the most important "gears" (the top singular vectors) that hold the model's general knowledge.
They add a special rule to the training: "You can learn math and twist the small, free gears as much as you want, BUT you are not allowed to rotate the big, locked gears."
If the training tries to twist those big gears, the system pushes back, like a spring trying to snap them back to their original position.

Why This Works (The "Fisher" Connection)

The paper explains that these "big gears" are actually the same parts of the brain that are most sensitive to forgetting. The authors found a clever shortcut: instead of doing expensive, complex calculations to find exactly which parts are sensitive (which would take too much computer power), they just looked at the biggest gears (the top singular vectors).

It turns out that protecting these biggest gears is a perfect, cheap proxy for protecting the model's memory. It's like saying, "If I protect the foundation of the house, I don't need to check every single brick to know the house won't fall down."

The Results: Best of Both Worlds

The authors tested this on different models (like Llama and Qwen) using math data. Here is what happened:

Math Skills: The RPSFT models learned math just as well as the standard models.
General Skills: Unlike the standard models, the RPSFT models didn't forget their general knowledge, safety, or reasoning skills. They stayed "balanced."
Future Training: Because the RPSFT model kept its brain structure more stable, it was also a better starting point for the next step of training (Reinforcement Learning), leading to even better final results.

Summary Analogy

Imagine you are remodeling a house to add a new gym (the math task).

Standard SFT: You knock down walls and move the foundation to make room for the gym. Now you have a great gym, but your kitchen and living room are ruined.
RPSFT: You build the gym by rearranging the furniture and adding new rooms, but you anchor the foundation and load-bearing walls so they cannot move. You get a great gym, and your kitchen and living room stay exactly as they were.

The paper claims that by simply "anchoring" the most important parts of the model's brain during math training, we can teach it new skills without making it forget who it is.

Technical Summary: Rotation-Preserving Supervised Fine-Tuning (RPSFT)

Problem Statement

Supervised Fine-Tuning (SFT) is a standard post-training stage for Large Language Models (LLMs) used to adapt models to specific tasks. However, SFT often leads to a degradation in Out-of-Domain (OOD) generalization, a phenomenon linked to "catastrophic forgetting." Prior research suggests this degradation is caused by excessive geometric drift, specifically the rotation of dominant singular subspaces within pretrained weight matrices. While these dominant directions are associated with high-variance and high-curvature structures critical for general capabilities, identifying the specific loss-sensitive directions using Hessian or Fisher information is computationally prohibitive at the scale of modern LLMs.

Methodology: Rotation-Preserving SFT (RPSFT)

The authors propose Rotation-Preserving Supervised Fine-Tuning (RPSFT), a regularization method designed to limit unnecessary geometric drift while retaining task adaptation.

Core Mechanism

RPSFT operates on the hypothesis that the dominant singular subspaces of pretrained weights overlap significantly with Fisher-sensitive directions (directions where changes cause large increases in loss for retained capabilities). Instead of computing the expensive Fisher matrix, RPSFT uses the pretrained Singular Value Decomposition (SVD) as a structural proxy.

For each selected weight matrix $W_0$ in the pretrained model, the method:

Precomputes the truncated SVD: $W_0 = U_0 \Sigma_0 V_0^\top$ .
Caches the top- $k$ left and right singular vector bases, $U_0^{(k)}$ and $V_0^{(k)}$ .
Adds a regularization term to the standard SFT loss function:
$\mathcal{L}_{RPSFT} = \mathcal{L}_{SFT} + \lambda \sum_{\ell \in \mathcal{M}} \| (U_{0, \ell}^{(k)})^\top (W_\ell - W_{0, \ell}) V_{0, \ell}^{(k)} \|_F^2$
where $\lambda$ is the regularization strength and $\mathcal{M}$ is the set of selected layers.

Key Design Choices

Projected Anchor: The penalty specifically targets the $k \times k$ block of the weight matrix projected onto the pretrained singular bases. This "anchors" the dominant subspace, preventing it from rotating.
Complementary Freedom: Unlike full weight-space anchoring (e.g., L2 regularization around the base model), RPSFT leaves the complementary directions (orthogonal to the top- $k$ singular vectors) free to adapt. This allows the model to learn task-specific features without being overly constrained.
Computational Efficiency: The method requires storing only the top- $k$ singular vectors and the reference block. The computational overhead is linear in the protected rank $k$ and negligible compared to the full parameter update, making it feasible for full-parameter fine-tuning.

Key Contributions

Efficient Proxy for Fisher Sensitivity: The paper demonstrates that the dominant pretrained singular subspaces serve as a low-rank structural proxy for Fisher-sensitive directions. Empirical diagnostics show that a small top- $k$ block (e.g., rank 768) captures a significant portion (approx. 20%) of the Fisher-projected gradient energy, validating the use of SVD as a cheap alternative to Fisher estimation.
A Simple Regularizer for SFT: RPSFT integrates directly into standard SFT pipelines without requiring additional data, task boundaries, or complex architectural changes. It functions as a soft constraint rather than a hard projection or freezing mechanism.
Improved ID/OOD Trade-off: Across multiple model families (Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-3B) and sizes, RPSFT consistently improves the trade-off between In-Domain (ID) task performance and OOD generalization compared to standard SFT and strong baselines like Dynamic Fine-Tuning (DFT) and Importance-Weighted SFT (IW).
Stronger RL Initializations: Models fine-tuned with RPSFT provide superior initializations for downstream Reinforcement Learning Fine-Tuning (RLFT) using DAPO. They achieve higher or competitive final RL performance, suggesting that preserving the pretrained geometry facilitates better subsequent optimization.

Experimental Results

The authors evaluated RPSFT on math reasoning data (OpenR1-Math) and tested performance on:

In-Domain (ID): Math benchmarks (AIME, AMC, MATH-500, etc.).
Out-of-Domain (OOD): General reasoning, safety, and knowledge benchmarks (GPQA, MMLU-Pro, TruthfulQA, etc.).

Key Findings:

Performance: RPSFT achieved the best tuned ID averages and pass rates across all model sizes. Crucially, it reduced OOD forgetting relative to vanilla SFT. For instance, on Llama-8B, RPSFT improved OOD performance over the base model, whereas standard SFT degraded it. On Qwen models, RPSFT mitigated the OOD degradation seen in standard SFT without sacrificing ID gains.
Geometric Stability: Analysis of weight-space rotation showed that RPSFT significantly reduces the rotation of dominant left-singular subspaces compared to SFT, DFT, and IW.
Representation Preservation: Hidden-state drift analysis (measuring centroid shifts in PCA space) revealed that RPSFT keeps tuned model representations closer to the base model than other methods, particularly DFT which exhibited the largest drift.
Hyperparameter Sensitivity: The method is robust to intermediate protected ranks ( $k$ ). The authors found that $k=768$ (approx. 5% of the matrix rank) is a practical default that captures sufficient Fisher energy while allowing adaptation. Overly large ranks (approaching full-rank L2 anchoring) were found to hinder task adaptation.

Significance and Claims

The paper claims that RPSFT offers a practical and theoretically grounded approach to mitigating the forgetting-adaptation trade-off in LLM post-training. By explicitly controlling the rotation of dominant pretrained subspaces, RPSFT stabilizes the "general-purpose" capabilities encoded during pretraining while permitting necessary task-specific updates.

The authors emphasize that their work provides empirical and theoretical evidence that:

Preserving dominant pretrained subspaces reduces geometric drift.
This preservation protects hidden-state representations from catastrophic forgetting.
Such stability leads to better initializations for downstream RL, suggesting that the benefits of RPSFT extend beyond the SFT stage.

The paper concludes that RPSFT is a simple, effective regularizer that can be deployed in standard fine-tuning pipelines to enhance the robustness of LLMs against OOD degradation. Future work is suggested to explore layer-selective regularization and application to larger model scales and non-mathematical datasets.