💬 NLP

LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure

This paper proposes LP-SFT, a supervised fine-tuning method that preserves the pretrained model's inherent multimodal entropy structure among non-target tokens via a locally normalized KL divergence, thereby improving downstream performance while mitigating the degradation of pre-existing capabilities and sampling diversity.

Original authors: Yueyang Wang, Baolong Bi, Shuo Lu, Jingyuan Zhang, Jiajun Shi

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yueyang Wang, Baolong Bi, Shuo Lu, Jingyuan Zhang, Jiajun Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant, well-read robot how to follow instructions. This robot has already read almost everything on the internet, learning the rhythm of language, the logic of math, and the structure of code. This initial learning phase is called "pretraining." Now, you want to teach it a specific job, like solving math problems or writing computer code. You show it examples of the right answers, and the robot adjusts its brain to match those examples. This process is called "Supervised Fine-Tuning" (SFT).

However, there's a catch. When the robot learns to be perfect at the new job, it sometimes forgets how to be a good general conversationalist. It becomes so obsessed with the single "correct" answer you showed it that it forgets all the other interesting, valid ways a sentence could end. It's like a student who memorizes one specific answer to a math problem so well that they can't solve a slightly different version of it, or a writer who only knows one way to start a story. The big question in this field is: How do we teach the robot the new job without making it forget its old, rich knowledge?

This is exactly what the paper "LP-SFT" tackles. The authors, a team of researchers from top universities, noticed that the standard way of teaching these robots (called "cross-entropy") is a bit too blunt. It tells the robot, "Only look at this one word I'm showing you; ignore everything else." The problem is, the robot's original brain actually holds a map of many possible "good" next words, not just one. By forcing the robot to focus only on the single target word, we accidentally erase the map of all the other possibilities, leading to a loss of creativity and a phenomenon known as "catastrophic forgetting," where the robot forgets skills it used to have.

The researchers discovered something fascinating about how these robots think. They found that the robot's brain doesn't just pick one word; it often hovers between a few equally good options, creating a "multimodal entropy structure." Think of it like a fork in the road. Sometimes, the robot sees two or three paths that are all equally valid. The standard training method forces the robot to pick just one path and burn the others, even if the other paths were perfectly reasonable.

To fix this, the team proposed a new method called LP-SFT (Local-Preserving Supervised Fine-Tuning). Instead of just screaming "Pick this word!" at the robot, LP-SFT says, "Pick this word, but please don't forget the other three or four words that were also pretty good."

Here is how it works, using a simple analogy: Imagine you are training a dog to fetch a specific ball. The standard method is to only reward the dog when it brings back that exact ball and ignore everything else. Eventually, the dog might stop looking for other toys entirely. LP-SFT is like saying, "Bring me the red ball (the target), but while you're at it, remember that the blue ball and the green ball were also fun to fetch." The method creates a small "safe zone" of the top 10 most likely words the robot could have chosen. It removes the one word you are teaching it (so it doesn't get confused) and then gently nudges the robot to keep the relative popularity of the remaining nine words the same as it was before.

The results of this approach are quite promising. When the researchers tested LP-SFT on different models (like Qwen and Llama) and tasks (math, coding, and general chat), they found that it did a better job than the old methods. The robots trained with LP-SFT were not only good at the specific task they were taught (getting high scores on math and coding tests) but also kept their ability to be diverse and creative. They didn't forget their general knowledge as much as the others did. In fact, the method achieved the best balance between getting the single answer right and keeping a wide variety of possible solutions available.

The paper suggests that by respecting the robot's original "map" of possibilities, we can teach it new skills without breaking its brain. It's a reminder that sometimes, to teach a machine to be perfect, you have to let it keep a little bit of its uncertainty. The researchers show that this local preservation helps stop the robot from collapsing into a boring, one-track mind, keeping it smart, flexible, and ready for whatever comes next.

Technical Summary: LP-SFT (Local-Preserving Supervised Fine-Tuning)

1. Problem Statement

Supervised Fine-Tuning (SFT) is the standard paradigm for adapting pretrained language models (PLMs) to downstream tasks. However, standard SFT, which relies on token-level cross-entropy (CE) loss, often induces capability degradation in two dimensions:

Catastrophic Forgetting: Over-specialization to the fine-tuning distribution leads to the loss of out-of-domain capabilities.
Reduced Generation Diversity: The model suppresses alternative valid solutions, collapsing the distribution toward a single token and reducing sampling-accessible diversity (measured by pass@k).

The authors argue that this degradation stems from a mismatch between the CE objective and the rich distributional knowledge encoded in PLMs. While PLMs learn broad capabilities resulting in next-token distributions with substantial information beyond the single observed target token, standard CE ignores the remaining preference structure. Repeatedly suppressing plausible alternatives during fine-tuning distorts the pretrained structural integrity.

2. Key Observations: Multimodal Entropy Structure

Before proposing a solution, the authors analyze the entropy structure of next-token predictions in base models.

Multimodal Entropy Peaks: Using Shannon ( $H_1$ ) and Rényi-2 ( $H_2$ ) entropies, the authors observe that the entropy distribution exhibits distinct multimodal peaks near $\ln k$ for integers $k$ .
Plateau-k States: These peaks correspond to "entropy-compressed states" where uncertainty is spread over $k$ $k$ similarly plausible next-token candidates.
- Low-order plateaus ( $k=1\text{--}5$ ): Often correspond to logical or semantic branching where multiple valid continuations exist.
- High-order plateaus ( $k \approx 10$ ): Often correspond to structural ambiguity (e.g., random digits, IDs) where the model distributes probability across a set of characters.
Implication: The base model intrinsically encodes rich distributional knowledge. Standard CE treats all tokens uniformly, unnecessarily distorting this heterogeneous structure by forcing the model to ignore these structured alternatives.

3. Methodology: LP-SFT

To address this, the authors propose LP-SFT (Local-Preserving Supervised Fine-Tuning). The method uses the frozen base model as a local structural reference to preserve the relative preferences among non-target alternatives without conflicting with the supervised learning signal.

Core Components

Local Top-K Preservation Set:
- At each token position $t$ , the method constructs a candidate set $S_t$ from the top- $K_{max}$ tokens of the frozen base model distribution ( $q_t$ ).
- By default, $K_{max} = 10$ , chosen to cover both low-order branching and the prominent high-order plateau observed in the analysis.
Target-Token Removal:
- To avoid conflicting with the primary cross-entropy objective (which optimizes the supervised target token $y_t$ ), $y_t$ is explicitly removed from the preservation set.
- The preservation set becomes $A_t = S_t \setminus \{y_t\}$ .
Local KL Normalization:
- Instead of applying a global KL divergence over the full vocabulary (which can be computationally expensive and overly restrictive), LP-SFT applies a locally normalized KL divergence over the set $A_t$ .
- The probabilities of the base model ( $q_t$ ) and the trainable model ( $p_t$ ) are normalized only within $A_t$ to obtain $\hat{q}_t$ and $\hat{p}_t$ .
- The loss is defined as: $L_{LP} = D_{KL}(\hat{q}_t \parallel \hat{p}_t)$ .
- Rationale: This preserves the relative preference structure among the remaining non-label alternatives without artificially constraining their absolute probability mass or the total mass assigned to the set.

Training Objective

The final objective combines the standard cross-entropy loss for the target token and the local preservation loss:
$L_{LP-SFT}(\theta) = \sum_{t=1}^{T} \left[ -\ln p_t(y_t) + \mu \cdot L_{LP}^t \right]$
where $\mu$ controls the preservation strength.

Implementation Efficiency

Unlike methods requiring a forward pass of the base model at every training step, LP-SFT employs a two-stage implementation:

Offline Precomputation: The base model processes the entire training dataset once to cache the top- $K_{max}$ logits.
Fine-Tuning: The training loop only requires the forward pass of the trainable model, using the cached logits to compute the local preservation loss. This significantly reduces computational overhead compared to full-vocabulary anchoring methods.

4. Experimental Results

The authors evaluated LP-SFT on three model families (Qwen3-4B, Llama-3.1-8B, Qwen3-14B) across mixed-domain and single-domain settings (Math, Code, General Instruction).

Performance vs. Vanilla SFT: LP-SFT consistently outperforms standard CE and recent SFT-enhancement baselines (DFT, EAFT, GEM, ASFT) in terms of overall average performance.
Pass@1 vs. Pass@k Trade-off:
- Standard CE often improves pass@1 (single-sample accuracy) at the cost of pass@k (diversity).
- Methods like DFT improve pass@1 but cause significant drops in pass@k.
- LP-SFT achieves the best balance: It improves pass@1 while maintaining or improving pass@k, indicating that it retains sampling-accessible diversity without sacrificing accuracy.
Catastrophic Forgetting: In mixed-domain fine-tuning, LP-SFT significantly mitigates capability degradation. For example, on the MMLU benchmark, LP-SFT retained performance nearly on par with the frozen base model, whereas standard CE suffered severe drops.
Ablation Studies: The method is relatively insensitive to the hyperparameter $\mu$ . Both target-token removal and local KL normalization were found to be critical for stability and performance.

5. Significance and Claims

The paper claims that LP-SFT offers a novel approach to supervised fine-tuning by explicitly protecting the inherent entropy structure of pretrained models.

Mechanism: By decoupling target-token learning from the preservation of local structural preferences, LP-SFT prevents the "overwriting" of the base model's structured uncertainty.
Outcome: This approach helps mitigate catastrophic forgetting and preserves generation diversity (sampling-accessible solutions) while maintaining or improving single-sample accuracy.
Efficiency: The method provides these benefits with modest computational overhead compared to full-vocabulary anchoring techniques, making it a practical alternative for preserving pretrained capabilities during domain adaptation.

The authors conclude that local preservation is an effective strategy to balance downstream adaptation with the retention of general capabilities, suggesting that the rich distributional knowledge learned during pretraining should be preserved rather than discarded during fine-tuning.