Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives

1. Problem Statement

The paper addresses a fundamental limitation in standard Supervised Fine-Tuning (SFT) for Large Language Models (LLMs): the use of the Negative Log-Likelihood (NLL) objective. NLL applies uniform token-level weighting, treating all tokens equally regardless of the model's confidence or the data quality. This rigidity leads to a "plasticity–stability dilemma" with two primary failure modes:

1. Amplification of Noise (Instability): NLL assigns maximal gradients to low-probability targets. If these targets represent noisy supervision or "confident conflicts" (where the pre-trained model is confident but the label disagrees), NLL forces large parameter updates that can lead to catastrophic forgetting of robust priors.
2. Inefficient Sharpening (Plasticity): As the model becomes confident in correct tokens, NLL gradients decay linearly with the error ($1-p$). This results in diminishing returns, failing to aggressively "sharpen" the probability distribution for tokens the model already knows well, thereby slowing convergence.

Existing solutions, such as offline token filtering or static reweighting (e.g., linear scaling based on probability), fail to dynamically balance the need to learn new knowledge (exploration) while preserving existing knowledge (exploitation).

2. Methodology

The authors propose a unified theoretical framework and a practical, parameter-free objective called Dynamic Entropy Fine-Tuning (DEFT).

A. Theoretical Unification: The Deformed-Log Family

The authors unify various token-level objectives within a family based on the Tsallis $q$-logarithm. They define a general loss function $L_\alpha(p) = \frac{1-p^\alpha}{\alpha}$, where $\alpha$ is a focus index.

• Gradient Structure: The gradient for this loss decomposes into a Gate $\times$ Error structure:
  $$\nabla L_\alpha \propto \underbrace{p^\alpha}_{\text{Trust Gate}} \times \underbrace{(1-p)}_{\text{Prediction Error}}$$
• Role of $\alpha$:
  • $\alpha \to 0$: Recovers NLL (Gate $\approx 1$). Maximizes coverage for uncertain tokens but risks noise overfitting.
  • $\alpha \to 1$: Recovers Linear Probability Loss (Gate $\approx p$). Suppresses low-confidence tokens, focusing on sharpening high-confidence regions.

B. The Optimization-Entropy Duality

The paper establishes a duality between the loss index $\alpha$ and the Tsallis entropy order. Minimizing $L_\alpha$ is equivalent to minimizing Tsallis entropy of order $q = 1+\alpha$.

• Low $\alpha$ targets Shannon entropy (information acquisition/coverage).
• High $\alpha$ targets Order-2 Tsallis entropy (collision entropy/sharpening).

C. The State-Dependent Trajectory (Cayley Transform)

To resolve the tension between coverage and sharpening, the authors argue that $\alpha$ should not be static but state-dependent, varying with the model's confidence $p$.

• Geometric Derivation: They derive a trajectory $\alpha(p)$ using the Cayley transform on the Fisher-Rao radius ($z = \sqrt{1-p}$).
• Boundary Conditions:
  • When $p \to 0$ (uncertain): $\alpha \to 0$ (Open gate, NLL-like behavior).
  • When $p \to 1$ (confident): $\alpha \to 1$ (Closed gate, Sharpening behavior).
• Formula: The optimal trajectory is given by:
  $$\alpha^*(p) = \frac{1 - \sqrt{1-p}}{1 + \sqrt{1-p}} = \tanh\left(\frac{-\ln(1-p)}{4}\right)$$
  This interprets the focus index as a function of normalized error surprisal.

D. DEFT: The Practical Objective

Since the ideal trajectory depends on the specific target token probability $p$ (which is unknown during training for the true distribution), the authors propose DEFT as a parameter-free surrogate.

• Mechanism: Instead of using the target token's probability $p$ to set $\alpha$, DEFT uses the concentration of the model's entire predictive distribution (Rényi-2 entropy).
• Focus Index:
  $$\alpha_{\text{DEFT}}(c) = \sum_{v \in V} P_\theta(v|c)^2$$
  • High Uncertainty (Diffuse distribution): $\alpha_{\text{DEFT}}$ is small $\to$ Gate opens $\to$ Prioritizes learning new concepts (Exploration).
  • High Confidence (Concentrated distribution): $\alpha_{\text{DEFT}}$ approaches 1 $\to$ Gate closes $\to$ Prioritizes sharpening known knowledge (Exploitation).
• Advantage: This allows the model to adaptively balance learning signals without expensive offline filtering or hyperparameter tuning.

3. Key Contributions

1. Unified Framework: A deformed-log family that explicitly reveals the "Gate $\times$ Error" structure of SFT objectives, unifying NLL, linear probability loss, and generalized cross-entropy.
2. State-Dependent Gating: A theoretical derivation of a dynamic trust gate using the Cayley transform, which smoothly interpolates between coverage and sharpening based on model confidence.
3. DEFT Algorithm: A practical, parameter-free method that uses distribution-level concentration (Rényi-2 entropy) to approximate the ideal state-dependent trajectory.
4. Theoretical Guarantees: Proof of the optimization-entropy duality and the geometric justification for the Cayley transform trajectory.

4. Experimental Results

The authors evaluated DEFT across seven model backbones (including LLaMA-3, Qwen2.5, DeepSeekMath) and three distinct regimes based on prior knowledge strength:

• Model-Strong (e.g., Math tasks where the model already knows the domain):
  • Standard NLL causes forgetting of robust priors due to noise.
  • DEFT and Cayley-Transform significantly outperform NLL and linear scaling ($-p$) by suppressing noisy gradients while sharpening correct predictions.
  • Result: DEFT achieved the highest average accuracy (e.g., 11.35% vs 6.49% for NLL on LLaMA-3.1-8B).
• Model-Weak (e.g., Novel reasoning tasks):
  • Linear scaling ($-p$) fails because it suppresses gradients for low-probability tokens needed for new learning.
  • DEFT adapts to behave like NLL (open gate) to acquire new knowledge, outperforming all baselines significantly.
• Model-Intermediate & Mixed:
  • DEFT consistently achieves the best balance, outperforming static objectives in both general reasoning and domain-specific benchmarks.
• Generalization (OOD):
  • In cross-domain transfer (Math $\to$ Medical), DEFT demonstrated superior robustness, preserving pre-trained capabilities better than NLL, which suffered from catastrophic forgetting.

5. Significance

This work fundamentally shifts the perspective on SFT from a static loss function to a dynamic, state-aware process.

• Solves the Plasticity-Stability Dilemma: It provides a mathematically grounded method to automatically decide when to learn new information and when to refine existing knowledge, without manual intervention.
• Parameter-Free Efficiency: Unlike methods requiring offline data filtering or complex hyperparameter tuning, DEFT is a drop-in replacement for the loss function that adapts automatically.
• Robustness: It offers a solution to the "confident conflict" problem, preventing models from overfitting to noisy labels in high-confidence regions, which is critical for deploying reliable LLMs in safety-critical or specialized domains.

In summary, DEFT demonstrates that gradients must "earn their influence" by being modulated by the model's current state of uncertainty, leading to more efficient, stable, and robust fine-tuning.