Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum

1. Problem Statement

Supervised Fine-Tuning (SFT) is the standard paradigm for post-training Large Language Models (LLMs) to elicit specific capabilities. However, SFT often suffers from limited generalization. The authors argue that this limitation stems not from the SFT paradigm itself, but from its default training objective: Negative Log Likelihood (NLL), defined as $-\log p$.

While NLL is theoretically optimal for training models from scratch on small-scale classification tasks, the authors posit that it violates optimality assumptions in the post-training regime because:

1. Pretrained Priors: LLMs already encode task-relevant priors from massive pretraining corpora.
2. Noisy Supervision: Post-training often involves long chains of thought (CoT) where intermediate tokens may be noisy or imperfect, yet NLL forces the model to replicate every token verbatim.

The central question is: Is NLL universally optimal for SFT, or does the effectiveness of a loss function depend on the model's existing capability and the nature of the task?

2. Methodology

A. Unified Probability-Based Objective Family

The authors generalize the training objective beyond NLL. They define a family of objectives parameterized by a differentiable, non-increasing function $f: [0, 1] \to \mathbb{R}$:
$$L_f(\theta) = \mathbb{E}_{(x, \tilde{y}) \sim \mathcal{T}} \left[ f(p_\theta(\tilde{y} | x)) \right]$$
They specifically analyze the parametric family:
$$f_\alpha(p) = \frac{1 - p^\alpha}{\alpha}$$

• As $\alpha \to 0$, $f_\alpha(p) \to -\log p$ (Standard NLL).
• When $\alpha = 1$, $f_\alpha(p) = 1 - p$ (Linear probability loss).
• When $\alpha > 1$, the function becomes concave (e.g., $-p^{10}$).

B. Theoretical Analysis: Gradient Dynamics

The paper analyzes the gradient of these objectives with respect to the model logits. They derive a weighting term $W_f(p)$ that determines how much learning signal a token contributes based on its predicted probability $p$:
$$W_f(p) = -f'(p) \cdot p(1-p)$$

• Convex Objectives (e.g., NLL, $\alpha < 1$): $W_f(p)$ is maximized at low probabilities ($p < 0.5$). These are Prior-Averse; they force the model to learn heavily from tokens it currently predicts as unlikely.
• Concave Objectives (e.g., $-p$, $\alpha \ge 1$): $W_f(p)$ is maximized at high probabilities ($p > 0.5$). These are Prior-Leaning; they reinforce tokens the model already predicts with high confidence, effectively downweighting low-probability "noise."

C. The Model-Capability Continuum

The authors introduce a critical dimension governing objective performance: the Model-Capability Continuum, defined by the strength of the base model's prior relative to the task.

1. Model-Strong (MS): The base model already has strong priors for the task (e.g., Math reasoning for models pretrained on math data). Predicted probabilities for correct tokens are high before SFT.
2. Model-Weak (MW): The base model has no relevant priors (e.g., novel puzzles or low-resource languages). Predicted probabilities are near random.
3. Model-Intermediate (MI): Partial priors exist (e.g., Medical reasoning).

3. Key Contributions

1. Discovery of the Continuum: The paper establishes that no single loss function is universally optimal. The performance of an objective reverses depending on whether the model is in the "Strong" or "Weak" regime.
2. Prior-Leaning vs. Prior-Averse:
   • In Model-Strong regimes: Prior-leaning objectives (like $-p$ or thresholded NLL) outperform NLL. They ignore low-probability tokens (which are likely noise in the supervision) and focus on refining high-confidence predictions.
   • In Model-Weak regimes: Prior-averse objectives (NLL) dominate. Since the model has no prior knowledge, it must learn from all tokens, including those it initially predicts as unlikely.
3. Theoretical Justification: Using gradient flow analysis, the authors prove sufficient conditions under which prior-leaning objectives achieve greater risk reduction in strong regimes, while NLL is superior in weak regimes.
4. Thresholding Mechanism: They demonstrate that simply thresholding NLL to exclude tokens with $p < 0.2$ mimics the behavior of prior-leaning objectives, offering a simple, effective alternative to complex loss functions.

4. Experimental Results

The authors conducted extensive experiments across 8 model backbones, 27 benchmarks, and 7 domains.

• Model-Strong (Math Reasoning):

  • On datasets like NuminaMath, the prior-leaning objective ($-p$) and thresholded NLL consistently outperformed standard NLL.
  • Result: Up to 16% absolute improvement in accuracy on benchmarks like AIME24 and AMC23 compared to standard NLL.
  • Mechanism: Standard NLL was hindered by trying to fit noisy, low-probability tokens.

• Model-Weak (Puzzles & Low-Resource Languages):

  • On synthetic "figfont" puzzles and low-resource language instruction tuning, standard NLL significantly outperformed $-p$.
  • Result: $-p$ failed to generalize because it reinforced the model's initial (incorrect) high-probability guesses, whereas NLL forced correction across the distribution.

• Model-Intermediate (Medical Reasoning):

  • In domains like medical QA, the performance gap between $-p$ and NLL narrowed, with neither consistently prevailing. This suggests a "neutral zone" where the choice of loss is less critical than data quality.

• General Instruction Tuning:

  • Experiments on general conversational alignment showed that as model scale increased (3B $\to$ 14B), the prior-leaning objective ($-p$) became increasingly effective, shifting the model from the MW to the MS regime.

5. Significance and Impact

• Paradigm Shift: The paper challenges the dogma that NLL is the default "best" loss for all SFT tasks. It argues that the optimal objective is context-dependent on the model's pretraining alignment with the task.
• Practical Guidance:
  • For strong models on familiar tasks (Math, Code), practitioners should use prior-leaning objectives (e.g., $-p$, $-p^{10}$, or thresholded NLL) to avoid overfitting to noisy supervision.
  • For weak models or novel domains, standard NLL remains the robust choice to ensure broad learning.
• Bridge to RL: The authors note that many Reinforcement Learning (RL) objectives implicitly act as prior-leaning updates (focusing on high-probability sequences). This work provides a supervised, off-policy framework to achieve similar benefits without the complexity of RL.
• Efficiency: The proposed improvements (like thresholding) are computationally cheap and easy to implement, offering immediate performance gains for SFT pipelines.

In conclusion, the paper provides a principled, theoretical, and empirical framework for adapting SFT objectives to the specific capability level of the model, moving beyond a "one-size-fits-all" approach to loss functions.