Technical Summary: A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design

Problem Statement

Supervised Fine-Tuning (SFT) is a critical stage in adapting Large Language Models (LLMs) to specific tasks by imitating expert demonstrations. However, standard SFT relies on a rigid one-hot target distribution, where the model is trained to maximize the probability of the observed token $y_t$ while assigning zero probability to all other tokens.

The authors argue that this formulation is suboptimal for several reasons:

1. Non-Uniqueness: In realistic datasets, a single prompt often admits multiple valid reasoning paths, phrasings, or intermediate steps. A single observed token is rarely the only correct continuation.
2. Noise and Misalignment: Observed tokens may be noisy or misaligned with the rich knowledge prior encoded in the pre-trained model.
3. Overconfidence: Strictly fitting to a one-hot target can amplify noise, induce overconfidence, interfere with the pre-trained prior, and degrade generalization.

Existing works attempt to mitigate these issues through token-level reweighting (e.g., DFT, ProFit) or distributional regularization (e.g., label smoothing, KL constraints). However, these methods are often presented as disparate algorithmic choices, lacking a unified theoretical framework to explain their connections or guide the construction of better objectives.

Methodology: The Q-Target Framework

The paper reinterprets SFT not as a loss minimization problem, but as a target distribution design problem. The core insight is that any token-level loss function implicitly drives the model to match a specific target distribution $Q_t$.

The Q-Target Formulation

The authors propose the Q-target framework, which decomposes the training supervision into two explicit design choices:
$$Q_t = \gamma_t \delta_{y_t} + (1 - \gamma_t) \tilde{\pi}_t$$
Where:

• $\delta_{y_t}$ is the one-hot distribution of the observed token.
• $\gamma_t \in [0, 1]$ represents the trust in the observed token. It controls how much probability mass is assigned to the label.
• $\tilde{\pi}_t$ is the residual distribution over plausible alternatives, receiving the remaining probability mass $(1 - \gamma_t)$.

This framework unifies existing SFT variants:

• Token-Weighted Variants (e.g., DFT, ProFit): Primarily adjust $\gamma_t$ based on model confidence or token quality, but implicitly set the residual $\tilde{\pi}_t$ to the model's own prior (self-matching), providing no corrective signal for alternatives.
• Distributional Variants (e.g., Label Smoothing, KL-constraints): Primarily specify $\tilde{\pi}_t$ (e.g., uniform, reference model) but often use a fixed $\gamma_t$.

TARGET-SFT: A Concrete Instantiation

Building on this framework, the authors propose TARGET-SFT, which explicitly designs both components of the Q-target:

1. Adaptive Trust ($\gamma_t$): Instead of fixed weights, $\gamma_t$ is derived from the model's own predictive probability $p_y = \pi_\theta(y_t | x_t)$.

   • If the model is confident ($p_y \approx 1$), $\gamma_t \approx 1$, and the objective approaches standard SFT.
   • If the model is uncertain ($p_y \approx 0$), $\gamma_t \approx 0$, weakening the strict imitation of the potentially noisy label.
   • This is motivated by modeling the "trust" in a token as a latent variable with a Beta distribution, where the posterior mean approximates the model probability.

2. Teacher-Guided Residual ($\tilde{\pi}_t$): Rather than defaulting the residual mass to the student's own prior (which offers no new information), TARGET-SFT allocates it to a teacher-guided distribution.

   • The residual is constructed via KL-regularized reward shaping: $\tilde{\pi}_t \propto \pi_\theta^{1-\eta} \pi_T^{\eta}$, where $\pi_T$ is a teacher distribution (e.g., an instruction-tuned model) and $\eta$ controls the intensity of the teacher signal.
   • This allows the model to learn from "plausible alternatives" suggested by the teacher when the observed label is uncertain, acting as a fallback supervision mechanism.

The final objective is the cross-entropy loss between the model distribution and the constructed $Q_t^{TARGET}$:
$$L_{TARGET} = \sum_t \left[ p_y \cdot CE(\delta_{y_t}, \pi_\theta) + (1 - p_y) \cdot CE(\tilde{\pi}_t^{guided}, \pi_\theta) \right]$$

Key Contributions

1. Target-Distribution Perspective: The paper introduces a fundamental shift in viewing SFT, arguing that the induced target distribution is the primary object of design, rather than the loss function itself.
2. Q-Target Framework: A unifying lens that decomposes SFT objectives into two explicit choices: label trust ($\gamma_t$) and residual allocation ($\tilde{\pi}_t$). This framework explains how existing variants (reweighting, regularization, distillation) are implicit choices within this space.
3. TARGET-SFT Algorithm: A concrete method that adaptively balances label imitation and teacher-guided residual supervision. It avoids overfitting to uncertain labels while leveraging teacher knowledge to guide the model toward plausible alternatives.

Experimental Results

The authors evaluated TARGET-SFT across 10 dataset-model settings, covering mathematical reasoning (NuminaMath-CoT, OpenR1-15k) and medical reasoning (m23k) with models ranging from 1.5B to 8B parameters (Qwen2.5, Qwen3, LLaMA).

• Performance: TARGET-SFT consistently achieved the highest Average@16 accuracy across all settings, outperforming standard SFT, probability-weighted SFT (p-loss), and knowledge distillation baselines.
• Mathematical Reasoning: Standard SFT often yielded modest gains or even degraded performance (e.g., on Qwen3-1.7B) due to rigid one-hot fitting. TARGET-SFT significantly improved upon p-loss, demonstrating that explicitly designing the residual distribution is superior to merely down-weighting the label.
• Medical Reasoning: While standard SFT performed well in some medical cases (suggesting stricter imitation is sometimes valid), TARGET-SFT still achieved the best average performance. This indicates that the teacher-guided residual branch provides value even when simple reweighting is less effective.
• Ablation Studies:
  • Varying the teacher signal intensity ($\eta$) showed that adaptive weighting is robust across different intensities.
  • Replacing the adaptive weight $(1-\gamma_t)$ with a fixed constant $c$ degraded performance, confirming the necessity of uncertainty-dependent weighting.
  • Distillation baselines were highly sensitive to the mixture weight $c$ and failed to match TARGET-SFT's performance, suggesting that using a teacher as a fixed full target is suboptimal compared to adaptive fallback supervision.

Significance and Claims

The paper claims that its formulation reveals a more fundamental design principle for SFT training. By shifting the focus from the loss function to the induced target distribution, the authors:

• Unify disparate SFT variants under a single theoretical framework.
• Expose a broader search space for SFT objectives, allowing for explicit control over how much to imitate data versus how to preserve priors or incorporate external guidance.
• Demonstrate that effective SFT arises from carefully designing both the imitation strength and the allocation of remaining probability mass, rather than relying on rigid one-hot targets or static regularization.

The authors conclude that TARGET-SFT offers a novel, more complete perspective on SFT, opening new avenues for future objective design that better balance dataset imitation, prior preservation, and alternative supervision.