Technical Summary: Probability-Entropy Calibration for Adaptive Fine-tuning

1. Problem Statement

Token-level reweighting is a established mechanism for controlling supervised fine-tuning (SFT) of Large Language Models (LLMs). However, existing approaches rely on one-dimensional indicators that fail to capture the full complexity of token importance:

• Prob-Dominant Weighting: Relies on ground-truth probability ($p_t$) to reflect downstream alignment. While effective for task alignment, it fails to distinguish between tokens that are genuinely difficult and those that are intrinsically ambiguous or replaceable (e.g., synonyms like "essentially" vs. "basically"). This can lead to over-penalizing semantically flexible positions, potentially harming linguistic flexibility.
• Entropy-Dominant Weighting: Relies on predictive entropy ($H_t$) to reflect intrinsic uncertainty induced by pre-training priors. While useful for identifying uncertainty, high-entropy tokens often include noisy filler words or irrelevant content. Up-weighting these tokens can amplify uninformative gradients and degrade alignment with the downstream task.

The core problem is that treating $p_t$ (alignment) and $H_t$ (intrinsic uncertainty) in isolation leads to misidentifying learning-critical tokens. Noisy tokens may be up-weighted due to high entropy, while replaceable tokens may be over-penalized due to low probability, even if they are not critical errors. The paper argues for a calibrated signal that contextualizes likelihood by intrinsic uncertainty.

2. Methodology: RankTuner

The authors propose RankTuner, a framework that introduces a Probability-Entropy Calibration signal via a rank-based discretization. The core idea is to compare the realized rank of the ground-truth token against its expected rank under the model's prediction distribution.

2.1. Theoretical Foundations

The method bridges the gap between probability and entropy using the Guessing Problem framework (Massey, 1994):

• Rank ($R_t$): The position of the ground-truth token when candidates are sorted by decreasing probability.
• Expected Rank ($E[R_t]$): The expected number of guesses required to find the target token, calculated as $\sum \hat{i} \cdot p_{\hat{i}}$.
• Bounds: The authors establish tight theoretical bounds linking these rank-based quantities to probability and entropy:
  • $R_t \leq 1/p_t$ (Rank-Probability Bound)
  • $E[R_t] \geq s(H_t)$ (Expected Rank-Entropy Bound, where $s(H_t)$ is a function derived from entropy).

2.2. The Relative Rank Indicator ($I_t$)

The method defines a Relative Rank Indicator ($I_t$) to measure token-level performance relative to uncertainty:
$$I_t = g(f(R_t) - f(E[R_t]))$$
Where $f(x) = \frac{1}{\log_2(x+1)}$ and $g(x) = 2^x$.

• Interpretation: $I_t$ decreases as the realized rank $R_t$ increases (lower accuracy) but increases as the expected rank $E[R_t]$ increases (higher difficulty).
• Calibration: This indicator effectively separates tokens into regimes:
  • Prioritized: Low $R_t$ and low $E[R_t]$ (easy, correct).
  • Negligible: High $R_t$ and high $E[R_t]$ (hard, but expected to be hard).
  • Noise Region: High $H_t$ but low $p_t$ (noisy/replaceable), which receives low emphasis.

2.3. Relative Scale ($S_t$) and Loss Reweighting

For fine-tuning, the method uses the inverse of the indicator, termed the Relative Scale ($S_t = I_t^{-1}$), as a token-wise weighting coefficient.

• Mechanism: Tokens where the model underperforms relative to the expected difficulty (low $I_t$) receive higher weights. Tokens that are intrinsically uncertain or replaceable (high $E[R_t]$) are down-weighted to avoid over-optimization.
• Implementation: The scale is approximated using the theoretical bounds to avoid explicit rank computation over the full vocabulary during training:
  $$S_t \approx (p_t \cdot s(H_t))^{-K(\xi_t)}$$
  Where $K(\xi_t)$ is a coefficient derived via the Cauchy Mean Value Theorem, typically approximated around 0.5 for reasoning tokens. The final loss is $\mathcal{L} = \sum -w_t \cdot S_t \log p_t$.

3. Key Contributions

1. Analysis of One-Dimensional Limitations: The paper demonstrates that Prob-Dominant and Entropy-Dominant weighting schemes fail to distinguish between "noisy" tokens (high entropy, low value) and "replaceable" tokens (low probability, high ambiguity), leading to suboptimal fine-tuning signals.
2. Rank-Based Calibration: Introduction of the Relative Rank Indicator, which unifies probability and entropy by comparing the ground-truth rank against the expected rank. This provides an uncertainty-aware adaptive reweighting scheme.
3. Empirical Validation: Extensive experiments across multiple backbones (Qwen2.5-Math, Qwen3, Llama-3.1) and reasoning benchmarks show consistent improvements over baselines.

4. Experimental Results

The authors evaluated RankTuner on mathematical reasoning and code generation tasks:

• Mathematical Reasoning: On benchmarks including MATH-OAI, Minerva Math, OlympiadBench, AIME 2024, and AMC 2023, RankTuner consistently outperformed standard SFT and strong baselines (OverTone, DFT, TALR, EAFT).
  • Notable gains were observed in Pass@1 (single-sample accuracy) and Pass@16 (multi-sample coverage), indicating improved accuracy without sacrificing diversity.
  • On difficult benchmarks like AIME 2024, where other methods suffered degradation, RankTuner maintained robust performance.
• Out-of-Distribution (OOD) Generalization: On ARC-C and GPQA benchmarks, RankTuner achieved the best Pass@1 scores, demonstrating robust transfer beyond the training distribution.
• Code Generation: On HumanEval and HumanEval+, RankTuner showed consistent improvements, particularly preserving performance on smaller models (Qwen2.5-Coder-3B) where standard SFT often degraded capabilities.
• Ablation Studies: Removing either the probability or entropy component degraded performance, confirming the complementary nature of both signals. Dynamic adaptation of the exponent $K(\xi_t)$ was shown to be superior to fixed exponents.

5. Significance and Claims

The paper claims that Probability-Entropy Calibration offers a simple yet effective principle for adaptive fine-tuning.

• Core Claim: By discretizing probability and entropy into a commensurate rank-based pair, RankTuner focuses updates on "genuinely under-learned" critical tokens while down-weighting noisy or replaceable positions.
• Robustness: The method avoids the pitfalls of over-penalizing semantically flexible words or amplifying noise, leading to more stable training and better generalization.
• Scope: The authors explicitly state that their empirical evidence is strongest in the supervised fine-tuning (SFT) setting. While they suggest the principle could extend to RL-style post-training (e.g., PPO/GRPO) and multimodal tasks, they do not validate these extensions in this work.
• Generalizability: The approach relies only on token-level predictive statistics, suggesting potential applicability to LLM personalization, embodied AI, and generative information extraction, though these remain directions for future work.

The paper concludes that this perspective is promising for generalizing to broader tasks and training paradigms, providing a mechanism to improve accuracy without collapsing solution diversity.