Technical Summary: Digit Entropy Loss for Numerical Learning of Large Language Models

1. Problem Statement

Large Language Models (LLMs) have demonstrated significant capabilities in mathematical reasoning and code generation, yet accurate numerical prediction remains a fundamental challenge. The standard training objective for LLMs, Maximum Likelihood Estimation (MLE), is sub-optimal for number prediction. MLE's zero-avoiding property forces models to assign non-trivial probabilities to observed tokens, leading to issues such as text degeneration and data void overestimation.

Recent penalty-driven approaches, specifically Number Token Loss (NTL) and Discretized Distance Loss (DIST2Loss), attempt to address this by introducing an inductive bias of numerical distance. However, these methods exhibit critical limitations:

• Over-sharpening and Over-flattening: NTL aggressively suppresses non-target token probabilities to near zero, creating over-sharpened distributions, while DIST2Loss enforces a smooth distribution that results in over-flattened, uncertain digit predictions.
• Reliance on Heuristic Distance: Both methods rely on manually defined distance terms (e.g., $L_1$ distance between digits). Analysis reveals that LLMs learn entangled representations where digits function as both numerical values and symbolic strings (e.g., dates, IDs). Consequently, manually imposed geometric priors often contradict the semantic representations learned from massive corpora.
• Lack of Holistic Awareness: Existing methods often operate within a single-digit optimization paradigm or treat decimals as separate integers, failing to capture the continuous nature of floating-point numbers and global numerical structures.

2. Methodology: Digit Entropy Loss (DEL)

The authors propose Digit Entropy Loss (DEL), a reformulation of the conventional unsupervised entropy optimization tailored for auto-regressive numerical learning. The methodology is built upon three key design principles that shift the paradigm from "criterion-distance" to "digit entropy optimization."

A. From Criterion-Distance to Entropy Optimization

Existing numerical learning methods follow a criterion-distance formulation (Eq. 4 in the paper):
$$\mathcal{L}_N = \mathcal{H}(p, g) + \alpha \sum \mathcal{C}(\hat{p}) \cdot \mathcal{D}(V_d, g_t)$$
where $\mathcal{C}$ is the criterion term and $\mathcal{D}$ is the distance term. The authors argue that the distance term $\mathcal{D}$ is sub-optimal because it imposes a geometric prior that may not align with the LLM's learned semantic structure of digits.

B. Three Key Designs of DEL

1. Supervised Entropy via Digit Conditional Probability and BCE:
   Instead of minimizing entropy in an unsupervised manner (which allows trivial one-hot solutions), DEL introduces a supervised criterion. It leverages the product of the digit conditional probability ($\hat{p}$) and Binary Cross-Entropy (BCE).
   $$\mathcal{L}_{DEL} = \mathcal{H}(p, g) + \alpha \sum_{g_t \in V_d} \hat{p}(\cdot|x_{<t}) \cdot \text{BCE}(\bar{p}(\cdot|x_{<t}), y_t)$$
   Here, $\bar{p}$ is the sigmoid-based binary classification probability, and $y_t$ is the one-hot label. This formulation guides the entropy optimization into a supervised manner, ensuring the probability mass is concentrated on the target digits without the "over-sharpening" caused by NTL's constant penalty.

2. Deprecation of the Distance Term:
   The authors explicitly remove the distance term ($\mathcal{D}$) from the loss function. Empirical analysis of digit token embeddings (via t-SNE) shows that the learned similarity structure of digits in LLMs does not strictly follow ideal numerical distances (e.g., '0' may be closer to '8' than '1' in embedding space). By removing the distance term, DEL avoids injecting contradictory geometric priors, allowing the model to rely on its learned semantic representations.

3. Generalization to Floating-Point Optimization:
   To handle multi-digit integers and decimals holistically, DEL introduces a floating-point number optimization strategy. Unlike previous methods that treat decimals as separate integers (causing discontinuous penalties), DEL applies a continuous penalty function $u(t)$:

   • Linear penalty for integer places ($i \ge 0$).
   • Exponential decay for decimal places ($i < 0$).
     This ensures a smooth transition between integer and decimal components, enabling accurate prediction of floating-point numbers.

3. Key Contributions

• Theoretical Analysis: The paper provides an in-depth analysis revealing that state-of-the-art numerical learning methods (NTL, DIST2Loss) share a "criterion-distance" paradigm that relies on heuristic distance terms, which are often misaligned with LLM semantic representations.
• Novel Loss Function: The proposal of Digit Entropy Loss (DEL), which reformulates numerical learning by combining digit conditional probability with binary cross-entropy, effectively guiding entropy minimization in a supervised manner.
• Holistic Number Awareness: The extension of numerical learning from single-digit optimization to floating-point number optimization, addressing the discontinuity issues in decimal prediction found in prior works.
• Empirical Validation: Comprehensive experiments demonstrating that DEL outperforms MLE, MixCE, EMO, NTL, and DIST2Loss across multiple architectures and benchmarks.

4. Experimental Results

The authors evaluated DEL on seven mathematical reasoning benchmarks (GSM8K, MATH, SVAMP, SimulEq, AQuA, SAT, MMLU) using four representative LLMs: CodeLlama, Mistral, DeepSeek-Math, and Qwen-2.5.

• Overall Accuracy: DEL consistently achieved the highest mean accuracy (mACC) across all model architectures and benchmarks. For instance, on the Qwen-2.5-7B model, DEL achieved an mACC of 70.6, outperforming the baseline MLE (68.3) and the next best method, DIST2Loss (69.2).
• Numerical Distance: DEL significantly reduced the numerical distance (Mean Absolute Error) between predicted and ground-truth numbers, demonstrating more precise number generation even without explicit distance terms.
• Entropy Analysis: Unlike DIST2Loss, which increased digital entropy (uncertainty), and NTL, which over-sharpened distributions, DEL maintained a "mild" optimization of number uncertainty, balancing determinacy with flexibility.
• Ablation Studies: Experiments confirmed that removing the distance term and incorporating floating-point optimization were critical for performance gains. Specifically, adding the distance term to DEL degraded performance, validating the decision to deprecate it.

5. Significance and Claims

The paper claims that Digit Entropy Loss (DEL) represents a significant step forward in numerical learning for LLMs by addressing the limitations of heuristic distance-based methods.

• Shift in Paradigm: The work argues that numerical learning should move away from manually defined geometric priors (distance terms) toward entropy-based optimization that respects the semantic representations learned by the model.
• Reliability: By achieving highly deterministic generation for numerical tokens, DEL serves as a prerequisite for reliable LLM deployment in downstream tasks requiring quantitative analysis and logical deduction.
• Modest Scope: The authors acknowledge that while DEL improves number generation accuracy, it operates at the token level and does not yet model complex mathematical operations (e.g., addition, multiplication) or global numerical semantics jointly. They position DEL as a foundational improvement for numerical generation rather than a complete solution for all aspects of mathematical reasoning.

In summary, the paper presents DEL as a robust, theoretically grounded alternative to existing numerical loss functions, demonstrating that removing artificial distance constraints and optimizing for supervised entropy yields superior performance in mathematical reasoning and code generation tasks.