Technical Summary: The Entropy of Floating-Point Numbers

Problem Statement

Floating-point numbers are the standard discrete representation for continuous variables in modern computing, particularly in deep learning algorithms and hardware. While the relationship between differential entropy and the discrete entropy of uniformly quantized random variables is well-established (specifically in the high-resolution limit), there is no analogous analytic framework for non-uniform quantization, such as floating-point representation.

The paper addresses the fundamental question: What is the output entropy of a random variable quantized onto a floating-point format? Existing methods rely on numerical evaluation of exact discrete entropy, which is computationally intensive. The authors seek to derive a predictive analytic approximation for this entropy, along with rigorous bounds on the approximation error.

Methodology

The authors develop a multi-step analytic framework to approximate the discrete entropy $H(X_{fp})$ of a floating-point quantized random variable $X_{fp}$:

1. Exact Relation Derivation: Extending the work of Kostina [9] on uniform quantization, the authors derive an exact identity relating the discrete entropy of non-uniform quantization to the differential entropy $h(X)$ of the underlying continuous variable $X$. This identity includes a correction term involving the Kullback-Leibler (KL) divergence between the true density $f_X$ and a piecewise-constant approximation $g$:
   $$H(X_{fp}) = h(X) - E[\log \Delta(X)] + D(f_X || g)$$
   where $\Delta(X)$ represents the variable bin size of the floating-point quantizer.

2. Approximation Strategy:

   • Neglecting KL Divergence: The authors propose neglecting the non-negative KL divergence term $D(f_X || g)$ as a first-order approximation. They analyze the conditions under which this term is small, showing it depends on the "spikiness" of the density relative to the bin size.
   • Smoothing the Bin Size: The exact floating-point bin size function $\Delta(x)$ is a step function. The authors introduce a smoothed approximation $\Delta_s(x) \approx \frac{|x|}{\sqrt{2}} 2^{1-p}$, which allows for closed-form integration. This extends the domain of the bin size function from the "granular region" (where representable values exist) to the entire real line $\mathbb{R}$.

3. Error Analysis:

   • Smoothing Error: The authors prove that the error introduced by smoothing the bin size and extending the domain is bounded by $d/2$ (where $d$ is the dimensionality) plus a term $\epsilon$ representing the probability mass in overflow and underflow regions.
   • Scale Invariance: A key theoretical step is proving that the approximate entropy is invariant under scaling of the underlying random variable (i.e., $H((aX)_{fp}) \approx H(X_{fp})$), provided overflow/underflow probabilities are negligible. This contrasts with differential entropy, which scales with $\log|a|$.

Key Contributions

1. Analytic Approximation Formula

The paper provides a closed-form approximation for the floating-point entropy of a random variable $X$ with precision $p$:
$$\tilde{H}_s(X_{fp}) = p - \frac{1}{2} - E[\log(|X| f_X(X))]$$
This formula effectively separates the entropy into a constant term dependent on precision ($p - 1/2$) and a distribution-dependent term termed the "differential floating-point entropy."

2. Closed-Form Expressions for Common Distributions

Using the derived approximation, the authors provide explicit closed-form expressions for the entropy of numerous common distributions (e.g., Gaussian, Uniform, Gamma, Laplace, Lognormal, Pareto, Beta, Student's t). Notably, for univariate distributions, the scale parameters cancel out in the approximation, confirming the scale-invariance property.

3. Rigorous Error Bounds

The paper establishes bounds on the error of the approximation:
$$|H(X_{fp}) - \tilde{H}_s(X_{fp})| \leq \frac{1}{2} + \epsilon + D(f_X || g)$$
where $\epsilon$ accounts for overflow/underflow contributions and $D(f_X || g)$ accounts for the deviation of the density from uniformity within bins. The authors demonstrate that for smooth distributions and sufficiently large exponent ranges, the error is minimal.

4. Scale-Invariance Property

The authors prove that, unlike differential entropy, the approximate floating-point entropy is scale-invariant for scalar random variables. This implies that the entropy of a floating-point representation depends primarily on the shape of the distribution and the precision $p$, rather than the magnitude of the values, as long as the values fit within the representable range.

Results

• Accuracy: The analytic approximations closely match numerically evaluated exact discrete entropies for various distributions (Gaussian, Uniform, etc.) across a wide range of scale parameters. The agreement holds well within the underflow and overflow boundaries defined by the exponent bits.
• Precision vs. Exponent Bits: The results show that the approximation remains accurate even with low precision ($p$) as long as the number of exponent bits ($E$) is sufficient to keep the probability of overflow and underflow low.
• Scale Invariance Verification: Empirical plots confirm that the entropy of a quantized Gaussian variable remains constant as the standard deviation $\sigma$ varies, until $\sigma$ approaches the limits of the floating-point exponent range.

Significance and Claims

The paper claims to uncover a "different quantity" (the differential floating-point entropy) that links differential entropy to discrete entropy in the context of floating-point quantization, distinct from the uniform quantization case.

The significance of this work lies in providing a predictive analytic tool for estimating the entropy of floating-point representations without resorting to expensive numerical integration. This is particularly relevant for:

• Variable-rate quantizer design: Understanding the entropy-distortion tradeoff for floating-point formats.
• Deep Learning: Analyzing the information content and compression limits of neural network weights and activations represented in floating-point formats (e.g., FP8, BF16).
• Hardware Design: Informing the tradeoffs between precision, dynamic range, and memory usage in novel floating-point formats.

The authors maintain a modest tone, emphasizing that their results are approximations with proven bounds, valid under the assumption that the probability of overflow and underflow is low and the distribution is sufficiently smooth relative to the quantization bins. They do not propose new hardware or specific algorithmic changes but rather provide the theoretical foundation to analyze existing floating-point representations.