📊 statistics

A Structural Characterization of Entropy Functionals

This paper introduces a measure-theoretic framework to structurally characterize entropy functionals by establishing a four-level hierarchy based on admissibility conditions, which resolves Rényi's axiomatization question and identifies specific criteria for generating new admissible entropies and divergences, including the Shannon and Rényi families.

Original authors: Daniel Lazarev

Published 2026-08-17

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Lazarev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve the mystery of "information." In the world of statistics and data science, information isn't just a vague idea; it's a measurable quantity, much like weight or temperature. To measure it, scientists use mathematical formulas called entropy. Think of entropy as a "surprise meter." If you flip a coin and it lands on heads, that's not very surprising, so the entropy is low. But if you flip a coin that is rigged to land on heads 99% of the time, and it suddenly lands on tails, that is a huge surprise, and the entropy is high.

For decades, scientists have had a toolbox full of different "surprise meters." The most famous one is Shannon entropy, which is like the standard ruler used in almost every school math class. But there are other rulers, too, like Rényi entropy and Tsallis entropy. These aren't just different sizes of the same ruler; they measure surprise in slightly different ways, sometimes giving different answers to the same question. The big problem is that picking which ruler to use has often been a matter of habit or convenience, like choosing a specific brand of pen just because your teacher uses it. There hasn't been a clear, structural reason to say, "You must use this one for this specific job."

This is where the paper by Daniel Lazarev comes in. It asks a fundamental question: Is there a set of basic rules that tells us which of these "surprise meters" are actually valid for measuring information, and which ones are just mathematical tricks? The paper doesn't just list the rulers; it builds a new framework to test them, revealing a hidden hierarchy that explains why the famous ones work and how to invent new, valid ones.

The Detective's New Rulebook

Daniel Lazarev's paper, "A Structural Characterization of Entropy Functionals," acts like a master key for the world of information theory. Instead of guessing which entropy formula is the "best," the author sets up a strict set of rules—like a constitution for data—to see which formulas pass the test.

The first and most important rule in this constitution is Structural Monotonicity. Imagine you have a map of a city (the "reference measure") and a specific route you are driving (the "input measure"). If your route is entirely contained within the city limits, your "surprise" about the route should never be higher than the surprise of the city itself. In simpler terms: if you are looking at a part of a picture, you shouldn't be more confused than if you were looking at the whole picture. If a formula breaks this rule, it's disqualified. It's like a thermometer that says it's hotter inside a cup of tea than it is in the boiling pot it came from; that thermometer is broken.

Once a formula passes this "part-of-the-whole" test, the paper introduces a second layer of testing: Generalized Means. When you combine two pieces of information (like merging two datasets), how do you average their "surprise" levels? Most people use the arithmetic mean (the standard average: add them up and divide by two). But Rényi, a famous mathematician, wondered: "What if we used a different kind of average, like a geometric mean or a power mean?"

Lazarev's paper proves that you can use these other averages, but only if the "generator" (the mathematical engine driving the average) follows a very specific shape. Think of the generator as the mold used to bake a cake. The paper shows that for the cake to rise properly (to be a valid entropy), the mold must be shaped in a way that is either strictly "convex" (curving outward like a bowl) or strictly "concave" (curving inward like a dome), depending on whether the mold is increasing or decreasing. If the mold is wobbly or flat, the cake collapses, and the entropy formula is invalid.

The Four-Level Hierarchy

The most exciting discovery in the paper is that these rules create a four-level hierarchy, like a ladder of strictness. As you climb up the ladder, the formulas become more specific and more rigid.

Level 1: The General Class. At the bottom, you have the most flexible formulas. These satisfy the basic "part-of-the-whole" rule and use a generalized average. This level includes a vast family of new, valid entropy formulas that nobody had fully categorized before. The paper shows you how to build them using simple math tricks, like integral transforms (which are like blending different flavors of information together).
Level 2: The Scale Fix. If you add a rule that says the "units" of surprise must add up in a simple way (like how 1 meter + 1 meter = 2 meters), you narrow the field down. This step fixes the "scale" of the entropy, making it behave more like a standard ruler.
Level 3: The Rényi Family. If you add a rule about how information behaves when you combine two independent systems (like flipping two separate coins), you land on the Rényi entropy family. This is the famous family that includes the standard Shannon entropy as a special case. The paper proves that Rényi entropy is the only family that fits this specific combination of rules.
Level 4: The Shannon Peak. At the very top, the most rigid level, is Shannon entropy. This is the "gold standard" we use in almost everything today. The paper shows that Shannon entropy is the only formula that satisfies an even stronger rule: that information must combine perfectly even inside a single system, not just between separate ones. It's the most restrictive, but also the most robust.

Why This Matters

This isn't just a game of mathematical classification. By mapping out this hierarchy, the paper solves a puzzle that Rényi himself posed in 1961. He asked, "Which of these weird averages can replace the standard average in our entropy formulas?" Lazarev's answer is a clear "Yes, but only these specific ones, and here is exactly why."

The paper also connects these entropy formulas to Csiszár f-divergences, which are tools used to measure how different two probability distributions are. The paper proves that if your entropy formula passes the structural tests, it automatically guarantees a property called Data Processing Inequality. In plain English, this means that if you process your data (like filtering a noisy signal or compressing a file), you can never create new information or surprise; you can only lose it or keep it the same. This is a fundamental law of information, and the paper shows that it holds true for a huge new class of formulas, not just the old ones.

New Tools for the Toolbox

Perhaps the most playful part of the paper is that it doesn't just explain the old formulas; it builds new ones. The author provides a "construction kit" for creating new, valid entropy formulas. For example, he shows how to use Laplace transforms (a type of mathematical averaging) to create an infinite family of new entropies. He even gives examples like "Arctangent entropy" and "Square root entropy," which behave differently than the standard ones but are just as mathematically sound.

These new formulas could be useful for specific types of data where the standard "surprise meter" isn't quite right. For instance, in robust statistics (where you want to ignore outliers or weird data points), these new formulas might offer a better way to estimate the truth without getting thrown off by a single bad data point.

The Bottom Line

Daniel Lazarev's paper doesn't declare one single entropy formula as the "winner" for all time. Instead, it provides a structural map. It tells us that the choice of entropy isn't arbitrary; it's a choice of which structural rules you want to follow. If you want the most flexible tool, you can choose from the broad family at the bottom. If you need the strict, reliable ruler used in almost every computer algorithm, you climb to the top to Shannon entropy.

The paper proves that the famous formulas we use today are not just lucky accidents or historical conventions. They are the inevitable result of following a specific set of logical rules. And best of all, it hands us the blueprint to build new, valid formulas whenever we need a different kind of ruler for a new kind of data. It turns the mystery of "which entropy to use" into a clear, logical journey.

Technical Summary: A Structural Characterization of Entropy Functionals

Problem Statement
Entropy functionals and their associated divergences are foundational to statistical inference, appearing in maximum entropy inference, minimum divergence estimation, and goodness-of-fit testing. However, the selection among various entropy families—such as Shannon, Rényi, Tsallis, and more general forms—is often driven by convention or analytic convenience rather than structural principle. The central question addressed is: what structural assumptions distinguish one entropy functional from another? Specifically, the paper seeks to resolve a question posed by Rényi (1961) regarding which generalized (quasi-arithmetic) means may replace the arithmetic mean in entropy axiomatizations while maintaining admissibility, and to characterize the resulting hierarchy of entropy functionals without assuming additivity as a starting point.

Methodology and Framework
The paper introduces a measure-theoretic framework for characterizing entropy functionals based on a minimal structural requirement: Structural Monotonicity (Postulate A). This postulate requires that for any pair of nonzero finite measures $\mu \ll \nu$ , the entropy of $\mu$ relative to $\nu$ must be bounded above by the entropy of $\nu$ relative to itself, with equality if and only if $\mu$ is proportional to $\nu$ . This condition reflects the information ordering underlying the comparison of statistical experiments.

The methodology proceeds by analyzing entropy functionals under Generalized Mean-Value Composition (Postulate 5'), where the entropy of a disjoint sum of measures is determined by a quasi-arithmetic mean (Kolmogorov-Nagumo mean) of the component entropies. The paper distinguishes between:

Atomic Additivity (Postulate 4at): A weak form of additivity fixing the entropy scale on single atoms.
External Additivity (Postulate 4): Additivity across independent product systems.
Internal Additivity (Postulate 4int): Additivity within a single system under the multiplication of densities.

The analysis utilizes the properties of generalized means, specifically the comparison of quasi-arithmetic means via the convexity/concavity of their generators, and connects these to Csiszár $f$ -divergences and data processing inequalities.

Key Contributions and Results

Admissibility Criterion for Generalized Means:
The paper establishes a necessary and sufficient condition for a continuous, strictly monotone generator $g$ to yield an admissible entropy functional satisfying Structural Monotonicity. A generator $g$ is admissible if and only if:
- If $g$ is increasing, the map $t \mapsto g(1/t)$ is strictly convex.
- If $g$ is decreasing, the map $t \mapsto g(1/t)$ is strictly concave.
  This criterion resolves Rényi's question by identifying exactly which generalized means are compatible with the structural monotonicity requirement. It is equivalent to the strict convexity of an associated Csiszár $f$ -divergence generator, ensuring data processing inequalities hold with exact equality conditions under Markov kernels.
Four-Level Hierarchy of Entropy Functionals:
The characterization yields a constructive four-level hierarchy, ordered by decreasing generality:
- Level 1: Generalized $(g, h)$ -Entropy. Satisfying Structural Monotonicity (Postulate A) and Generalized Mean-Value (Postulate 5'). Here, $g$ is the mean generator and $h$ is the entropy scale.
- Level 2: Generalized $g$ -Entropy. Satisfying Level 1 plus Atomic Additivity (Postulate 4at). This fixes the scale to $h(r) = -\log r$ , yielding $H_g(\mu) = -\log M_g(1/\mu(X) \cdot d\mu/d\nu)$ .
- Level 3: Rényi Entropy. Satisfying Level 2 plus External Additivity (Postulate 4). This restricts the generator $g$ to power-law forms ( $g(t) = t^{p-1}$ ), recovering the Rényi family for $p > 0$ .
- Level 4: Shannon Entropy. Satisfying Level 3 plus Internal Additivity (Postulate 4int), or satisfying External Additivity (Postulate 4) combined with the Arithmetic Mean-Value (Postulate 5). This uniquely selects the logarithmic generator ( $g(t) = \log t$ ).
Constructive Generation of New Functionals:
The framework is constructive, allowing for the generation of new admissible entropy and divergence families. The paper provides examples including:
- Sharma-Mittal entropy: Demonstrating the separation between the mean generator and the outer entropy scale.
- Arctangent entropy: Using $g(t) = \arctan t$ .
- Integral-transform entropies: A broad class derived from positive kernels (e.g., Laplace transforms), yielding infinite-dimensional families of admissible entropies.
- Tsallis Entropy: Identified as a member of the generalized $(g, h)$ -entropy class (Level 1) rather than the generalized $g$ -entropy class (Level 2), as it employs a $q$ -additive scale rather than the logarithmic scale, despite sharing the same power-law generator as Rényi entropy.

Significance and Claims
The paper claims to provide a "structural explanation" for the entropy functionals commonly used in statistics and a "principled mechanism" for generating new ones. By decoupling the constraints on the mean generator from the entropy scale and additivity assumptions, the work clarifies which features of classical formulas (Shannon, Rényi) are forced by structural monotonicity versus those imposed by specific additivity postulates.

The results offer a diagnostic tool: one can determine the admissibility of a proposed entropy functional by checking the convexity of its generator's reciprocal transformation. Furthermore, the paper demonstrates that the same power-law generators underpinning Rényi entropy and Cressie-Read statistics are singled out specifically by external additivity, while Shannon entropy is singled out by the stronger internal additivity. The framework thus unifies existing entropy families and extends the landscape of admissible statistical information measures beyond the classical power-law and logarithmic cases.