Learn your entropy from informative data: an axiom ensuring the consistent identification of generalized entropies

This paper introduces an axiom stating that entropic parameters cannot be inferred from uniform distributions, which uniquely selects Rényi entropy among generalized families and enables the consistent, data-driven estimation of these parameters while ensuring that the maximized log-likelihood always equals the negative Shannon entropy.

The Gist

Imagine you are a detective trying to solve a mystery. You have a bag of clues (data), but you don't know the rules of the game. In the world of science and statistics, there is a famous tool called Entropy. Think of Entropy as a "measure of confusion" or "uncertainty."

• Low Entropy: You know exactly what's happening (low confusion).
• High Entropy: Everything is a chaotic mess, and you have no idea what's going on (high confusion).

For decades, scientists have used a specific type of Entropy called Shannon Entropy (named after Claude Shannon) as the gold standard. It's like a trusted, Swiss Army knife that works perfectly for most situations.

However, in recent years, scientists studying complex systems (like the stock market, the brain, or weather patterns) realized that the Swiss Army knife sometimes isn't sharp enough. They invented new, "generalized" versions of Entropy with extra knobs and dials (called parameters) to handle these weird, complex situations.

The Problem: Too Many Knobs, No Manual

The problem with these new, fancy Entropies is that they come with extra knobs (parameters) that you have to turn. But nobody knew how to set them!

• If you set them wrong, your model breaks.
• If you set them based on "gut feeling," you aren't being scientific.
• If you try to figure them out using the data, the math gets messy and contradictory.

It was like having a car with a gas pedal, a brake, and a mysterious "mystery dial" that you had to guess the setting for before you could even drive.

The Solution: The "Blank Page" Rule

The authors of this paper, Andrea Somazzi and Diego Garlaschelli, proposed a simple, common-sense rule to fix this mess. They call it the "Uninformativeness Axiom."

Here is the analogy:
Imagine you have a blank piece of paper. It has no writing on it. It is completely uninformative.

• If you ask a scientist, "How much information is on this blank page?" the answer should be zero (or a fixed maximum number, depending on how you measure it).
• It shouldn't matter which ruler or measuring tape you use. A blank page is a blank page.

The authors say: "If your Entropy formula gives you a different 'confusion score' for a blank page just because you turned a different knob, then that formula is broken."

They demand that for a completely random, uniform situation (the blank page), the Entropy score must be the same, no matter what settings you choose.

The Great Filter: Who Survives?

When they applied this "Blank Page Rule" to the popular families of generalized Entropies, it acted like a sieve:

1. Tsallis Entropy (The Popular Contender): This was a very famous new Entropy used for complex systems. But when the authors applied their rule, it failed. It gave different scores for a blank page depending on the knob settings. It was disqualified.
2. Rényi Entropy (The Survivor): This was another candidate. When they applied the rule, it passed perfectly. It gave the same score for a blank page, no matter the settings. It was the winner.

The Result: The only "generalized" Entropy that makes sense without prior knowledge is Rényi Entropy.

The Magic Trick: Learning from Data

The most exciting part of the paper is what happens next. Because they found the "correct" Entropy (Rényi), they could now teach a computer to learn the settings (the knobs) purely from the data, without any human guessing.

Think of it like this:

• Old Way: You have a complex machine. You need to know the secret manual to set the dials before you can use it.
• New Way: You feed the machine data. The machine automatically adjusts the dials to find the perfect setting that explains the data best.

The authors showed that if you use their method, the machine doesn't just find the best settings; it also solves a deep mathematical puzzle. It turns out that when the machine finds the perfect setting, the "confusion score" (Entropy) it calculates magically becomes equal to the standard Shannon Entropy again.

Why Does This Matter?

1. No More Guessing: Scientists no longer need to guess the "secret parameters" for complex systems. The data tells them what to use.
2. Consistency: It fixes the contradiction where using multiple data points would require a different Entropy formula than using a single data point. The new rule unifies everything.
3. Model Selection: It gives a clear way to decide which mathematical model fits the real world best, just like choosing the best story to explain a mystery.

Summary in One Sentence

The paper introduces a simple rule—"a blank page must always look the same to any measuring tool"—which eliminates the confusing, inconsistent versions of Entropy, leaving only Rényi Entropy as the valid choice, allowing computers to automatically learn the best way to describe complex systems directly from data.

Technical Summary

1. Problem Statement

The paper addresses fundamental inconsistencies in the application of Generalized Maximum Entropy Principles (GMEP) for statistical inference. While Shannon entropy is uniquely defined by the Shannon-Khinchin (SK) and Shore-Johnson (SJ) axioms, various generalizations (e.g., Tsallis, Rényi, Hanel-Thurner) have been proposed to describe non-extensive or non-ergodic systems. These generalizations introduce "entropic parameters" (e.g., $q$, $c$, $d$) that index families of entropy functionals.

The authors identify three critical problems with current approaches to these generalized entropies:

1. Incompatibility with Maximum Likelihood (ML): For many generalized entropies, the distribution that maximizes the entropy under constraints does not yield a log-likelihood consistent with the ML principle. Specifically, the maximized log-likelihood does not equal the negative of the entropy, breaking the standard link used for model selection.
2. Inability to Infer Parameters from Data: Determining the correct entropic parameter (e.g., the correct $q$) usually requires a priori knowledge of the system's scaling properties (e.g., how the number of states scales with system size). It is generally impossible to infer these parameters purely from data without encountering inconsistencies.
3. The Independence Paradox: If a system is observed multiple times (independent and identically distributed, i.i.d.), the Shore-Johnson axioms imply the system should be described by Shannon entropy. However, if the system is described by a non-Shannonian entropy for a single observation, a contradiction arises regarding how to describe the same system under multiple observations.

2. Methodology

The authors propose a new, simple axiom to restrict the form of parametric entropy families, ensuring they are consistent with information-theoretic principles and the ML framework.

The Uninformativeness Axiom:
In a parametric family of entropies, the value of the entropy attained by the uniform probability distribution ($P_u$) must be independent of the value of the entropic parameter(s).

• Rationale: A uniform distribution represents a state of maximum uncertainty (no information). Therefore, the "amount" of uncertainty (entropy) assigned to this state should be a universal constant (specifically $\ln \Omega$, where $\Omega$ is the number of states) regardless of the specific entropy functional chosen from the family.
• Implication: This axiom acts as a "vertical" constraint across the family, ensuring a universal scale for uncertainty.

Analytical and Numerical Approach:

• The authors apply this axiom to two major families of generalized entropies:
  1. Uffink-Jizba-Korbel (UJK) entropies: $S^{(f)}_q[P] = f(U_q[P])$, where $U_q$ is the Uffink functional.
  2. Hanel-Thurner (HT) entropies: Trace-form or composable entropies defined by scaling limits.
• They reformulate the Generalized Maximum Entropy Principle (GMEP) using the surviving entropy family, deriving the corresponding probability distributions (q-exponentials) and Lagrange multipliers.
• They extend the ML principle to jointly estimate structural parameters (Lagrange multipliers) and the entropic parameter $q$.
• Numerical simulations are performed on systems with exponential and power-law distributions (including cases with diverging means) to test the inference procedure.

3. Key Contributions

A. Selection of Rényi Entropy

The application of the Uninformativeness Axiom acts as a rigorous filter:

• UJK Family: The axiom forces the function $f$ in $S^{(f)}_q[P] = f(U_q[P])$ to be the natural logarithm ($f(x) = \ln x$). This uniquely selects Rényi entropy as the only viable member of the family.
  • Result: Tsallis entropy is rejected because its value on a uniform distribution depends on $q$ ($S_{Tsallis}[P_u] = \frac{\Omega^{1-q}-1}{1-q}$), violating the axiom.
• HT Family: The axiom restricts the scaling parameters $(c, d)$ to $(1, 1)$. This identifies the equivalence class of additive entropies, which includes both Shannon and Rényi entropies.

B. Restoration of ML Consistency

The paper demonstrates that using Rényi entropy restores consistency with the Maximum Likelihood principle:

• For a single observation ($M=1$), the maximized log-likelihood equals the negative Rényi entropy: $\ell_q = -S_q$.
• For multiple independent observations ($M > 1$), the authors show that while the average log-likelihood no longer equals the negative Rényi entropy, it exactly equals the negative Shannon entropy ($S_1$) when the entropic parameter $q$ is optimized via ML.
• This resolves the "Independence Paradox": When multiple independent observations are available, the system is correctly described by Shannon entropy (as per SJ axioms), even if the underlying single-system distribution maximizes Rényi entropy.

C. Pure Data-Driven Parameter Inference

The authors propose a generalized ML procedure to infer the entropic parameter $q$ purely from data:

1. Maximize the log-likelihood $\ell_q(\psi)$ with respect to the structural parameter $\psi$ (Lagrange multiplier) for a fixed $q$.
2. Maximize the resulting profile likelihood with respect to $q$.
3. The optimal $q^*$ is found where the maximized log-likelihood equals minus the Shannon entropy ($S_1[P_{q^*}] = -\ell_{q^*}$).
   This allows the "correct" entropy to be learned from the data without prior knowledge of the system's scaling laws.

4. Results

• Theoretical Validation: The Uninformativeness Axiom successfully eliminates Tsallis entropy from the set of consistent inference functionals while preserving Rényi entropy.
• Model Selection: The paper proves that model selection based on the generalized ML principle (maximizing log-likelihood over both structural and entropic parameters) is equivalent to selecting the model that minimizes Shannon entropy. This provides a unified framework where Rényi entropy defines the functional form of the distribution, but Shannon entropy governs the model selection criterion.
• Numerical Examples:
  • Exponential Distribution ($q=1$): The inference correctly identifies $q^*=1$.
  • Power-Law Distributions ($q \neq 1$): The method correctly infers $q^*$ (e.g., $1.3$ and $1.6$) from data.
  • Diverging Means: In cases where the first moment diverges (e.g., $q=1.6$), the method successfully uses the q-mean (escort mean) as the constraint, whereas using the standard arithmetic mean would fail or diverge.
  • Bernoulli Case: The method correctly identifies that for a binary variable, the data cannot distinguish between different $q$ values (degenerate case), as all $q$-distributions collapse to the same empirical frequency.

5. Significance

This paper provides a foundational resolution to long-standing issues in non-extensive statistical mechanics and information theory:

1. Unification: It bridges the gap between generalized entropies and standard statistical inference (ML), showing that they are not mutually exclusive but rather complementary when the correct entropy (Rényi) is selected.
2. Practical Applicability: It offers a concrete, data-driven method to determine the "correct" entropy for a given dataset, removing the need for subjective or theoretical a priori assumptions about system scaling.
3. Theoretical Rigor: By introducing the Uninformativeness Axiom, the authors establish a necessary condition for any generalized entropy to be a valid quantifier of uncertainty, effectively ruling out popular but inconsistent functionals like Tsallis entropy for inference purposes.
4. Handling Heavy Tails: The framework naturally accommodates systems with heavy-tailed distributions and diverging moments by utilizing q-means, making it robust for complex systems where standard Shannon-based methods might fail due to non-convergence.

In summary, the paper argues that Rényi entropy is the unique generalization of Shannon entropy that allows for consistent parameter inference from data, restores the link between entropy and likelihood, and resolves the contradictions arising from system independence.