🔢 mathematics

Convergence of Differential Entropies -- II

This paper establishes that differential entropy converges under convergence in measure of probability density functions when the entropy integrands are uniformly integrable and tight, providing a complete characterization on bounded domains and recovering several existing sufficient conditions as corollaries.

Original authors: Mahesh Godavarti

Published 2026-02-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mahesh Godavarti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Entropy" Thermometer

Imagine you are a weather forecaster trying to predict the "chaos" or "disorder" of a system. In information theory, this chaos is called Entropy.

Now, imagine you have a sequence of weather maps (probability density functions, or $f_n$ ) that are slowly changing to look more and more like a final, perfect map ( $f$ ). You can see the maps getting closer and closer visually.

The Big Question: Just because the maps look the same, does the calculated chaos (the entropy) of the new maps also settle down to match the chaos of the final map?

Sometimes, yes. Sometimes, no. This paper is about figuring out exactly when the chaos settles down and why it sometimes doesn't.

The Problem: The "Ghost in the Machine"

The authors found that even if your maps look identical, the entropy calculation can go haywire because of "ghosts."

Think of the entropy calculation like a sensitive scale. It weighs the "surprise" of the data.

If a map has a tiny, incredibly tall spike (a very rare event that is extremely surprising), it adds a huge amount of weight to the scale.
If that spike is very thin, it might not change the look of the map much (the map still looks smooth to the eye).
But if that spike is tall enough, it can throw the entropy calculation off completely, even if the map looks perfect.

The paper asks: What rules do we need to put on these maps to ensure the entropy scale stays stable?

The Solution: The "Vitali" Safety Net

The authors use a classic mathematical tool called Vitali's Convergence Theorem. Think of this as a "Safety Net" for your calculations.

For the entropy to converge (settle down), the "weight" of the surprise (the entropy integrands) must satisfy two conditions:

Uniform Integrability (The "No Wild Spikes" Rule):
Imagine you are carrying a stack of boxes. You need to make sure that no single box in the stack is so heavy that it breaks your back, and no box is so heavy that it makes the whole stack unstable. The "weight" of the surprise must be distributed evenly. You can't have one tiny, infinitely heavy box hiding in the corner.
Tightness (The "Stay Close to Home" Rule):
Imagine the boxes are also moving around. "Tightness" means the boxes can't run off to infinity. They must stay within a reasonable neighborhood. If a heavy box of "surprise" runs off to the edge of the universe, it can mess up the total calculation even if it's far away.

The Main Discovery: If you ensure your "surprise boxes" don't get too heavy (Uniform Integrability) and don't run away (Tightness), then the entropy will definitely converge.

The New Rulebook: The "Orlicz" Condition

Before this paper, there were strict rules. One famous rule said: "You must prove that the surprise doesn't grow faster than a specific power (like $x^2$ or $x^{1.5}$ )."

The authors found a looser, smarter rule. They introduced an Orlicz Condition.

The Old Rule: "You can't exceed a speed limit of 100 mph." (Strict, fixed limit).
The New Rule: "You can go as fast as you want, as long as your fuel consumption follows a specific, slightly curved curve that gets steeper and steeper."

This new rule is strictly weaker, meaning it allows for more types of maps to be valid. It catches cases that the old, rigid rules would have rejected. It's like upgrading from a rigid speed limit to a smart traffic system that adapts to the car's engine.

The Disproof: Shattering a Belief

For a long time, mathematicians (Godavarti and Hero) had a hunch. They thought: "If we let the speed limit slowly drop from 100 mph down to 1 mph (but never quite reach 1), maybe that would be enough to keep the entropy stable."

The authors proved this wrong.

They built a specific "counter-example" (a trap).

They created a map with a spike that gets taller and thinner.
They tuned the math so that it just barely passed the "slowly dropping speed limit" test.
Result: The map looked fine, passed the test, but the entropy calculation still failed.

This is like a car that passes a safety inspection because the inspector is looking at the wrong angle, but the car still crashes because the engine is actually broken. The "moving limit" wasn't a strong enough safety net.

The "Bounded" Bonus: When the Room is Small

There is a special case: Bounded Domains (like a map of a small, closed room, rather than the whole world).

In a small room, the "Stay Close to Home" rule (Tightness) is automatic because there is nowhere to run.

The Result: In a small room, the only thing that matters is that the "surprise boxes" aren't too heavy.
Why it matters: This gives a perfect, complete answer for small systems. If the boxes aren't too heavy, the entropy works. If they are, it doesn't. There are no gray areas.

Summary for the Everyday Reader

The Goal: We want to know when the "chaos" of a changing system settles down to match the final system.
The Trap: Visual similarity isn't enough; hidden, tiny, super-heavy spikes can break the math.
The Fix: We need to ensure those spikes aren't too heavy and don't run off to infinity.
The Upgrade: The authors found a new, more flexible rule (Orlicz) that covers more situations than the old rules.
The Correction: They proved that a popular idea (letting the rules get slightly looser over time) is actually dangerous and doesn't work.
The Takeaway: If you are working with a limited space (like a bounded signal), checking if the "surprise" is under control is all you need to do to guarantee your entropy calculations are correct.

In short: Don't just look at the picture; check the weight of the hidden surprises.

1. Problem Statement

The paper addresses a fundamental question in information theory and probability: Under what conditions does the convergence of a sequence of probability density functions (PDFs) $\{f_n\}$ to a limiting PDF $f$ imply the convergence of their differential entropies?

Mathematically, if $f_n \to f$ (in measure), when does $H(f_n) \to H(f)$ , where $H(f) = -\int f(x) \log f(x) \, dx$ ?

Previous literature (specifically Godavarti & Hero, 2004) established sufficient conditions, such as:

Bounded densities and bounded moments.
A "fixed- $\alpha$ " condition: $\sup_n \int f_n |\log f_n|^\alpha dx < \infty$ for a fixed $\alpha > 1$ .

However, a critical open question remained: Can the fixed $\alpha > 1$ be replaced by a sequence $\alpha_n \downarrow 1$ ? (The Godavarti–Hero conjecture). Additionally, there was a lack of a unified framework explaining why various disparate sufficient conditions work.

2. Methodology

The author employs classical tools from real analysis and measure theory, specifically adapting them to the context of information theory:

Convergence in Measure: The baseline assumption is convergence in Lebesgue measure (which implies Total Variation convergence for PDFs).
Vitali's Convergence Theorem: The core analytical tool. The paper utilizes the theorem's requirement that a sequence converges in $L^1$ if and only if it converges in measure and is Uniformly Integrable (UI) and Tight (T).
De la Vallée–Poussin Theorem: Used to characterize Uniform Integrability via the existence of a superlinear convex function (Orlicz condition).
Entropy Integrands: The analysis focuses on the sequence of functions $g_n = f_n |\log f_n|$ . The convergence of entropy is reduced to the $L^1$ convergence of these integrands.

3. Key Contributions

A. Unification via Uniform Integrability and Tightness (UI & T)

The paper establishes that entropy convergence is equivalent to the Uniform Integrability and Tightness of the entropy integrands $\{f_n |\log f_n|\}$ .

Lemma 5: If $f_n \to f$ in measure and $\{f_n |\log f_n|\}$ is UI & T, then $H(f_n) \to H(f)$ .
Bounded Domains (Theorem 7): On bounded domains, tightness is automatic. Thus, on bounded domains, $H(f_n) \to H(f)$ if and only if $\{f_n |\log f_n|\}$ is uniformly integrable. This provides a complete characterization for bounded domains.
Unbounded Domains ( $\mathbb{R}^d$ ): Tightness is not automatic. Example 6 demonstrates that entropy can converge even if the integrands are not tight (though the converse fails in general).

B. The Orlicz Condition (Theorem 8)

The author introduces a new, strictly weaker sufficient condition for entropy convergence:
$\sup_n \int_{\mathbb{R}^d} f_n(x) \Psi(|\log f_n(x)|) \, dx < \infty$
where $\Psi$ is a single convex, non-decreasing function with $\Psi(0)=0$ and $\Psi(t)/t \to \infty$ (superlinear).

Significance: This condition targets the "information random variable" $|\log f_n(X)|$ directly. It is strictly weaker than the fixed- $\alpha$ condition ( $\Psi(t) = t^\alpha$ ) for any $\alpha > 1$ .
Examples of $\Psi$ : Functions like $t \log(1+t)$ or $t \log(e+t)$ satisfy the condition but grow slower than any polynomial $t^\alpha$ ( $\alpha > 1$ ).

C. Disproof of the Godavarti–Hero Conjecture (Theorem 15)

The paper explicitly disproves the conjecture that the fixed $\alpha > 1$ can be replaced by a sequence $\alpha_n \downarrow 1$ .

Counterexample: A sequence of PDFs on $[0,1]$ $[0, 1]$ is constructed where:
1. $f_n \to f$ almost everywhere.
2. $\sup_n \int f_n |\log f_n|^{\alpha_n} dx < \infty$ for $\alpha_n = 1 + 1/\log n$ .
3. However, $H(f_n) \not\to H(f)$ .
Mechanism: The "moving exponent" condition fails because no single superlinear function $\Psi$ dominates the sequence $t^{\alpha_n}$ as $\alpha_n \to 1$ . The entropy integrands fail to be uniformly integrable despite satisfying the moving-exponent bound.

D. Recovery of Previous Results

The paper demonstrates that the sufficient conditions from Godavarti–Hero (2004), Piera–Parada (2006), and Ghourchian–Gohari–Amini (2019) are all corollaries of the new Orlicz condition (Theorem 8).

The fixed- $\alpha$ condition is a special case with $\Psi(t) = t^\alpha$ .
Bounded density ratios imply the Orlicz condition via de la Vallée–Poussin.
Bounded density and moment conditions also imply the Orlicz condition.

4. Key Results Summary

Concept	Result
General Condition	Entropy converges iff $\{f_n \|\log f_n\|\}$ is UI & T (Lemma 5).
Bounded Domains	Entropy converges iff $\{f_n \|\log f_n\|\}$ is UI (Theorem 7).
Sufficient Condition	$\sup_n \int f_n \Psi(\|\log f_n\|) < \infty$ for superlinear $\Psi$ (Theorem 8).
Conjecture Status	The Godavarti–Hero conjecture ( $\alpha_n \downarrow 1$ ) is FALSE (Theorem 15).
Unification	All prior sufficient conditions are special cases of the Orlicz condition.

5. Significance and Impact

Theoretical Clarity: The paper provides a "single mechanism" (UI & T of entropy integrands) that explains why various disparate conditions in the literature work. It shifts the focus from ad-hoc moment bounds to the structural properties of the entropy integrands.
Optimality: By proving the necessity of UI on bounded domains and providing a strictly weaker sufficient condition (Orlicz) on general domains, the paper tightens the theoretical bounds for entropy convergence.
Resolution of Open Problems: It definitively settles the open question regarding the replacement of fixed $\alpha$ with a sequence $\alpha_n \downarrow 1$ , showing that such a relaxation is insufficient for convergence.
Practical Application: The Orlicz condition offers a more flexible tool for verifying entropy convergence in applications like plug-in entropy estimation, mutual information continuity, and Central Limit Theorem approximations, particularly when densities are unbounded or have heavy tails that fail fixed-power moment conditions but satisfy slower-growing Orlicz bounds.

In conclusion, this work synthesizes classical measure theory with information theory to provide a rigorous, unified, and nearly complete characterization of differential entropy convergence.