🔢 mathematics

An Information Theoretic Proof of the Radon-Nikodym Theorem

This paper presents an accessible proof of the Radon-Nikodym theorem using information-theoretic concepts, aiming to bridge the gap between measure theory and information theory where the theorem is frequently cited but its proof often omitted due to perceived complexity.

Original authors: Peter Harremoës

Published 2026-01-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Peter Harremoës

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Measuring the Unmeasurable

Imagine you are trying to describe the weather. You could say, "It is raining," or you could say, "There is a 30% chance of rain." In mathematics, the first is a simple fact, but the second is a measure—a way of assigning a value (probability) to an event.

Usually, mathematicians use a very strict, heavy-duty toolkit called Measure Theory to handle these values. One of the most famous tools in this kit is the Radon-Nikodym Theorem. Think of this theorem as a "translator." It allows us to convert one way of measuring things (like a complex, abstract probability) into a simple, readable function (like a density curve or a graph).

However, most textbooks say, "This theorem exists, but the proof is too hard and boring, so we'll just skip it."

Harremoës' paper does something different. He says, "Let's prove this using the tools of Information Theory instead." He treats uncertainty not as a rigid geometric shape, but as a flow of information. By doing this, he not only proves the theorem but also gives us a way to measure how close a rough guess is to the perfect answer.

Part 1: The Building Blocks (Lattices and Valuations)

To understand his proof, we first need to change the scenery.

The Analogy: The Classification Game
Imagine you have a box of objects (like fruits) and a list of properties (like "red," "sweet," "round").

A Lattice is just a system for organizing these objects based on how they overlap. If you have a "Red" group and a "Sweet" group, the lattice helps you find the "Red and Sweet" group.
A Valuation is a way of counting or weighing these groups. In standard math, you usually demand that the total weight of everything equals exactly 1 (like a probability). Harremoës relaxes this rule. He allows the total weight to be anything—2, 100, or even infinite. He calls these "Expectation Measures."

Why does this matter?
Standard math is like a strict accountant who only accepts balanced ledgers. Harremoës is like a flexible data scientist who says, "I don't care if the total is 1; I just want to know how the pieces relate to each other." This flexibility makes the math easier to handle.

Part 2: The Core Concept (Information Divergence)

The paper introduces a concept called Information Divergence (specifically, the Kullback-Leibler divergence).

The Analogy: The "Surprise" Meter
Imagine you have two maps of a city:

Map A (The Truth): Shows exactly where the traffic jams are.
Map B (Your Guess): Shows where you think the traffic jams are.

Information Divergence measures how "surprised" you would be if you used Map B but the reality was Map A.

If the maps are identical, the divergence is zero (no surprise).
If Map B says "no traffic" but Map A says "gridlock," the divergence is huge.

Harremoës uses this "Surprise Meter" as a ruler. He proves that if the "Surprise" between two ways of measuring things is finite (not infinite), then one of those ways can be perfectly translated into a function (the Radon-Nikodym derivative).

Part 3: The Proof Strategy (Zooming In)

How does he prove the theorem? He uses a strategy of approximation.

The Analogy: Pixelating an Image
Imagine you have a high-definition photo (the perfect Radon-Nikodym derivative) that is too complex to draw.

Step 1: You take a low-resolution version of the photo (a coarse grid). You calculate the "Surprise" between your low-res guess and the real image.
Step 2: You make the grid finer (more pixels). You recalculate the surprise.
Step 3: You keep zooming in.

Harremoës proves that if the total "Surprise" is finite, then as you keep zooming in (making your grid finer), your low-resolution guesses will eventually lock onto the perfect image. They won't just get close; they will converge to the exact answer.

He calls this process an "Information Projection." It's like shining a light on a shadow; as you refine the light source, the shadow sharpens until it reveals the true shape of the object.

Part 4: The "Almost Sure" Guarantee

One of the most powerful parts of the paper is proving that this convergence happens everywhere (or "almost surely").

The Analogy: The Crowd Vote
Imagine a crowd of people guessing the temperature.

Some days, the crowd is chaotic, and the guesses jump around wildly.
Harremoës proves that if the "Information Divergence" (the total error) is under control, the crowd's guesses will eventually stop jumping around. They will settle down and agree on the true temperature at almost every single point in the city.

He uses a mathematical tool called Doob's Maximal Inequality (think of it as a safety net) to show that the guesses can't get too crazy before they settle down.

Part 5: The "Necessary" Condition

Finally, the paper asks: "What happens if the 'Surprise' is infinite?"

The Analogy: The Broken Compass
If the divergence is infinite, it means your two maps are fundamentally incompatible. Harremoës shows that in this case, no matter how hard you try to refine your grid, your guesses will never settle. The "maximum error" will keep growing forever, proving that a perfect translation (the Radon-Nikodym derivative) simply doesn't exist in that specific scenario.

Summary: What Did We Learn?

New Perspective: You don't need the heavy, abstract machinery of traditional measure theory to prove the Radon-Nikodym theorem. You can use the intuitive tools of Information Theory (entropy and divergence).
Quantifiable Accuracy: This method doesn't just say "the answer exists." It gives you a way to measure exactly how close a rough approximation is to the real answer.
Flexibility: By using "valuations" (which don't need to sum to 1), the math becomes more adaptable to real-world data where total mass might be unknown or infinite.

In short, Harremoës took a famous, difficult math theorem and re-proved it using the language of "information" and "surprise," showing that when information flows smoothly, the mathematical translation between different ways of measuring the world is guaranteed to exist.

Technical Summary: An Information Theoretic Proof of the Radon-Nikodym Theorem

Problem Statement
The Radon-Nikodym theorem is a cornerstone of measure theory, providing the theoretical foundation for defining density functions, Shannon entropy, and $f$ -divergences in information theory. However, standard proofs in measure theory are often considered too abstract or difficult for inclusion in textbooks focused on probability or information theory. Consequently, the existence of Radon-Nikodym derivatives is frequently omitted in information-theoretic literature, despite its centrality to the field. Furthermore, existing proofs typically establish existence without quantifying the convergence properties of finite approximations to the derivative. This paper addresses the need for an accessible, information-theoretic proof that not only establishes existence but also characterizes the convergence of approximations.

Methodology
The author, Peter Harremoës, departs from traditional measure-theoretic approaches by utilizing information projections and the theory of valuations on lattices. The methodology proceeds through the following stages:

Generalization to Valuations and Lattices: Instead of restricting the analysis to $\sigma$ -algebras and normalized probability measures, the paper formulates the problem using valuations (set functions satisfying strictness, monotonicity, and modularity) defined on distributive lattices. This framework accommodates unnormalized measures (expectation measures) and allows for the modeling of uncertainty without the constraint of total mass equaling 1. The paper notes that entropy functions on lattices of functional dependencies and concept lattices naturally satisfy these valuation properties.
Information Divergence on Lattices: The concept of information divergence ( $D(\mu \parallel \nu)$ ) is defined for valuations on finite distributive lattices. For a finite lattice, the divergence is calculated over irreducible elements. The paper establishes that in a distributive lattice, the divergence is invariant across all maximal chains.
Information Projections: The proof constructs a sequence of finite sublattices ( $M_n$ ) approximating the full lattice. On each sublattice, a measure $\mu_n$ is defined as the information projection of $\nu$ onto the set of valuations matching $\mu$ on $M_n$ . This projection minimizes the divergence $D(\theta \parallel \nu)$ subject to the constraint $\theta|_{M_n} = \mu|_{M_n}$ .
Convergence Analysis: The sequence of densities (Radon-Nikodym derivatives) associated with these projections, denoted $f_n$ , is analyzed. Using properties of $f$ -divergences (specifically the Pythagorean inequality for projections) and martingale inequalities (Doob's maximal inequalities), the paper demonstrates that the sequence $f_n$ forms a Cauchy sequence in $L^1$ and converges pointwise almost surely.

Key Contributions and Results

Information-Theoretic Proof of Existence: The paper provides a proof of the Radon-Nikodym theorem that relies on the properties of information divergence and information projections rather than abstract measure-theoretic constructions. It demonstrates that if the divergence $D(\mu \parallel \nu)$ is finite, a Radon-Nikodym derivative exists.
Quantification of Approximation: Unlike classical proofs that merely assert existence, this approach quantifies the closeness of finite approximations. It shows that the sequence of derivatives derived from finite sublattices converges to the true derivative.
Almost Sure Convergence: A central result (Corollary 20) states that if $D(\mu \parallel \nu) < \infty$ , the sequence of Radon-Nikodym derivatives on a countable net of finite partitions converges pointwise almost surely to the derivative $\rho$ .
Necessity of Finite Divergence: The paper establishes that a finite information divergence is a necessary condition for the convergence of the maximum of densities over finite subalgebras. If $D(\mu \parallel \nu) = \infty$ , there exists a sequence of subalgebras where the integral of the maximum density tends to infinity (Theorem 22).
Extension to General Measures: The results are formulated for s-finite measures (measures that can be decomposed into a countable sum of finite measures), which is a broader class than the $\sigma$ -finite measures often assumed in standard texts. The paper proves that if $D(\mu \parallel \nu) < \infty$ and $\mu$ is s-finite, then $\nu$ must also be s-finite.

Significance and Claims
The paper claims that this approach offers a more intuitive path to the Radon-Nikodym theorem for researchers in information theory and statistics, as it utilizes tools (divergence, projections) already central to the field. The author argues that the theory of valuations on concept lattices provides a language closer to information processing than traditional topology-based measure theory.

The work suggests that this framework not only proves the existence of the derivative but also provides a mechanism to understand how finite approximations behave, offering a "quantified" version of the theorem. The author modestly notes that while the results can be extended to more general settings like point-free topology (lattices of locals), the current presentation focuses on topological spaces and finite distributive lattices to maintain simplicity. The paper concludes that this valuation-based approach may serve as a theoretical basis for unifying information theory, statistics, and probability theory in the future.