📊 statistics

Breaking the Finite-Sample Barrier in Entropy Coupling

This paper introduces the minimum list entropy coupling to demonstrate that allowing arbitrary dependence among marginally constrained observations can eliminate residual uncertainty exactly after a finite number of samples, contrasting with the exponential reduction seen in independent settings, and provides structural conditions, a greedy algorithm, and applications to representation learning and randomness extraction.

Original authors: Shahab Asoodeh, Jun Chen

Published 2026-05-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shahab Asoodeh, Jun Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Magic" of Teamwork

Imagine you are trying to guess a secret number (let's call it X) that someone is holding. You can ask questions to get clues. In the world of this paper, the "clues" are a series of observations (Y1, Y2, ... Ym).

Usually, in statistics, we assume these clues are independent. Think of them like asking three different strangers on the street for directions. If they all give you slightly different, random advice, you get a little better at guessing the destination with each new person, but you might never be 100% sure. You need an infinite number of people to be absolutely certain.

This paper discovers a "magic trick": If you are allowed to coordinate your clues before you ask them (making them dependent on each other), you can figure out the secret number exactly after just a few clues.

The authors call this breaking the finite-sample barrier. Instead of slowly getting closer to the answer, you can jump straight to the perfect answer in a finite number of steps.

The Core Concept: Entropy Coupling

To understand how this works, let's use a Puzzle Analogy.

The Source (X): A picture of a landscape that is hidden inside a box. You don't know what it is.
The Marginals (The Rules): You are given a set of rules. For example, "The first clue must look like a blue sky," and "The second clue must look like green grass." These are the marginals. The clues must look like these specific things.
The Coupling (The Strategy): This is how you arrange the clues together.

Scenario A: The Independent Strategy (The Old Way)
You ask three friends to draw a piece of the picture. You tell Friend 1, "Draw a blue sky." Friend 2, "Draw green grass." Friend 3, "Draw a mountain."
If they draw these independently, they might draw a sky that doesn't match the grass, or a mountain that doesn't fit the sky. You get a jumbled mess. You can guess the picture better with more friends, but you'll likely never get the exact picture perfectly right unless you have infinite friends. The uncertainty (entropy) just gets smaller and smaller, but never hits zero.

Scenario B: The Dependent Strategy (The New Way)
This is what the paper proposes. You tell your friends: "I need you to draw a picture together, but you must follow the rules: Friend 1 draws a blue sky, Friend 2 draws green grass, etc."
Crucially, you let them talk to each other (or you coordinate them) to make sure their drawings fit together perfectly.

Friend 1 draws a sky.
Friend 2 looks at Friend 1's sky and draws grass that matches the horizon.
Friend 3 looks at both and draws a mountain that fits the scene.

Because they are dependent (coordinated), the final result is a perfect, complete picture of the landscape. You didn't need infinite friends; you only needed a specific number of them to make the puzzle fit perfectly. The uncertainty dropped to zero.

Key Findings Explained Simply

1. The "Phase Transition"

The paper shows a sharp difference between the two strategies:

Independent: Uncertainty fades away slowly, like a sunset. It takes a long time to get dark.
Dependent: Uncertainty vanishes instantly once you cross a certain threshold, like flipping a light switch. Once you have enough coordinated clues, the mystery is solved completely.

2. The "Shamir's Secret Sharing" Trick

The authors use a clever mathematical trick (similar to a game of "Secret Sharing") to prove this.
Imagine you want to hide a secret number $X$ . You give a piece of the secret to $Y_1$ , another to $Y_2$ , and so on.

If $Y_1$ and $Y_2$ are random and independent, they tell you nothing about $X$ .
But if you tell $Y_1$ and $Y_2$ to pick numbers that add up to $X$ (modulo some number), then knowing $Y_1$ and $Y_2$ tells you exactly what $X$ is.
Even though $Y_1$ and $Y_2$ individually look like random noise (they satisfy the "marginal" rules), their relationship to each other holds the secret.

3. How Many Clues Do You Need?

The paper calculates exactly how many coordinated clues you need to solve the puzzle.

It turns out you don't need a huge number. If the secret is complex, you might need a number of clues proportional to the logarithm of the complexity.
Analogy: If the secret is a 10-digit phone number, you don't need 10 billion clues. You might only need a handful of coordinated clues to figure it out exactly.

4. The Algorithm (The "Greedy" Solver)

The authors also built a computer program (an algorithm) to find the best way to coordinate these clues.

Think of it like a puzzle solver that tries different ways to fit the pieces together.
It starts with a "smart guess" (a structured way of linking the clues) and then refines it step-by-step to make the uncertainty as low as possible.
The paper shows that if you start with a random guess, the computer gets stuck. But if you start with a "coordinated" guess, it quickly finds the perfect solution.

Real-World Examples Mentioned in the Paper

The paper doesn't just talk about theory; it shows where this "magic" applies:

Perfect Data Compression (Representation Learning):
Imagine you want to send a secret message (the source) to a friend, but you are forced to send it in a format that looks like random noise (the marginal constraints).
- Old way: You send many random-looking packets. The friend can only guess the message with some errors.
- New way: You coordinate the packets so they fit together perfectly. The friend receives the noise, but because the noise is coordinated, they can reconstruct the exact original message with zero errors.
Creating Perfect Randomness (Randomness Extraction):
Imagine you have a biased coin (it lands on Heads 70% of the time) and you want to create a perfectly fair coin (50/50).
- Old way: If you flip the biased coin many times independently, you can get close to 50/50, but you can never get a perfectly fair bit from a finite number of flips due to math constraints.
- New way: If you are allowed to coordinate the flips (make them dependent), you can create a perfectly fair bit from just two flips. You simply define a rule: "If the flips are different, it's Heads; if they are the same, it's Tails." With the right coordination, this creates a perfect 50/50 outcome.

Summary

The paper proves that coordination is powerful.
If you are allowed to link your observations together (make them dependent) while keeping their individual appearances the same, you can solve mysteries and extract information with perfect precision using only a small, finite number of samples. This breaks the old rule that said you needed infinite data to get a perfect answer.

Technical Summary: Breaking the Finite-Sample Barrier in Entropy Coupling

Problem Formulation

The paper addresses the problem of minimum list entropy coupling, denoted as $H(P\|Q_1, \dots, Q_m)$ . Given a source distribution $P$ on a finite alphabet $\mathcal{X}$ and a set of prescribed marginal distributions $\{Q_i\}_{i=1}^m$ , the objective is to find a joint distribution of $(X, Y_1, \dots, Y_m)$ such that $X \sim P$ and $Y_i \sim Q_i$ for all $i$ , which minimizes the conditional entropy $H(X|Y_1, \dots, Y_m)$ .

The central inquiry is whether the residual uncertainty about $X$ can be eliminated exactly (i.e., $H(X|Y_1, \dots, Y_m) = 0$ ) after a finite number of observations $m$ , or if uncertainty merely decays asymptotically. This is contrasted with classical settings where observations are often assumed to be independent (i.i.d.) or conditionally independent given $X$ . The paper investigates how allowing arbitrary dependence among the observations $Y_1, \dots, Y_m$ (while keeping their marginals fixed) alters the fundamental limits of uncertainty reduction.

Methodology and Theoretical Framework

1. Coupling Spaces and Dependence

The authors distinguish between different classes of couplings:

Independent/Conditionally i.i.d. Couplings: Observations are independent or conditionally independent given $X$ . In these regimes, uncertainty reduction is typically exponential but rarely reaches zero exactly at finite $m$ unless strong structural conditions (like deterministic pushforwards) are met.
General Couplings: The paper allows $Y_1, \dots, Y_m$ to be arbitrarily dependent. This enlarges the space of admissible joint distributions, enabling the construction of joint supports that are not product-structured.

2. Analytical Tools

Quantile Coupling: Used to establish baseline exponential decay of entropy under product-law restrictions (independent observations).
Modular Sums and Convolution: The core mechanism for achieving zero entropy involves constructing block sums of observations modulo $r$ (the alphabet size). The paper leverages the fact that convolutions of distributions tend toward uniformity (or full support) as the block length increases.
Simplex Spanning: The theoretical proof demonstrates that if block sums have a fully supported distribution, they can approximate the vertices of the probability simplex. By inverting this basis, any target distribution $P$ can be represented exactly as a deterministic function of the coupled observations.
Greedy Linear Programming: To compute $H(P\|Q_1, \dots, Q_m)$ , the authors propose a greedy algorithm (Algorithm 1) that iteratively minimizes a linear surrogate of the conditional entropy over a restricted support set.

Key Contributions and Results

1. The Zero-Entropy Phase Transition

The paper establishes a sharp dichotomy between independent and dependent observations:

Independent Case: Under the product-law restriction ( $Y^m \sim Q_1 \otimes \dots \otimes Q_m$ ), the conditional entropy decays exponentially: $H(P\|Q_1, \dots, Q_m) \leq \zeta_m \log r$ , where $\zeta_m$ depends on the maximum atom of the marginals.
Dependent Case: Allowing arbitrary dependence induces a phase transition where the conditional entropy becomes exactly zero at a finite sample size $m_0$ .

2. Exact Recovery Conditions

The authors provide necessary and sufficient conditions for finite-sample exact recovery ( $H(P\|Q_1, \dots, Q_m) = 0$ ):

Homogeneous Case: If $Q$ is such that its $\ell$ -fold convolution $Q^{*\ell}$ is fully supported, then exact recovery is possible for all $m \geq m_0$ , where $m_0 = O(\log(1/P_{\min}))$ . Here, $P_{\min}$ is the minimum non-zero mass of $P$ .
Heterogeneous Case: The result extends to non-identical marginals $Q_i$ if blocks of observations can be aggregated (via modular sums or other maps) into a fully supported distribution.
Specific Examples:
- Uniform Marginals: For any $P$ on $[r]$ , $H_m(P\|\text{Unif}([r])) = 0$ for all $m \geq 2$ . This is achieved via a secret-sharing style construction ( $Y_m = X - \sum_{i=1}^{m-1} Y_i \mod r$ ).
- Bernoulli Case: Explicit formulas are derived showing that for $m \geq 2$ , exact recovery occurs if the mass of the target event fits within the capacity of the joint support of the observations (e.g., $p \leq mq$ for $Ber(p)\|Ber(q)$ ).

3. Algorithmic Approach

The paper introduces a greedy LP-based algorithm to approximate $H_m(P\|Q)$ .

Mechanism: It iteratively solves a linear program to minimize a surrogate objective (cross-entropy) over the current active support.
Initialization: The algorithm relies heavily on initialization. Standard independent initialization fails to find low-entropy solutions. The authors propose a list-coupling initialization (maximizing $\Pr[X \in \{Y_1, \dots, Y_m\}]$ ) or a perturbed version to ensure the active support contains the necessary dependent structures.
Convergence: The algorithm guarantees monotone descent and local linear convergence under specific alignment conditions, though global optimality is not guaranteed due to the non-convexity of the problem.

4. Lower Bounds and Necessary Conditions

The paper derives lower bounds on $H(P\|Q_1, \dots, Q_m)$ based on:

Entropy Budget: $H(P) - \sum H(Q_i)$ .
Support Size: The number of distinct values the joint observation $Y^m$ can take must be at least the size of the support of $P$ . Specifically, $|\text{supp}(P)| \leq \prod |\text{supp}(Q_i)|$ is a necessary condition for zero entropy.

Significance and Applications

The paper claims that its framework formalizes finite-sample limits in two primary domains where dependence is crucial:

Distribution-Matching Representation Learning:
In scenarios where a representation $Y^m$ must match prescribed marginal constraints $Q_i$ (e.g., for fairness or privacy) while retaining information about a source $X$ , the paper shows that lossless representation (zero reconstruction error) is achievable in finite dimensions if dependence is allowed. This contrasts with independent representations, which only allow approximate recovery at finite blocklengths.
Randomness Extraction:
The framework addresses the problem of extracting a uniform $k$ -ary symbol from observations with prescribed marginals.
- Classical Limitation: Under i.i.d. sampling, exact extraction is often impossible at finite blocklengths due to arithmetic constraints (e.g., probabilities of the form $a/3^m$ cannot equal $1/2$ ).
- Dependent Solution: By allowing dependence, exact extraction becomes possible in finitely many samples. For instance, a uniform bit can be extracted from two samples of a biased coin if they are coupled correctly, a feat impossible with i.i.d. samples.

Conclusion

The paper demonstrates that the "finite-sample barrier" in entropy coupling is not intrinsic to marginal constraints but is a consequence of restricting observations to be independent. By enlarging the coupling space to include arbitrary dependence, the authors show that uncertainty can collapse to zero exactly after a logarithmic number of samples. This reveals a fundamental phase transition in information-theoretic recovery, with direct implications for lossless representation learning and exact randomness extraction.