🤖 AI

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

This paper introduces a comprehensive, copula-theoretic benchmarking framework with diverse synthetic and real-world tests to evaluate mutual information estimators, revealing that no single method universally outperforms others and highlighting specific limitations across non-parametric, discriminative, and generative categories.

Original authors: Alberto Foresti, Ivan Butakov, Alexander Tolmachev, Giulio Franzese, Alexey Frolov, Pietro Michiardi

Published 2026-07-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alberto Foresti, Ivan Butakov, Alexander Tolmachev, Giulio Franzese, Alexey Frolov, Pietro Michiardi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to measure the "closeness" of two friends. In the world of data science, this closeness is called Mutual Information (MI). It tells us how much knowing one thing (like a person's favorite color) helps us predict another thing (like their favorite food). If knowing the color gives you a huge clue about the food, they have high MI. If they are totally unrelated, the MI is zero.

For years, scientists have built different "rulers" (estimators) to measure this closeness. But there was a big problem: most of these rulers were only tested on simple, toy examples—like measuring the distance between two dots on a piece of paper. No one knew if they would work when the "dots" were actually complex, high-dimensional things like photos of cats, stock market trends, or medical records.

This paper is like a massive, rigorous stress test for all these rulers. The authors built a new, comprehensive "gym" to see which ruler actually holds up under pressure.

The New Gym: Two Types of Workouts

The authors realized that previous tests were too easy or too narrow. So, they designed two new types of workouts to test the rulers from every angle:

The "Synthetic Gym" (Copula-First):
Imagine you have a simple, predictable relationship between two variables (like a straight line). Now, imagine you take that relationship and twist it, stretch it, and wrap it around a complex shape (like a pretzel or a twisted ribbon) using mathematical transformations.
- The Analogy: It's like taking a simple dance step and forcing the dancers to perform it while wearing heavy, awkward costumes and walking on a trampoline. The "closeness" (MI) is still mathematically known, but the "dance" (the data) looks incredibly messy and complex. This tests if the ruler can handle complex shapes and high dimensions.
The "Real-World Gym" (Marginals-First):
Here, they took real-world data, like thousands of photos of digits (0–9) from the MNIST dataset or colorful images from CIFAR-10. They then created pairs of images that were "linked" in a specific, known way.
- The Analogy: Imagine taking two photos of the same cat, but one is slightly brighter than the other. You know exactly how they are linked (the brightness), but the images themselves are complex, high-resolution pictures. This tests if the ruler can handle real, messy data while still knowing the true answer.

The Contestants: Three Teams of Rulers

They tested three main families of "rulers" (estimators) in this gym:

The "Old School" Team (Non-parametric/k-NN): These are like using a tape measure. They are simple, fast, and work great for small, simple jobs.
The "Discriminative" Team (Variational/Neural): These are like a smart detective. They try to guess if two things are related by training a neural network to spot the difference between "related pairs" and "random pairs."
The "Generative" Team (Diffusion-based): These are like a sculptor. They try to build a model of how the data was created in the first place to figure out the relationship. These are usually the most complex and computationally expensive.

The Big Surprise: No "Super Ruler" Exists

The most important finding of the paper is a shattering of a common myth.

For a long time, people assumed that the most complex, expensive, "AI-powered" rulers (the Generative and Discriminative teams) were always the best because they use fancy neural networks.

The paper says: "Not so fast."

In simple, low-dimensional tasks: The "Old School" tape measures (k-NN) were actually the most accurate and efficient. The fancy AI rulers were overkill and sometimes even less accurate.
In high-dimensional, complex tasks: The "Old School" team broke down completely. They couldn't handle the complexity.
In specific high-MI scenarios: The "Generative" sculptors (like MINDE) often performed best, but they struggled with other specific types of noise.
In other scenarios: The "Discriminative" detectives did well, but they hit a "ceiling" where they couldn't measure very high levels of closeness accurately.

The Takeaway: There is no "universal winner." Just like you wouldn't use a sledgehammer to crack a nut, or a scalpel to chop down a tree, you must choose the ruler based on the specific job you are doing.

The Hidden Traps (Why it's so hard)

The paper also explains why measuring this "closeness" is so difficult, even for the best rulers. They identified four "traps" that trip everyone up:

The "Needle in a Haystack" Problem (Sample Complexity): To measure high closeness, you need an exponentially huge amount of data. It's like trying to guess the exact weight of a specific grain of sand in a desert; you need to look at almost the whole desert to be sure.
The "Fragile Ruler" Problem (Numerical Instability): Sometimes, the math gets so sensitive that tiny rounding errors in the computer's calculation cause the result to explode or become useless. It's like trying to balance a pencil on its tip; a tiny breeze (error) knocks it over.
The "Noisy Signal" Problem (High Variance): Even if the ruler is working, the answer can jump around wildly from one test to the next because the data itself is inherently noisy.
The "Blurred Photo" Problem (Diffusion Smoothing): The fancy "sculptor" rulers work by adding a little bit of noise to the data to learn patterns. However, if the data is already very concentrated (like a sharp, clear image), adding noise blurs the relationship so much that the ruler underestimates the closeness.

Conclusion

This paper doesn't just say "here is a new tool." Instead, it says, "Stop looking for a magic tool."

It provides a map that tells researchers and engineers:

If your data is simple and small? Use the simple, fast ruler.
If your data is complex and high-dimensional? You might need the expensive AI ruler, but be careful of its specific weaknesses.
If your data has very high "closeness"? Be prepared for the math to get unstable.

By exposing these specific failure modes, the paper gives the community a clear roadmap for building the next generation of tools, rather than just blindly picking the most expensive one available today.

Technical Summary: Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

Problem Statement

Mutual Information (MI) estimation is a fundamental task in machine learning, information theory, and statistical inference, underpinning applications ranging from representation learning and generative modeling to feature selection. Despite significant methodological progress, existing evaluation frameworks suffer from critical limitations. Current benchmarks predominantly rely on simplified, low-dimensional distributions (e.g., multivariate Gaussians) or artificial scenarios that fail to capture the complexity of real-world data. Consequently, the performance of estimators on complex, high-dimensional, and realistic distributions remains largely unexplored. Furthermore, the absence of a unified theoretical framework hinders the systematic organization of benchmarks, often leading to superficial conclusions, such as the assumption that computationally expensive estimators are invariably superior.

Methodology

The authors propose a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective. This perspective leverages the Data Processing Inequality (DPI) to decompose the challenges of MI estimation into two primary sources: complex dependency structures (copulas) and complex marginal distributions.

1. Unified Theoretical Framework

The paper establishes that all existing MI benchmarks can be viewed as evaluations of how well estimators capture the DPI: $I(V; W) \leq I(X; Y) \leq I(A; B)$ . By fixing the marginal distributions or the dependency structure, the authors unify synthetic and real-data benchmarks under a single theoretical umbrella.

2. Two Complementary Benchmark Families

The authors introduce two distinct families of tests to systematically stress-test estimators:

Copula-First Family (Synthetic & Flow-Based):
- Goal: Systematically vary ground-truth MI, dimensionality, and marginal complexity while maintaining a tractable dependency structure.
- Base Distributions: Includes a suite of synthetic distributions with known analytical MI, such as Correlated Normal, Correlated Student's $t$ , Log-Gamma Exponential, and Rare Event Channel.
- Transformations: Utilizes Continuous Normalizing Flows (CNFs) and handcrafted invertible mappings to transform simple latent variables into high-dimensional, complex marginals (e.g., images of rectangles or Gaussian blobs) while preserving the ground-truth MI.
- Discrete/Mixed Cases: Includes tests for discrete-continuous mixtures and quantized distributions.
Marginals-First Family (Real-World Data):
- Goal: Evaluate estimators on realistic, high-complexity marginal distributions (e.g., MNIST, CIFAR-10) while controlling the dependency structure.
- Methodology: Extends the "same-class-pairing" paradigm to a general data-coupling method. By defining Markov kernels that map latent variables (e.g., class labels or continuous brightness levels) to specific data distributions with disjoint supports, the authors construct paired samples with known ground-truth MI.
- Implementation: For CIFAR-10, a ResNet feature extractor is used to project images into an embedding space before applying the coupling scheme.

3. Estimator Classes Evaluated

The framework evaluates three broad classes of estimators:

Non-parametric: KSG (Kraskov-Stögbauer-Grassberger) and WKL (Wang-Kulkarni-Lee).
Discriminative (Variational): MINE (Mutual Information Neural Estimation), NWJ, and InfoNCE.
Generative: MINDE (Mutual Information Neural Diffusion Estimation) in both continuous (MINDE-C) and joint (MINDE-J) formulations.

Key Contributions

Unified Theoretical Perspective: The paper demonstrates that diverse existing benchmarks can be unified under a copula-theoretic view centered on the DPI. This allows for the principled decomposition of MI estimation challenges into copula complexity and marginal complexity.
Comprehensive Benchmark Suite: The authors provide a diverse suite of tests spanning purely synthetic distributions to real-world image data, enabling controlled variation of MI, dimensionality, and sample size.
Identification of Fundamental Barriers: Beyond known issues, the paper identifies and analyzes specific barriers to MI estimation:
- Exponential Sample Complexity: MI estimation requires sample sizes that grow exponentially with the true MI value.
- Numerical Instability: Even for simple distributions (e.g., Gaussians), high MI leads to numerical instability due to the sensitivity of the log-determinant of the covariance matrix.
- High Intrinsic Variance: The Pointwise Mutual Information (PMI) itself can have high variance, propagating error to the MI estimate.
- Sensitivity to Noise: Diffusion-based estimators may smooth sharp distributions, penalizing highly concentrated dependencies.
Systematic Evaluation: An extensive evaluation of the three estimator classes across all benchmark families reveals that there is no universal winner. Each category (non-parametric, discriminative, generative) systematically outperforms others under specific setups.

Experimental Results

Low-Dimensional Regimes: Non-parametric (k-NN) estimators generally outperform variational estimators in low dimensions and low MI settings, offering better accuracy with lower computational cost.
High-Dimensional & Complex Marginals: Variational estimators (e.g., InfoNCE) show robustness to representation dimension but struggle with raw high-dimensional data (e.g., MNIST pixels) due to quadratic complexity in batch size. Generative estimators (specifically MINDE-C) demonstrate superior performance in high-dimensional and high-MI tasks but require complex modeling.
Impact of Transformations: Non-linear transformations via CNFs significantly increase estimation difficulty for all methods, exposing limitations in handling intricate dependency structures.
Real-World Data: On MNIST and CIFAR-10, generative estimators (MINDE-C) generally outperform discriminative methods in high-MI scenarios, though they fail in specific high-dimensional embedding experiments where they struggle to learn irrelevant marginal information.
Failure Modes:
- MINDE-J consistently fails across real-world tasks, suggesting its parameterization is suboptimal compared to MINDE-C.
- InfoNCE with MLP critics becomes computationally prohibitive in high dimensions without dimensionality reduction.
- k-NN methods fail on raw high-dimensional data due to the curse of dimensionality but remain effective when paired with powerful feature extractors.

Significance and Claims

The paper claims that the prevailing assumption of a "universal winner" in MI estimation is incorrect. Instead, the choice of estimator must be context-dependent, guided by the specific characteristics of the data (dimensionality, marginal complexity) and the target MI magnitude.

The authors emphasize that their work provides a rigorous baseline for future developments by exposing inherent pitfalls of state-of-the-art estimators. By identifying specific failure modes—such as the exponential sample complexity in high-MI regimes and the numerical instability of diffusion-based approaches on concentrated distributions—the paper offers practitioners a roadmap for estimator selection. The work does not propose a new estimator but rather a comprehensive evaluation framework necessary to advance the next generation of neural MI estimation methods. The open-source code is provided to facilitate reproducibility and further research.