🤖 machine learning

NMINE: Normalized Mutual Information Neural Estimation

This paper introduces NMINE, a fully neural estimator for normalized mutual information that combines MINE-based mutual information estimation with neural marginal entropy learning to provide a more accurate and dimensionality-robust alternative to existing k-nearest-neighbor methods for continuous multidimensional variables.

Original authors: Petra Eerikinharju, Marko Tuononen, Ville Hautamäki

Published 2026-07-31

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Petra Eerikinharju, Marko Tuononen, Ville Hautamäki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out how much two things in the universe are connected. Maybe you're checking if the weather affects your mood, or if the number of steps you take relates to how hungry you get. In the world of data science, there's a special tool called Mutual Information that acts like a super-sensitive radar. Unlike a simple ruler that only measures straight lines, this radar can spot hidden, squiggly, and complex relationships between variables, whether they are moving in a straight line or dancing in a chaotic spiral.

However, this radar has a tricky quirk: its readings are unbounded and depend on the "units" of the things you are measuring. It's like trying to compare the weight of a feather to the weight of a mountain using a scale that changes its own definition of "heavy" every time you switch objects. To make these readings fair and comparable, scientists use a trick called Normalization. While some normalization methods squeeze scores into a neat 0-to-1 range, the specific approach used in this paper (asymmetric normalization) doesn't force the score into a fixed box. Instead, it preserves the ranking of connections, ensuring that if one variable explains another better than a second variable does, the score reflects that order clearly, even if the raw numbers aren't capped at 1. The big challenge? When you have many variables at once (like a high-dimensional puzzle), the old tools used to calculate these scores often get confused, shaky, or just plain wrong.

This is where a new team of researchers steps in with a fresh idea. They propose a method called NMINE (Normalized Mutual Information Neural Estimation), which swaps out the old, clunky tools for a team of smart, trainable neural networks. Instead of trying to count neighbors in a crowded room (the old way), their system learns to "feel" the shape of the data directly. By training these digital brains to spot the differences between how variables act together versus how they act alone, the NMINE method creates a more accurate and stable score for how connected things really are. Their experiments show that this neural approach is a promising new direction, especially when dealing with complex, multi-dimensional data where traditional methods start to stumble.

The Problem: The "Neighbor" Counting Game

For a long time, the standard way to measure these connections was the KSG method (named after Kraskov, Stogbauer, and Grassberger). Imagine you are in a giant, multi-story library (representing high-dimensional data). To see if two books are related, the KSG method asks you to find the five closest books to your target and count them. It works great in a small, single-story library (low dimensions). But as the library grows taller and wider with more floors and aisles (higher dimensions), finding those "closest" books becomes a nightmare. The distances get weird, the counts become unreliable, and the whole system starts to produce noisy, inaccurate results. It's like trying to find your best friend in a stadium full of people by only looking at the five people standing nearest to you; you might grab a stranger just because they happened to be standing there.

The Solution: Teaching a Neural Network to "Feel" the Data

The authors of this paper, Petra Eerikinharju, Marko Tuononen, and Ville Hautamäki, decided to stop counting neighbors and start training a neural network to do the heavy lifting. Think of their method, NMINE, as a team of three highly trained detectives (neural networks) working together to solve the mystery of connection.

The Joint Detective: This network looks at the two variables together (let's call them X and Y) and tries to figure out how much they "know" about each other. It uses a mathematical trick called the Donsker–Varadhan representation to estimate the Mutual Information.
The Solo Detectives: Two other networks look at X alone and Y alone. Their job is to estimate the Entropy (a measure of uncertainty or "surprise") for each variable.
The Reference Trick: Here is the clever part. Instead of trying to guess the exact shape of the data (which is hard), these networks compare the data to a simple, uniform "blank canvas" (a uniform reference distribution). Imagine trying to describe a complex painting by measuring how different it is from a plain white wall. If the painting is very different from the wall, it has high complexity (entropy). By measuring this "difference" (divergence) using the neural networks, they can mathematically recover the entropy without needing to know the exact shape of the data.

Once the networks have estimated the connection (Mutual Information) and the individual uncertainties (Entropy), they combine them. The paper specifically uses asymmetric normalization, which answers the question: "How much of Y is explained by X?" This is chosen because it keeps the ranking of connections consistent, ensuring that if X is a better predictor of Y than Z is, the score reflects that clearly.

What They Found: Smarter in Higher Dimensions

The team tested their new neural detective against the old "neighbor-counting" KSG method using synthetic data that looked like a cloud of points (Gaussian data) in spaces ranging from 1 to 8 dimensions.

The Results: In the lower dimensions (1 and 2), the old KSG method actually followed the theoretical truth very closely. However, as they cranked up the complexity to 4 and 8 dimensions, the KSG method started to fall apart. It began to overestimate the connections, essentially shouting "They are totally connected!" even when they weren't, especially when the variables were strongly linked.
The Neural Edge: The NMINE method, while slightly conservative (it tended to slightly underestimate the strength of the connection in the highest dimensions), stayed much more stable. It didn't get as jittery or noisy as the old method.
The Numbers: When they measured the error (how far off the estimate was from the true value), NMINE was significantly better overall. For example, in 1-dimensional data, NMINE reduced the error by about 74% compared to KSG. Even in the tough 8-dimensional test, it still cut the error by nearly 47%. A statistical test confirmed that this improvement wasn't just luck; it was a real, significant difference.

They also ran a quick test on data that looked like a "Student-t" distribution (which has heavier tails, meaning extreme outliers are more common). While they didn't have a perfect "true answer" to compare against, the neural method still showed a smooth, logical response as the connections got stronger, suggesting it might work well even on messy, real-world data that isn't perfectly smooth.

Why It Matters (and What's Next)

The paper concludes that replacing the old, rigid neighbor-counting tools with flexible, trainable neural networks is a winning strategy for measuring connections in complex, multi-dimensional data. This is a big deal for fields like molecular dynamics (studying how molecules move) and interpretable machine learning (understanding why AI makes certain decisions), where understanding subtle, non-linear dependencies is crucial.

However, the authors are careful not to call this a "solved" problem. They note that their method requires training multiple neural networks, which takes more computer power and time than the old methods. They also point out that their current setup trains the networks separately, and future work could try to train them all together to make them even better. Additionally, while the method works well on the data they tested, they admit that more work is needed to see how it handles truly wild, non-Gaussian real-world datasets.

In short, NMINE offers a promising new way to measure the invisible threads connecting our data, proving that sometimes, to find the truth in a complex world, you need a neural net rather than just a ruler.

Technical Summary: NMINE – Normalized Mutual Information Neural Estimation

Problem Statement

Mutual information (MI) is a fundamental measure of statistical dependence capable of capturing both linear and nonlinear relationships between random variables. However, for continuous and multidimensional variables, exact computation of MI is infeasible, necessitating estimation from samples. A critical limitation of raw MI is that it is unbounded and its units depend on the underlying variables, making raw values incomparable across different datasets, dimensions, or applications.

While Normalized Mutual Information (NMI) addresses this by converting MI into a bounded dependency score, existing estimators face significant challenges. Classical approaches, particularly those based on k-nearest neighbors (KNN) such as the Kraskov-Stogbauer-Grassberger (KSG) estimator, rely on explicit density estimation or nearest-neighbor statistics. These methods are known to suffer from numerical instability, increased variance, and degraded accuracy as dimensionality increases. Furthermore, many existing NMI estimators lack scale invariance when based on differential entropy, which changes under coordinate scaling.

Methodology

The paper proposes NMINE (Normalized Mutual Information Neural Estimation), a fully neural framework for estimating NMI between continuous multidimensional variables. The approach replaces traditional nearest-neighbor statistics with neural divergence estimation, combining two primary components:

1. Neural Mutual Information Estimation

The framework utilizes the Mutual Information Neural Estimation (MINE) architecture (Belghazi et al. 2018), which is based on the Donsker–Varadhan (DV) variational representation of the Kullback–Leibler (KL) divergence. MI is estimated as the difference between the joint divergence and the sum of marginal divergences:
$\hat{I}(X; Y) = \hat{D}_{KL}(p_{XY} \parallel m_{XY}) - \hat{D}_{KL}(p_X \parallel m_X) - \hat{D}_{KL}(p_Y \parallel m_Y)$
Here, $m$ represents uniform reference distributions defined over the observed support of the variables. Three independent neural critics are trained to estimate these divergences.

2. Neural Entropy Estimation via Relative Entropy

To obtain the normalization factor, marginal and joint entropies are recovered from the estimated divergences. Inspired by Neural Entropic Estimation (NEE) (Chan et al. 2019) and Jaynes' relative entropy formulation, entropy is not estimated directly but recovered from the divergence between the empirical distribution and a uniform reference distribution:
$\hat{H}(X) = \log V_X - \hat{D}_{KL}(p_X \parallel m_X)$
where $V_X$ is the volume of the support of the uniform reference distribution (estimated from the observed sample ranges). This approach ensures that the entropy estimates are derived from the same neural divergence framework used for MI, avoiding explicit density estimation.

3. Asymmetric Normalization

The paper adopts an asymmetric normalization scheme (Jerdee, Kirkley, and Newman 2025) to preserve the ranking of mutual information values:
$\widehat{NMI}(X; Y) = \frac{\hat{I}(X; Y)}{\hat{H}(Y)}$
This formulation is chosen because it maintains the ordering induced by mutual information ( $I(X_1; Y) > I(X_2; Y) \implies NMI(X_1; Y) > NMI(X_2; Y)$ ) and provides a clear interpretation in terms of information recovery. The use of relative entropy (based on uniform references) rather than differential entropy ensures scale invariance.

Key Contributions

The authors present the following contributions:

A Fully Neural Estimator: Introduction of NMINE, a complete neural pipeline for estimating NMI between continuous multidimensional variables without relying on nearest-neighbor statistics.
Unified Framework: Integration of MINE-based mutual information estimation with MI-NEE-inspired entropy estimation within a single framework.
Divergence-Based Entropy Recovery: Derivation of entropy estimates from neural divergence estimates relative to uniform reference distributions, enabling entropy recovery without explicit density estimation.
Empirical Validation: Evaluation of the proposed method on multidimensional Gaussian data, demonstrating improved accuracy over a KSG-based NMI baseline.

Experimental Results

The method was evaluated on synthetic multivariate Gaussian data across dimensions $d \in \{1, 2, 4, 8\}$ and correlation coefficients $\rho \in [0, 0.95]$ .

Accuracy: NMINE achieved lower Mean Absolute Error (MAE) compared to the KSG baseline across all tested dimensions. Relative error reductions ranged from 30.7% to 74.0%.
Statistical Significance: A paired t-test confirmed a statistically significant difference in favor of NMINE ( $t = 9.31, p = 7.89 \times 10^{-19}$ ).
Behavioral Analysis:
- In low dimensions ( $d=1, 2$ ), NMINE closely followed the theoretical NMI curve.
- In higher dimensions, while both estimators became less accurate, their failure modes differed. The KSG estimator increasingly overestimated NMI, often approaching the upper limit of the normalized scale for strong correlations. Conversely, NMINE exhibited a systematic underestimation bias but successfully preserved the correct monotonic relationship between correlation strength and NMI.
Robustness: Preliminary experiments on multivariate Student-t distributions suggested that NMINE maintains a monotonic response to increasing correlation and exhibits lower variability than KSG in moderate-correlation settings, though a comprehensive evaluation of non-Gaussian data is left for future work.

Significance and Claims

The paper claims that NMINE demonstrates that normalized mutual information can be estimated effectively using a fully neural framework, offering a viable alternative to traditional nearest-neighbor-based estimators, particularly in high-dimensional continuous settings where KNN methods degrade.

The authors position the work as a step toward more robust dependency measurement in applications such as:

Interpretable Machine Learning: Comparing information content across neural representations.
Molecular Dynamics: Analyzing multidimensional dependencies.
Representation Learning: Characterizing latent representations.

The paper maintains a modest tone regarding its current scope. It acknowledges that the implementation relies on separate training of MI and entropy estimators and uses standard DV objectives without advanced variance reduction techniques (e.g., SMILE or exponential moving averages). The authors note that the use of observed sample ranges to define reference supports introduces a finite-sample approximation, which may affect sensitivity to sample size and extreme observations. Future work is identified as necessary for joint optimization, evaluation on nonlinear and real-world datasets, and further investigation into the method's behavior on distributions with unbounded support.