🧬 biology

ML-PWS: Estimating the Mutual Information Between Experimental Time Series Using Neural Networks

This paper introduces ML-PWS, a method that combines machine learning with Path Weight Sampling to estimate a rigorous lower bound on information transmission rates from experimental time-series data without requiring a prior stochastic model.

Original authors: Manuel Reinhardt, Gašper Tkačik, Pieter Rein ten Wolde

Published 2026-08-11

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Manuel Reinhardt, Gašper Tkačik, Pieter Rein ten Wolde

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to figure out how much a secret message is being passed between two people, Alice and Bob, who are talking over a very noisy walkie-talkie. In the world of science, this is called "information transmission." Whether it's neurons in a brain firing signals, a stock market reacting to news, or a cell sensing its environment, scientists want to know: How fast and how clearly is the story getting through? The standard way to measure this is called the "information rate." It's like counting how many unique words Alice can say per second that Bob actually understands, ignoring the static and the misunderstandings.

The problem is that real-life signals aren't simple words; they are complex, twisting paths of data that change every millisecond. Trying to calculate the information rate for these messy, high-dimensional paths is like trying to count every single grain of sand on a beach while a hurricane blows. Traditional math methods often break down because the data is too complicated, and they either guess too much or get stuck in the noise. Scientists have been looking for a way to measure this "information rate" directly from the data itself, without needing to know the exact rules of the game beforehand.

This is where a new method called ML-PWS comes in, developed by researchers Manuel Reinhardt, Gašper Tkačik, and Pieter Rein ten Wolde. Think of their approach as a clever two-step magic trick. First, they use a "generative model"—a type of artificial intelligence that acts like a super-smart student—to study the messy data and learn how the system behaves. It's like the student watching Alice and Bob talk for hours and then building a perfect simulation of their conversation style. But here is the twist: just having a simulation isn't enough to get a precise number.

That's where the second step, called Path Weight Sampling (PWS), enters the stage. Imagine the AI student has built a map of all possible conversations. PWS is a rigorous mathematical technique that walks through this map, checking every possible path the conversation could have taken to see how likely it is. By combining the AI's learned "map" with this careful mathematical walk, the researchers can calculate a rigorous lower bound on the information rate. In plain English, this means they can say with absolute certainty: "The information is flowing at at least this much speed." They proved this works by testing it on fake data where they already knew the exact answer, and their method hit the target perfectly, while other popular methods either guessed too low or got the answer wrong. They even applied it to real data from salamander neurons, showing that this new tool can measure how much information a group of brain cells is actually sharing, revealing that when cells work together, they sometimes repeat themselves, which lowers the total information speed.

The Core Idea: Learning the Rules to Count the Secrets

The paper tackles a tricky problem: how do you measure how much information is flowing through a system when you don't know the underlying rules? Usually, to calculate the "information rate," you need to know the exact probability of every possible outcome. If you have a mathematical model of the system (like a set of equations describing how a neuron fires), you can use a technique called Path Weight Sampling (PWS) to get an exact answer. It's like having the rulebook for a board game; you can calculate the odds of winning perfectly.

But in the real world, we rarely have the rulebook. We only have the "time-series data"—a recording of what happened. We see the input (the stimulus) and the output (the response), but we don't know the hidden math connecting them. Previous methods tried to guess the rules using approximations (like assuming everything is a simple bell curve) or by using other machine learning tricks that often gave unreliable results, either underestimating the information or failing completely when the data got too complex.

The Solution: AI as a Detective, Math as the Judge

The authors propose ML-PWS, a hybrid method that combines the best of two worlds.

The Detective (Machine Learning): First, they feed the experimental time-series data into a neural network. This network is trained to be a "generative model." It doesn't just memorize the data; it learns the conditional probability. In our walkie-talkie analogy, it learns: "If Alice says this specific phrase, what is the probability Bob will hear that specific phrase?" The network learns to predict the output based on the entire history of the input and previous outputs. It's like the AI learns the "style" of the conversation so well that it can predict the next word with high accuracy.
The Judge (Path Weight Sampling): Once the AI has learned this style, the researchers don't just stop there. They use the AI's predictions as the "rulebook" for the PWS technique. Because the AI has learned the conditional probability (the likelihood of an output given an input), the PWS method can now mathematically "marginalize" over the inputs. This is a fancy way of saying they can calculate the total probability of the output happening, regardless of what the input was, by summing up all the possibilities the AI predicted.

Why This is a Big Deal: The "Lower Bound" Guarantee

The most exciting part of this paper is the guarantee. Many machine learning methods for measuring information are "black boxes." You run them, and they give you a number, but you don't know if that number is too high, too low, or just lucky.

The authors prove mathematically that their method provides a rigorous lower bound. This means the number they calculate is always less than or equal to the true information rate. It might be a bit conservative (underestimating the truth), but it will never lie and say there is more information than there actually is. This is crucial for science because it gives researchers a safe, reliable floor to stand on. If they say "the information rate is at least 5 bits per second," they know for a fact it isn't zero.

They tested this on three different synthetic systems (fake data generated by known mathematical models) where they knew the "ground truth" (the real answer).

The Test: They compared ML-PWS against other popular methods like the Gaussian approximation (which assumes everything is a simple curve), DoE (Difference of Entropies), and variational estimators like MINE and InfoNCE.
The Result: In almost every case, ML-PWS was the most accurate. It stayed incredibly close to the ground truth.
- The Gaussian approximation failed when the system was non-linear (when the relationship between input and output wasn't a straight line).
- Variational estimators (like MINE and InfoNCE) hit a "ceiling." They couldn't measure high information rates because their math gets stuck when the data gets too complex or the trajectories get too long. They essentially gave up and said, "It's at most this much," even when it was actually much higher.
- DoE sometimes overestimated the information, giving a number that was too high and therefore unreliable.

Real-World Application: Listening to Neurons

To show this isn't just a toy experiment, the authors applied ML-PWS to real biological data: recordings from 50 retinal ganglion cells in a salamander's eye. These cells were watching a bar move up and down on a screen.

The Setup: They trained a neural network to predict the firing patterns of these neurons based on the movement of the bar.
The Finding: They calculated the information rate for individual neurons and for the whole group working together.
The Surprise: When they looked at the group, they found that the total information rate was lower than the sum of the individual rates. This is because the neurons were "redundant"—they were all saying the same thing. The group wasn't adding new information; they were just repeating it.
Channel Capacity: They also used the model to ask, "What kind of moving bar would carry the most information?" By optimizing the input (the movement of the bar), they found a specific pattern that could carry significantly more information than the one used in the experiment. This suggests that the salamander's visual system could be much more efficient if the world moved in a different way.

The Bottom Line

This paper doesn't just offer a new way to guess; it offers a new way to know. By teaching a machine to learn the rules of a system and then using rigorous math to count the information based on those learned rules, the authors have created a tool that is both powerful and honest. It admits when it doesn't know everything (by providing a lower bound) but guarantees that what it does know is solid. For anyone trying to understand how information flows through complex systems—from brains to stock markets to engineered devices—this method provides a reliable compass in a sea of data.

Technical Summary: ML-PWS: Estimating the Mutual Information Between Experimental Time Series Using Neural Networks

Problem Statement
Quantifying information transmission in systems driven by time-varying signals is fundamental to analyzing natural and engineered systems, from biochemical signaling to neural coding. The standard metric for this is the information transmission rate, defined as the asymptotic growth rate of the path mutual information (MI) between input and output trajectories. However, calculating this rate directly from experimental time-series data is notoriously difficult due to the high dimensionality of trajectory spaces. Conventional non-parametric methods (e.g., histograms, kernel density estimation) fail in high dimensions, while advanced estimators like k-nearest-neighbor (KNN) suffer from uncontrolled biases. Existing machine learning approaches, such as neural variational estimators (MINE, InfoNCE) or difference-of-entropies (DoE) estimators, often provide loose bounds, suffer from saturation effects (e.g., the $\ln N$ ceiling in InfoNCE), or fail to provide rigorous bounds when marginal distributions are misspecified. Furthermore, the Path Weight Sampling (PWS) technique, which allows for exact calculation of information rates for stochastic models, requires a known mechanistic model and cannot be applied directly to experimental data lacking such a model.

Methodology: ML-PWS
The authors propose ML-PWS, a hybrid method that combines machine learning with the Path Weight Sampling (PWS) framework to estimate information rates directly from experimental time-series data without a prior mechanistic model. The methodology proceeds in three main stages:

Learning the Conditional Path Likelihood: The method employs an autoregressive generative model (specifically, a deep neural network) trained on experimental input-output pairs $(s_{1:n}, x_{1:n})$ . This network learns to predict the conditional path likelihood $Q(x_{1:n}|s_{1:n})$ by factorizing it into a product of step-wise conditional probabilities: $Q(x_{1:n}|s_{1:n}) = \prod_i Q(x_i | s_{1:i}, x_{1:i-1})$ . The network predicts the mean and variance of a Gaussian distribution for each time step, conditioned on the full history of inputs and outputs.
Exact Marginalization via PWS: Unlike methods that train a separate network to approximate the marginal distribution $Q(x_{1:n})$ , ML-PWS computes this quantity by exact marginalization over the known input distribution $P(s_{1:n})$ . Since the input distribution is controlled or characterized by the experimenter, the marginal is estimated as $Q(x_{1:n}) = \int ds_{1:n} P(s_{1:n}) Q(x_{1:n}|s_{1:n})$ . This integral is evaluated efficiently using Sequential Monte Carlo (SMC) techniques (specifically the RR-PWS scheme), which propagate an ensemble of input trajectories and reweight them to focus on high-likelihood regions, thereby avoiding the exponential inefficiency of brute-force sampling.
Estimating the Lower Bound: The mutual information is estimated as the difference between the marginal entropy and the conditional entropy, both computed as cross-entropies with respect to the learned model $Q$ . The authors prove that because the marginal $Q(x_{1:n})$ is derived exactly from the conditional $Q(x_{1:n}|s_{1:n})$ via the known $P(s_{1:n})$ , the resulting estimator $\hat{I}_Q$ constitutes a rigorous lower bound on the true mutual information $I(S; X)$ . The bias of this bound is non-negative and vanishes as the generative model $Q$ converges to the true conditional distribution $P$ .

Key Contributions and Benchmarks
The paper validates ML-PWS against ground-truth results (obtained by applying PWS directly to known stochastic models) and against alternative estimators (Gaussian approximation, DoE, MINE, InfoNCE, SMILE) across three synthetic benchmarks:

Minimal Nonlinear Channel (Example II): In a system with a saturating sigmoid nonlinearity, ML-PWS significantly outperforms variational bounds (InfoNCE, MINE, SMILE), which severely underestimate MI in high-information regimes due to sample size limitations. It also outperforms the Gaussian approximation, which becomes loose as nonlinearity increases, and the DoE estimator, which overestimates MI due to errors in learning the marginal distribution. ML-PWS remains indistinguishable from the ground truth across varying signal-to-noise ratios and nonlinearity strengths.
Variance-Modulated Two-Channel Output (Example III): In a system where the signal is encoded entirely in the output covariance (leaving the mean uncorrelated), Gaussian approximations fail completely. Here, the marginal distribution is a heavy-tailed mixture of Gaussians. ML-PWS converges to the ground truth as training data increases, whereas DoE plateaus at an overestimate because its separate marginal network cannot capture the heavy-tailed nature of the data. Crucially, ML-PWS provides a valid lower bound even when the generative model is misspecified (e.g., assuming diagonal covariance), whereas DoE fails to provide a bound in such cases.
Neural Data Application: The method is applied to spike-train data from 50 retinal ganglion cells in a salamander responding to a moving bar. The authors trained both single-neuron and collective population models. The generative models accurately reproduced experimental spike trains (reaching ~95% of the noise ceiling). The analysis revealed that the information rate of the collective population is significantly lower than the sum of individual rates due to redundant coding. Furthermore, by optimizing the stimulus statistics within the learned model, the authors estimated a channel capacity (38.0 bit/s) that is substantially higher than the rate achieved with the experimental stimulus, demonstrating the utility of ML-PWS for stimulus optimization.

Significance and Claims
The paper claims that ML-PWS offers a robust, rigorous alternative for estimating information rates from experimental data where mechanistic models are unavailable. Its primary significance lies in providing a strict lower bound on mutual information that is not subject to the saturation limits of contrastive estimators or the marginalization errors of difference-of-entropies methods. By leveraging the known input distribution to perform exact marginalization via SMC, ML-PWS decouples the accuracy of the bound from the difficulty of learning the marginal distribution directly. The authors note that the method is particularly effective in high-dimensional, nonlinear regimes and can be extended to estimate transfer entropies and causation entropies. The code for the implementation is made openly available.

The Core Idea: Learning the Rules to Count the Secrets

The Solution: AI as a Detective, Math as the Judge

Why This is a Big Deal: The "Lower Bound" Guarantee

Real-World Application: Listening to Neurons

The Bottom Line

More like this