🧬 biology

An Entropy-based Coefficient of Determination with Adjustment of Optimization Bias

This paper introduces an entropy-based coefficient of determination ( $R^2$ ) that utilizes Entropic Variance to provide a scale-independent, distribution-agnostic measure of model fit, offering a robust alternative to classical metrics that suffer from sample-size inflation and coordinate-scale dependence while significantly reducing false discovery rates in variable selection.

Original authors: Longhai Li

Published 2026-08-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Longhai Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you have a very tricky tool: a magnifying glass that gets bigger and bigger the more clues you find. In the world of statistics, this tool is the "sample size." Usually, having more data is a good thing—it helps us see the truth more clearly. But in the world of scientific research, this magnifying glass has a glitch. If you look at a massive pile of data, even the tiniest, most meaningless speck of dust can look like a giant, glowing alien artifact. This is the "statistical significance crisis." Scientists are finding things that are "statistically significant" (meaning they are definitely not random noise) but are so tiny in real life that they don't matter at all. It's like being told you've won a prize, only to realize the prize is a single grain of sand.

To fix this, researchers usually try to measure the "size" of the effect, not just its existence. They want to know: "Is this clue actually useful, or is it just a tiny speck?" The most famous tool for this is called $R^2$ (R-squared). Think of $R^2$ as a scorecard that tells you how much of the mystery your model has solved. If your score is 100%, you've cracked the case. If it's 0%, you're guessing. But here's the problem: the old scorecards were built for simple, straight-line puzzles. When scientists started using them for complex, curved, or weirdly shaped data (like the patterns of bacteria in our guts), the scorecards broke. They would give scores that were negative, or scores that changed just because you decided to measure things in different units (like switching from inches to centimeters). It was like a ruler that gave you a different length depending on whether you were holding it with your left or right hand.

This paper introduces a brand new, super-smart scorecard called the Entropy-based Coefficient of Determination (or EV- $R^2$ ). The author, Longhai Li, realized that instead of measuring the "spread" of data like a traditional ruler, we should measure the "volume" of the information itself. Imagine your data is a cloud of gas. Old methods tried to measure the cloud by looking at its average height. This new method measures the actual space the cloud takes up. If your model is good, it squeezes that cloud into a tiny, dense ball. If your model is bad, the cloud stays huge and fluffy. This new scorecard, which the author calls $R^2_{SV}$ and $R^2_{SVP}$ , is special because it doesn't care about the units you use, and it has a built-in "lie detector" that stops you from getting excited about fake clues.

The paper finds that this new method is a game-changer for picking the right variables in a model. In a series of computer simulations, the author tested this new scorecard against the old ways of doing things. When they used the old methods on a massive dataset with lots of noise, the "lie detector" failed, and the model kept adding useless variables, thinking they were important. The false discovery rate (the percentage of times the model picked a fake clue) was a whopping 80%. But when they used the new EV- $R^2$ scorecard, that rate plummeted to just 6%, while still keeping the real, important clues. It's like switching from a detective who grabs every shiny object to one who only picks up the ones that actually solve the case.

The paper also applied this to a real-world mystery: Parkinson's disease and the bacteria living in our guts. When the researchers used the old "internal" method (where they used the same data to find the clues and then check them), they found about 20 bacterial clues and claimed the model explained 81% of the mystery. It sounded amazing, but it was likely a mirage caused by the data tricking itself. However, when they used the new method with a strict "data-splitting" rule (using one set of data to find clues and a completely different set to check them), the model shrank down to just 4 key bacterial clues. The new scorecard showed a realistic explanation of 34%. This suggests that the old method was wildly overconfident, while the new method gave a much more honest, grounded answer.

The author is very sure about the math behind this new scorecard. They didn't just guess; they proved that this new way of measuring follows a specific, predictable pattern (an F-distribution) that allows them to calculate exact probabilities and confidence intervals. They showed that this new method works perfectly for simple straight-line models (recovering the classic results) but also fixes the broken scorecards for complex, non-linear models. They also demonstrated that unlike the old methods, this new scorecard stays the same even if you change the way you measure the data (like switching from Celsius to Fahrenheit), which is a huge deal for making sure scientists are comparing apples to apples.

In short, this paper offers a new, more honest way to measure how well a model explains the world. It stops us from being fooled by massive datasets that make tiny things look huge, and it helps us find the real, meaningful patterns in complex data like our microbiome. It's not just a new formula; it's a new way of thinking about what it means to "solve" a statistical mystery, ensuring that when we say we've found a breakthrough, we actually mean it.

Technical Summary: An Entropy-based Coefficient of Determination with Adjustment of Optimization Bias

Problem Statement
The paper addresses the "statistical significance crisis" in applied research, where classical likelihood-ratio tests (LRT) and information criteria like $\Delta$ AIC scale with sample size, routinely flagging negligible improvements as highly significant in large datasets. While causal estimands like the Average Treatment Effect (ATE) attempt to quantify practical magnitude, they rely on the expectation operator ( $\mathbb{E}$ ), tethering them to the data's original coordinate scale and making them vulnerable to non-linear transformations.

Existing pseudo- $R^2$ metrics fail to resolve these issues adequately:

Variance-based measures (e.g., Efron's $R^2$ , Gelman's $R^2$ ) ignore higher-order distributional improvements (e.g., heteroscedasticity, heavy tails) and lack invariance to monotone transformations.
Entropy-based measures (e.g., McFadden's $R^2$ , Cox & Snell's $R^2$ ) encounter geometric paradoxes in continuous domains. Differential entropy is unbounded below, leading to metrics that can violate $[0, 1]$ bounds or lack scale invariance because linear ratios of cross-entropy are sensitive to additive constants introduced by unit changes.
Optimization Bias: Standard estimators often suffer from training optimism (overfitting), and existing frameworks lack a rigorous finite-sample inferential machinery (e.g., $p$ -values, confidence intervals) that does not require intractable Fisher information matrices.

Methodology
The author proposes a framework based on Entropic Variance (EV), defined as the exponentiated twice cross-entropy ( $v = \exp(2h)$ ). They argue that EV, rather than entropy itself, provides the rigorous generalization of the Ordinary Least Squares (OLS) residual variance for regular parametric families.

Population Definitions:
- Entropic Variance ( $\nu^*$ ): Defined as the geometric mean of individual EVs across the population.
- Population Effect Size ( $\rho^2_V$ ): The proportional reduction in EV between a restricted model ( $\mathcal{M}_0$ ) and a fuller model ( $\mathcal{M}_1$ ): $\rho^2_V = 1 - \nu^*_1 / \nu^*_0$ .
- Standardization ( $\rho^2_{SV}$ ): For discrete outcomes, a correction factor is applied to ensure the metric spans $[0, 1]$ even when the null model has low entropy.
Geometric Interpretation:
- The paper establishes that $\rho^2_V$ represents the proportional reduction in the squared effective number of states (for discrete data) or the squared geometric mean of kernel volumes (for continuous data).
- Crucially, $\rho^2_V$ is strictly invariant to strictly monotone transformations of the response variable, a property shared by neither variance-based nor standard entropy-based metrics.
Estimators and Bias Correction:
- $R^2_V$ : An empirical estimator correcting for finite-sample training optimism by applying a degrees-of-freedom penalty ( $n-p$ ) to the empirical EV, analogous to the unbiased variance estimator in OLS.
- $R^2_{SVP}$ : An out-of-sample predictive estimator that applies a squared complexity penalty to account for both training deflation and predictive inflation.
Inferential Framework:
- The author derives the distribution of the empirical Entropic Variance Signal-to-Noise Ratio (EV-SNR).
- They establish that the statistic $F_V = \frac{n-p_1}{k} \hat{SNR}_V$ asymptotically follows a non-central $F$ -distribution ( $F_{k, n-p_1}(\lambda_V)$ ), where $k$ is the difference in parameter counts.
- This allows for the calculation of refined $p$ -values and continuous confidence intervals for $\rho^2_V$ without requiring complex Fisher information matrix derivations.

Key Results

Theoretical Unification: The paper demonstrates that the EV framework recovers classical OLS theory exactly (recovering RSS, adjusted $R^2$ , and the ANOVA $F$ -test) when applied to Gaussian linear models.
Simulation Studies:
- In logistic regression simulations, the proposed $F_V$ test yields substantially lower Type I error rates compared to traditional $\chi^2$ and Mahalanobis distance ( $F_{MD}$ ) tests.
- Variable selection using $R^2_{SVP}$ (with a threshold of $\ge 0.05$ ) drastically reduces the False Discovery Rate (FDR) from 80% (under standard LASSO-CV) to 6% while preserving high recall of true signals.
- The estimators show robust finite-sample properties with confidence intervals achieving nominal coverage rates.
Real-World Application (Parkinson's Disease Microbiome):
- Applied to a gut microbiome dataset ( $n=328$ ), the method utilized a strict data-splitting approach to separate variable ordering (LASSO) from evaluation.
- The method identified a biologically interpretable model with only four predictors (including Roseburia, Lactobacillus, Bacteroidales, and an age-sex interaction) achieving a realistic out-of-sample $R^2_{SV}$ of 34%.
- In contrast, internal evaluation (using the same data for selection and evaluation) produced an inflated $R^2_{SV}$ of 81% with approximately 20 predictors, illustrating the severity of optimization bias.

Significance and Claims
The paper claims to resolve long-standing paradoxes in defining a universally applicable predictive effect size by shifting from raw entropy to Entropic Variance. The primary contributions are:

Scale Independence: Providing a dimensionless measure of effect size that is invariant to monotone transformations, unlike variance-based or standard entropy-based metrics.
Inferential Rigor: Establishing a unified inferential machinery based on the $F_V$ distribution, enabling precise $p$ -values and confidence intervals for effect sizes in general regular parametric families without intractable analytic derivations.
Practical Utility: Demonstrating that evaluating models via predictive effect size ( $R^2_{SVP}$ ) rather than statistical significance or unpenalized information criteria leads to superior variable selection, significantly reducing false discoveries in high-dimensional settings.

The author positions this work as a bridge between abstract information theory and practical model evaluation, offering a robust criterion for variable selection grounded in practical predictive utility rather than mere statistical detectability. They note that future work is needed to generalize the framework to singular likelihoods (e.g., Bayesian inference, penalized regression with pre-selection) where simple parameter counting ( $p$ ) may be insufficient.

Technical Summary: An Entropy-based Coefficient of Determination with Adjustment of Optimization Bias

More like this