🧬 biology

Information Entropy of Biological Data: An Empirical Analysis

This paper presents a unified empirical analysis across six biological data modalities demonstrating that finite-sample bias severely distorts information-theoretic estimates, and argues that reporting entropy and mutual information without specifying the estimator and sample size is indefensible given that bias-corrected methods reveal distinct biological signals and correct systematic underestimation of entropy and overestimation of information content.

Original authors: Alexander Memming

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexander Memming

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine biology not just as a collection of cells and chemicals, but as a massive, noisy library of information. Every living thing, from a tiny bacterium to a human, is constantly reading, writing, and transmitting messages. To understand how much "surprise" or "variety" exists in these messages, scientists use a mathematical tool called entropy. Think of entropy as a measure of how unpredictable a message is. If a DNA strand is just a long string of the same letter repeated over and over, it has low entropy (it's boring and predictable). If it's a chaotic mix of different letters, it has high entropy (it's full of surprises).

Closely related to entropy is mutual information, which measures how much two things tell you about each other. If you know the weather outside, does that tell you what to wear? Yes, that's mutual information. If you know a specific gene sequence, does that tell you what protein it makes? Also yes. Scientists love these tools because they help decode the rules of life. However, there's a catch: these tools are notoriously tricky to use when you don't have a perfect, infinite amount of data. In the real world, we only have a limited number of samples—like trying to guess the flavor of a giant ice cream shop by tasting just a few scoops. If you aren't careful, your guess might be wildly wrong, not because the ice cream is weird, but because your sample was too small.

This is the puzzle that Alexander Memming tackles in a new study. The paper asks a simple but critical question: When scientists measure the "information" in biological data, how much of what they find is real biology, and how much is just a trick of the math caused by having too few samples?

The author set out to test this by looking at six different types of biological data: DNA sequences, protein structures, gene activity, and even the communities of bacteria living in our guts. They didn't just look at the data; they ran a massive experiment using six different mathematical "estimators" (methods for calculating entropy) on the same datasets. To see which method was telling the truth, they first tested them on fake, computer-generated data where the answer was already known.

The results were eye-opening. The study found that the most common, "naive" way of calculating these numbers (called the "plug-in" estimator) is often a liar. When scientists use this simple method on small datasets, they systematically underestimate how much entropy (variety) exists in DNA and proteins, making them look more predictable than they really are. Conversely, they overestimate mutual information, making unrelated genes or proteins look like they are talking to each other when they aren't.

For example, in the world of DNA, the study confirmed that the genetic code is indeed highly complex, with an entropy rate of about 1.9 bits per base (where the maximum possible is 2 bits). However, the naive method made it look slightly less complex. In the world of proteins, the error was huge: the simple method underestimated the variety of amino acids by about 12%. In gene networks, the naive method was so biased that it created "ghost connections," making it look like genes were interacting when they were actually independent. The study showed that by using better, bias-corrected methods, scientists can recover the true signal. For instance, when predicting how proteins fold, using the corrected methods improved the accuracy of contact predictions significantly, lifting the success rate from 25% to 67% for the best methods.

The paper also looked at single cells and found that as cells become more specialized (differentiating from stem cells to specific tissue types), their internal "noise" or entropy drops, confirming that specialized cells are more predictable. Furthermore, they discovered a "floor" of about 2.3 bits of entropy per gene that cannot be removed, even by the most advanced AI models, suggesting there is a fundamental, irreducible randomness to how genes are expressed.

Ultimately, this research acts as a reality check for the field. It proves that the bias isn't random; it follows a predictable pattern based on how much data you have compared to how many possibilities exist. The author argues that reporting an entropy number without saying which method was used and how much data was available is no longer acceptable. Just as you wouldn't trust a weather forecast without knowing the tools used to make it, we shouldn't trust biological conclusions about information without knowing how the math was corrected for the limits of our samples. The study doesn't invent new biology, but it provides the necessary calibration to ensure that the biology we do find is real, and not just a mirage created by a small sample size.

Technical Summary: Information Entropy of Biological Data: An Empirical Analysis

Problem Statement
Information-theoretic quantities—specifically Shannon entropy, mutual information (MI), and relative entropy—are foundational to biological data analysis, ranging from genomic sequence analysis to single-cell transcriptomics. However, these quantities are notoriously difficult to estimate from finite samples. The standard "plug-in" estimator, which substitutes empirical frequencies into the entropy formula, is systematically biased: it underestimates entropy and overestimates mutual information when the sample size ( $N$ ) is small relative to the alphabet size ( $m$ ). Despite the severity of this bias, which can inflate false edges in networks or manufacture spurious conservation signals, the applied biological literature frequently reports entropy values without specifying the estimator used, the sample size, or the magnitude of the resulting bias. This paper addresses the lack of empirical calibration regarding how much of a measured information-theoretic value represents biological signal versus sampling artifact.

Methodology
The authors conducted a unified empirical study across six distinct biological data modalities:

Genomic DNA (four reference genomes)
Protein multiple-sequence alignments (MSAs) (150 Pfam families)
Transcription-factor (TF) binding sites (62 motifs)
Bulk gene expression and gene regulatory networks (GRNs) (E. coli, DREAM5)
Single-cell transcriptomes (CELLxGENE census)
Microbiome communities (Human Microbiome Project)

For each modality, the study applied a common suite of six estimators:

Plug-in (PI): The naive empirical frequency baseline.
Miller–Madow: Analytic first-order bias correction.
Chao–Shen: Coverage-adjusted for unseen symbols (unbounded alphabets).
James–Stein Shrinkage: Regularization toward a uniform distribution for small $N$ , large $m$ .
NSB (Nemenman et al.): Bayesian estimation with a flat entropy prior.
KSG ( $k$ -nearest-neighbour): For continuous variables, estimating MI directly from neighbor distances without binning.

The methodology involved two phases:

Synthetic Validation: Estimators were tested on synthetic data with known ground-truth entropy and mutual information (Zipf distributions and Gaussian pairs) to quantify bias as a function of sample size.
Empirical Application: The calibrated, bias-corrected estimators were applied to real biological data to generate a cross-domain "entropy atlas," comparing naive values against corrected values to isolate biological signal from finite-sample artifacts.

Key Results

Synthetic Validation: On synthetic controls, the plug-in estimator was found to be severely biased. At $N=64$ samples for a 64-symbol alphabet, the plug-in estimator underestimated entropy by 0.66 bits and reported 0.56 bits of spurious mutual information between independent variables. Bias-corrected estimators (Chao–Shen, James–Stein, KSG) recovered the ground truth to within a few hundredths of a bit.
Genomic DNA: Bias-corrected entropy rates for four genomes ranged from 1.87 to 1.91 bits/base, confirming local statistics are near-maximal but not maximal. The study reconciled this with compressibility data: while local entropy is high, repeat-aware compression reveals significant long-range redundancy (up to 0.25 bits/base gap in human chromosomes) invisible to short-range block entropy.
Protein Coevolution: In contact prediction, the study confirmed a monotone improvement in precision as estimation methods became more sophisticated. Raw mutual information achieved a precision@ $L$ of 0.25; Average-Product Correction (APC) raised this to 0.42; mean-field Direct Coupling Analysis (mfDCA) to 0.55; and pseudolikelihood DCA (plmDCA) to 0.67. The study noted that in shallow alignments, all methods degrade toward the raw-MI baseline due to undersampling.
Gene Regulatory Networks: Replacing the plug-in estimator with bias-corrected alternatives (shrinkage, B-spline, or KSG) improved network inference performance (AUPR and AUROC) by amounts comparable to switching between different network inference algorithms (e.g., Relevance networks vs. CLR).
Transcription Factor Motifs: The study validated Schneider's information-conservation law ( $R_{sequence} \approx R_{frequency}$ ) across 62 motifs with a correlation of $r=0.98$ . It noted that the slight supra-unit slope observed was an artifact of plug-in bias inflating information content in low-count columns.
Single-Cell Transcriptomics: Signaling entropy was found to decrease monotonically with differentiation potency (Spearman $\rho = -0.87$ ). Furthermore, analysis of foundation model loss curves revealed an irreducible per-gene conditional entropy floor of approximately 2.3 bits, interpreted as the intrinsic stochasticity of gene expression.
Microbiome Diversity: Using Hill numbers and coverage-based correction, the study demonstrated that naive richness estimates significantly understate true diversity, particularly for rare taxa, mirroring the coverage problems seen in sequence-level entropy.

Cross-Domain Synthesis (The Entropy Atlas)
The study assembled a "cross-domain entropy atlas" revealing a predictable pattern of bias:

Sign: Entropies are systematically underestimated by plug-in estimators, while mutual information and information content are systematically overestimated.
Magnitude: The bias magnitude is a function of the sampling regime ( $N$ vs. $m$ ). In well-sampled regimes (genomic DNA), the bias is negligible (<1%). In undersampled regimes (protein columns, microbiome OTUs, bulk expression MI), the bias is substantial (10–80%), sometimes exceeding the biological signal entirely.

Significance and Claims
The paper argues that reporting entropy or mutual information without specifying the estimator and sample size is no longer defensible. The authors claim that:

The finite-sample bias is not random but predictable in both sign and magnitude based on the quantity being measured and the sampling regime.
Bias correction is not merely a technical refinement but a prerequisite for accurate biological inference in undersampled regimes, often yielding improvements comparable to changing the underlying biological model.
Modern self-supervised "foundation" models trained on biological data can be interpreted as large conditional-entropy estimators, subject to the same century-old sampling constraints; their loss floors represent empirical measurements of irreducible biological stochasticity.
The study provides a quantitative framework to distinguish how much of a measured information-theoretic value is biology and how much is a sampling artifact, urging the field to adopt bias-corrected estimation as a standard practice.

More like this