📊 statistics

Finite-Sample Bias Correction for Plug-in Estimators of Extropy, Rényi Extropy, and Tsallis Extropy

This paper derives explicit first-order finite-sample bias formulas for plug-in estimators of Shannon, Rényi, and Tsallis extropies, proposes bias-corrected versions that demonstrate superior finite-sample performance while remaining asymptotically equivalent to the uncorrected estimators.

Original authors: Amadou Diadie Ba

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amadou Diadie Ba

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of information and probability, scientists have long relied on a concept called entropy to measure how much uncertainty exists in a system. Think of it as a gauge for how unpredictable a set of events is; if you are flipping a coin that always lands on heads, there is no uncertainty, but if it lands randomly, the uncertainty is high. For decades, this measure of unpredictability has been the standard tool for fields ranging from cryptography to biology. However, there is a complementary way to look at uncertainty that focuses not on what happens, but on what does not happen. This alternative measure, known as extropy, quantifies the uncertainty associated with the non-occurrence of events. While entropy looks at the probability of an event occurring, extropy looks at the probability of it failing to occur. This distinction makes extropy particularly sensitive to rare events and the tails of a distribution, offering a sharper lens for analyzing risks, financial crashes, or the failure of a machine. Because of this sensitivity, researchers in finance, reliability engineering, and machine learning have begun to use extropy to better understand rare but catastrophic outcomes.

Despite the usefulness of extropy, measuring it from real-world data is tricky. When researchers try to estimate extropy from a finite set of observations—like a sample of one hundred stock prices or a hundred patient records—the standard method they use, known as a plug-in estimator, tends to be slightly off. This is not a random error that averages out perfectly; it is a systematic bias that consistently pushes the estimate in one direction. Imagine trying to weigh a heavy object on a scale that has not been zeroed correctly; every measurement you take will be slightly too light, no matter how many times you repeat the experiment. In the case of extropy, this bias can be significant when the sample size is moderate, leading to inaccurate conclusions about the true level of uncertainty in a system. The problem is especially pronounced when the data involves rare events or when the mathematical parameters used to tune the measurement are set to specific values.

A recent study by Amadou Diadie Ba at Gaston Berger University in Senegal addresses this specific problem by developing a way to correct these systematic errors. The researcher focused on three main variations of extropy: the standard Shannon version, and two generalizations known as Rényi and Tsallis extropies. These variations allow scientists to adjust how much weight is given to rare events versus common ones. The study began by analyzing the mathematical behavior of the intermediate calculations used to find these values. By looking closely at how the estimates change as the sample size grows, the author derived precise formulas that describe exactly how much the standard estimates are off. It was found that the error is not random but follows a predictable pattern that shrinks as the sample size increases, specifically at a rate related to the inverse of the sample size.

Using these formulas, the researcher constructed new, corrected estimators. These new tools take the standard calculation and add a small adjustment term that cancels out the systematic error. The study demonstrated that for moderate sample sizes, these corrected versions are significantly more accurate than the traditional ones. In a series of computer simulations, the researcher generated thousands of artificial datasets to test the performance of both the old and new methods. The results showed that while the uncorrected estimates consistently underestimated the true value, the corrected estimates hovered much closer to the truth, effectively removing the bias. This improvement was most noticeable when the data included rare events or when the mathematical parameters were set to values that emphasized those rare occurrences.

Crucially, the study also proved that as the sample size becomes very large, the difference between the corrected and uncorrected versions disappears. The two methods converge to the same result, meaning that for massive datasets, the simpler, uncorrected method is perfectly fine to use. This finding provides a clear guideline for practitioners: if you are working with a large amount of data, the simple method is sufficient and efficient. However, if you are working with a moderate amount of data, which is common in many real-world applications like financial risk analysis or medical studies, the corrected method is far superior. The research confirms that while the simple tools are mathematically sound in the long run, they require a small, calculated adjustment to be reliable in the short term. This work ensures that when scientists use extropy to measure the uncertainty of rare and critical events, their measurements are as accurate as possible, preventing the systematic underestimation of risk that could otherwise lead to poor decisions.

Technical Summary: Finite-Sample Bias Correction for Plug-in Estimators of Extropy Measures

Problem Statement
This paper addresses the finite-sample bias inherent in plug-in estimators for Shannon, Rényi, and Tsallis extropies when applied to finitely supported discrete random variables. While extropy serves as a complementary measure to entropy—quantifying uncertainty associated with the non-occurrence of events and proving sensitive to distribution tails—the standard plug-in estimators for these measures are known to be biased in finite samples. This bias can significantly degrade estimation accuracy, particularly when the sample size $n$ is moderate, the support size $r$ is moderate, or the parameter $\alpha$ (in Rényi and Tsallis extropies) deviates from 1. Existing methods to mitigate this bias, such as kernel-based approaches, Bayesian techniques, and bootstrap resampling, often suffer from computational intensity, sensitivity to tuning parameters, or difficulty in application to discrete data.

Methodology
The author adopts an analytical bias correction approach. The core methodology involves:

Intermediate Functional Analysis: The study focuses on the intermediate functional $S_\alpha(p) = \sum_{i=1}^r (1-p_i)^\alpha$ , which is central to defining Rényi and Tsallis extropies.
Taylor Expansion: By applying a second-order Taylor expansion to the function $\phi(t) = (1-t)^\alpha$ around the true probabilities $p_i$ , the author derives an explicit first-order bias formula for the empirical estimator $\hat{S}_\alpha$ .
Propagation of Bias: Using the derived bias of $\hat{S}_\alpha$ , the author analytically derives the first-order bias ( $O(1/n)$ ) for the plug-in estimators of Shannon, Rényi, and Tsallis extropies.
Correction Construction: Explicit bias-corrected estimators are constructed by subtracting the estimated bias term from the original plug-in estimators.
Asymptotic Analysis: The paper proves that the difference between the uncorrected and corrected estimators is of order $O(1/n)$ , establishing their asymptotic equivalence.

Key Contributions
The paper provides the following specific theoretical and practical contributions:

Explicit Bias Formulas: Derivation of closed-form first-order bias expressions ( $O(1/n)$ $O (1/ n)$ ) for:
- Shannon Extropy: The bias is found to be $-1/(2n)$ , independent of the specific probability distribution $p$ .
- Tsallis Extropy: The bias depends on $\alpha$ and the probabilities, specifically $-\frac{\alpha}{2n} \sum p_i(1-p_i)^{\alpha-1}$ .
- Rényi Extropy: A complex bias term involving the first and second moments of the transformed probabilities and the functional $S_\alpha$ .
Bias-Corrected Estimators: Proposal of specific estimators that incorporate these bias terms to improve finite-sample accuracy.
Asymptotic Equivalence Proof: A rigorous demonstration that the corrected and uncorrected estimators differ only by terms of order $O(1/n)$ , implying they share the same asymptotic properties (consistency and asymptotic normality).
Simulation Validation: A comprehensive simulation study ( $M=1000$ replications) across various sample sizes ( $n=10$ to $2000$) and parameters ( $\alpha=0.5, 3$ ) validates the theoretical bias formulas and confirms that corrected estimators significantly outperform uncorrected ones in moderate sample sizes.

Results

Finite-Sample Performance: The simulation results confirm that the uncorrected plug-in estimators systematically underestimate the true extropy values (negative bias) for finite $n$ . The bias-corrected estimators effectively remove this leading bias term, resulting in negligible bias even for small to moderate sample sizes.
Asymptotic Behavior: As $n \to \infty$ , the bias of the uncorrected estimators vanishes at a rate of $1/n$ , and the standard deviations of both corrected and uncorrected estimators decay at a rate of $1/\sqrt{n}$ . Consequently, the Root Mean Squared Error (RMSE) and the limit of the estimators are identical for both versions in the asymptotic regime.
Parameter Sensitivity: The magnitude of the finite-sample bias is shown to be dependent on $\alpha$ . Estimators with $\alpha = 0.5$ (emphasizing rare events) exhibit larger biases than those with $\alpha = 3$ (emphasizing dominant events), but the correction remains effective in both cases.

Significance and Claims
The paper claims that its primary significance lies in providing a theoretical justification for the use of simpler, uncorrected plug-in estimators in asymptotic settings (such as those involving almost sure convergence and asymptotic normality), as demonstrated in the companion paper [1]. The author argues that because the corrected and uncorrected versions are asymptotically equivalent, the simpler uncorrected estimators are sufficient for large-sample theoretical analysis.

However, for practical applications involving moderate sample sizes, the paper advocates for the use of the proposed bias-corrected estimators to ensure higher accuracy. The study emphasizes that the analytical approach offers a computationally efficient, interpretable, and parameter-free alternative to more complex bias reduction techniques like bootstrapping or Bayesian methods. The work concludes by noting that future research directions include extending these results to countably infinite alphabets and developing bootstrap-based variance estimators.

Technical Summary: Finite-Sample Bias Correction for Plug-in Estimators of Extropy Measures

More like this