Estimation of differential entropy for normal populations under prior information

This paper addresses the point-wise and interval estimation of differential entropy for two normal populations under order-restricted prior information by deriving improved estimators that dominate the best affine equivariant estimator and comparing various confidence interval methods through numerical studies and a real-world application.

The Gist

Imagine you are a detective trying to solve a mystery about uncertainty. In the world of statistics, this "uncertainty" is called Entropy. Think of entropy as a measure of how messy or chaotic a system is. If you have a perfectly organized library, the entropy is low. If you have a room where books are thrown everywhere, the entropy is high.

This paper is about two detectives (statisticians) trying to measure the "messiness" (entropy) of two different groups of data that follow a Normal Distribution (the famous Bell Curve). Think of these two groups as two different factories producing lightbulbs. Factory A and Factory B both make bulbs, but they might have slightly different average lifespans (means), though they share the same level of consistency (variance).

Here is the twist: The detectives have a clue (prior information). They know for a fact that Factory A's bulbs last at least as long as Factory B's, or perhaps Factory B's are the "better" ones ($\mu_1 \le \mu_2$). The paper asks: How can we use this clue to guess the "messiness" of the system more accurately than if we ignored the clue?

The Main Characters and Tools

1. The Goal: Estimate the "messiness" (Entropy) of the two factories.
2. The Old Way (The "Naive" Detective): Usually, detectives use a standard tool called the Maximum Likelihood Estimator (MLE). It's like guessing the average height of a crowd by just measuring a few people and doing a quick math calculation. It's okay, but it's not perfect.
3. The New Way (The "Smart" Detective): The authors propose new, smarter tools that take the clue ($\mu_1 \le \mu_2$) into account. They call these Improved Estimators.

The Analogy: The "Bowl" and the "Slippery Slope"

To understand how they improved the guess, imagine a bowl (a loss function).

• If you guess the wrong amount of messiness, you fall into the bowl. The deeper you fall, the worse your guess was.
• The "Standard Detective" (BAEE) stands at a fixed spot in the bowl. They are good, but they don't know about the clue.
• The "Smart Detective" looks at the clue. They realize that if Factory A is actually better than Factory B, they should shift their position in the bowl slightly to avoid falling as deep.

The paper proves mathematically that by shifting your guess based on the clue, you always end up with a shallower fall (less error) than the Standard Detective, no matter what the true situation is.

The Three Types of "Smart" Guesses

The paper introduces three main ways to make these better guesses:

1. The "Restricted" Guess: This is like a rulebook. "If the data looks like Factory A is worse than B, ignore that and assume they are equal." It's a simple fix that works well.
2. The "Smooth" Guess: This is a more sophisticated version. Instead of a hard rule, it gently nudges the guess in the right direction depending on how strong the evidence is. It's like a self-driving car that gently steers toward the correct lane rather than slamming the brakes.
3. The "Pitman" Guess: This is a different way of judging success. Instead of just looking at the average error, it asks: "Is my guess closer to the truth than the other guy's guess more than half the time?" The paper shows their new guesses win this race too.

The Interval Problem: Drawing a Net

Estimating a single number is hard, but sometimes you need a range (a net) to catch the true value. The paper tries to draw the smallest possible net that still catches the truth 95% of the time.

They tested four different ways to draw this net:

• The Asymptotic Net: A quick, standard calculation.
• The Bootstrap Net: A method where the computer simulates thousands of fake factories to see what happens.
• The Generalized Net: A complex mathematical trick to find the perfect range.
• The HPD Net: A Bayesian method using a computer simulation (MCMC) to find the "most likely" range.

The Verdict: The simulation showed that the Generalized Net and the Bootstrap-t Net were the best. They caught the truth most often (high coverage) without being unnecessarily wide (short length).

The Real-World Test: Airplane AC Units

To prove their math works in real life, the authors looked at data from Boeing 720 jet planes. They analyzed the failure times of air-conditioning systems on two different planes.

• They checked if the data looked like a Bell Curve (it did).
• They checked if one plane's AC was more reliable than the other (it was).
• They applied their new formulas.

Result: Their new formulas gave a slightly different (and theoretically better) estimate of the system's uncertainty compared to the old standard methods.

Why Does This Matter?

In the real world, we often have extra information.

• Medicine: We know a new drug can't be worse than the placebo.
• Economics: We know a stock price can't be negative.
• Engineering: We know a bridge must hold at least a certain weight.

This paper teaches us that ignoring these known facts is a waste. By building them into our math, we can make predictions that are more accurate, safer, and more efficient. It's the difference between guessing the weather based on a random hunch versus looking at the barometer and knowing the wind direction.

In short: The paper gives statisticians a new set of "smart glasses" that let them see the truth more clearly by using clues they already have, ensuring their guesses are the best they can possibly be.

Technical Summary

Here is a detailed technical summary of the paper "Estimation of differential entropy for normal populations under prior information" by Somnath Mandal and Lakshmi Kanta Patra.

1. Problem Statement

The paper addresses the statistical problem of estimating the differential entropy of two independent normal populations, $N(\mu_1, \sigma^2)$ and $N(\mu_2, \sigma^2)$, subject to a prior order restriction on the means: $\mu_1 \leq \mu_2$.

• Target Parameter: The Shannon entropy for a normal distribution is $H(\sigma) = 1 + \ln(2\pi) + 2\ln\sigma$. Estimating $H(\sigma)$ is mathematically equivalent to estimating the parameter $\tau = \ln\sigma$.
• Context: In many real-world scenarios (e.g., reliability, molecular sciences), prior knowledge suggests an ordering of parameters. Ignoring this information leads to suboptimal estimators. The authors aim to utilize the restriction $\mu_1 \leq \mu_2$ to derive estimators that dominate standard unbiased or equivariant estimators.
• Loss Function: The study considers a general location-invariant loss function $L(t)$ satisfying strict convexity and differentiability conditions. Specific cases analyzed include the Squared Error Loss ($L_1(t) = t^2$) and the Linex Loss ($L_2(t) = e^{a_1 t} - a_1 t - 1$).

2. Methodology and Theoretical Framework

The authors employ a decision-theoretic approach to derive improved estimators and confidence intervals.

A. Point Estimation

1. Baseline Estimators:

   • MLE and Restricted MLE (RMLE): Derived based on sufficient statistics $(X, S^2)$.
   • UMVUE: The Uniformly Minimum Variance Unbiased Estimator for $\tau$.
   • Best Affine Equivariant Estimator (BAEE): Under the general loss function, the BAEE is of the form $\delta_0(X, S) = \ln S + d_0$, where $d_0$ is a constant determined by the loss function.

2. Improvement over BAEE (Non-Smooth):

   • The authors define a class of estimators $\delta_\phi = \ln\sqrt{V} + \phi(W)$, where $W$ is a statistic derived from the difference of sample means and the pooled variance.
   • Using conditional risk analysis, they prove that the BAEE can be improved by modifying the constant $d_0$ based on the value of $W$.
   • Theorem 2.1: An estimator $\delta_S$ is derived that dominates the BAEE. It involves a "min-max" structure depending on the sign of $W$:
     $$\delta_S = \ln\sqrt{V} + \begin{cases} \max\{d_0, m_0 + \ln\sqrt{1 + nW^2/2}\} & W < 0 \\ \min\{d_0, m_0 + \ln\sqrt{1 + nW^2/2}\} & W > 0 \end{cases}$$
     where $m_0$ is a specific constant derived from the loss function.

3. Improvement over BAEE (Smooth):

   • To overcome the non-smoothness of the previous estimator, the authors apply the Brewster and Zidek technique.
   • They derive a smooth improved estimator $\delta_{SE}$ which is also minimax. This estimator uses a continuous function $\psi^*(W)$ that adjusts the shift term based on the magnitude of $W$.
   • Connection to IERD: The paper demonstrates that these smooth estimators coincide with the Integral Expression of Risk Difference (IERD) type estimators proposed by Kubokawa, unifying two different derivation techniques.

4. Generalized Pitman Closeness (GPC):

   • The authors extend the analysis to the GPC criterion, which measures the probability that one estimator is closer to the true parameter than another.
   • They derive an estimator $\delta_{\psi^*}$ that is "Pitman closest" to the true $\ln\sigma$ compared to the affine equivariant estimator, utilizing median-based truncation rules.

B. Interval Estimation

The paper proposes four distinct methods for constructing confidence intervals (CIs) for $\ln\sigma$:

1. Asymptotic CI: Derived using the Delta method applied to the MLE.
2. Bootstrap CIs:
   • Bootstrap-p: Based on the percentile of the bootstrap distribution.
   • Bootstrap-t: Based on the studentized statistic.
3. Generalized Confidence Interval (GCI): Constructed using the Generalized Pivotal Quantity (GPQ) approach based on the UMVUE.
4. HPD Credible Interval: Derived using Markov Chain Monte Carlo (MCMC) methods (Gibbs sampling and Random Walk Metropolis-Hastings) under Jeffreys' non-informative prior.

3. Key Contributions

1. Derivation of Dominating Estimators: The paper provides explicit formulas for estimators that strictly dominate the Best Affine Equivariant Estimator (BAEE) under the order restriction $\mu_1 \leq \mu_2$. This holds for both quadratic and Linex loss functions.
2. Unification of Techniques: It establishes a theoretical link between the Brewster-Zidek smoothing technique and the Kubokawa IERD method, showing they yield equivalent classes of improved estimators in this context.
3. Comprehensive Interval Comparison: It introduces and compares four different interval estimation methodologies (Asymptotic, Bootstrap, GCI, HPD) specifically for differential entropy under order restrictions, a gap in existing literature.
4. Unified Performance Metric: The authors propose using Probability Coverage Density (PCD) (the ratio of Coverage Probability to Average Length) as a unified criterion to rank interval estimators, balancing the trade-off between precision (length) and reliability (coverage).

4. Results and Numerical Findings

The authors conducted extensive Monte Carlo simulations (70,000 samples for point estimation; 30,000 for interval estimation) to validate their theoretical findings.

Point Estimation Results:

• Risk Improvement: The proposed improved estimators ($\delta_S$ and $\delta_{SE}$) consistently outperform the BAEE and the standard MLE.
• Relative Risk Improvement (RRI):
  • The RRI is highest when the parameter difference $\eta = (\mu_2 - \mu_1)/\sigma$ is small (near the boundary of the restriction).
  • As $\eta$ increases (moving away from the restriction), the advantage of the improved estimators diminishes, converging to the performance of the unrestricted estimators.
  • The smooth estimators generally perform better in moderate regions of $\eta$, while non-smooth estimators excel near the boundary.
  • Risk improvement decreases as sample size $n$ increases.

Interval Estimation Results:

• Coverage Probability (CP) vs. Average Length (AL):
  • Bootstrap-t and Generalized CIs consistently achieve the nominal 95% coverage level.
  • Asymptotic CIs often have shorter lengths but fail to maintain the 95% coverage, especially for small sample sizes.
  • HPD CIs have very short lengths but can suffer from lower coverage probabilities.
• Ranking:
  • Shortest Length: Asymptotic < HPD < Bootstrap < Generalized.
  • Highest Coverage: Generalized > Bootstrap-t > HPD > Asymptotic.
  • Best Overall (PCD): The Generalized Confidence Interval and Bootstrap-t intervals generally rank highest when considering the PCD criterion, offering the best balance of accuracy and precision.

5. Significance and Application

• Theoretical Impact: The work advances the field of decision theory by solving the entropy estimation problem under order restrictions, a complex nonlinear functional estimation problem. It provides a rigorous framework for incorporating prior order information into entropy estimation.
• Practical Application: The paper validates its methods using a real-life dataset: failure times of air-conditioning systems on Boeing 720 jet planes. The analysis confirms that the data fits the normal distribution and that the order restriction holds, allowing for the calculation of improved entropy estimates and confidence intervals.
• Broader Relevance: Differential entropy is crucial in fields like signal processing, biology (DNA analysis), and economics. This paper provides statisticians with superior tools to estimate uncertainty in systems where parameters are known to be ordered, leading to more efficient resource allocation and better model selection.

In conclusion, the paper successfully bridges the gap between theoretical decision theory and practical application, offering a suite of improved estimators and robust interval methods for differential entropy in ordered normal populations.