Sub-Gaussian Concentration and Entropic Normality of the Maximum Likelihood Estimator

Technical Summary: Sub-Gaussian Concentration and Entropic Normality of the Maximum Likelihood Estimator

Problem Definition
This paper investigates the asymptotic behavior of the Maximum Likelihood Estimator (MLE) for a parameter $\theta \in \Theta \subset \mathbb{R}$ based on i.i.d. samples $X_1, \dots, X_n$ drawn from a density $f(x|\theta)$. The primary object of study is the normalized estimation error $Z_n = \sqrt{n}(\hat{\theta}_n - \theta_0)$, where $\hat{\theta}_n$ is the MLE and $\theta_0$ is the true parameter.

While classical statistical theory establishes that $Z_n$ converges in distribution to a standard Gaussian random variable $Z \sim \mathcal{N}(0,1)$ under standard regularity conditions, this paper seeks to strengthen this result. Specifically, the authors aim to upgrade convergence in distribution to stronger modes of convergence, including convergence of moments and convergence in relative entropy (Kullback-Leibler divergence).

Methodology and Assumptions
The authors employ a method-of-moments approach combined with entropy-control arguments, rather than relying on convolution or entropy-monotonicity methods typically used for sums of independent variables (which do not directly apply to the MLE due to its non-linear dependence on the data).

To achieve these stronger results, the paper introduces three specific regularity conditions beyond the standard MLE assumptions:

1. Sub-Gaussian Lipschitz Envelope (Assumption 1): The log-likelihood difference is bounded by a sub-Gaussian random variable $H(X)$ times the parameter difference. This ensures exponential consistency and controls the tail behavior of the score function.
2. Sub-Exponential Control of Score Derivative (Assumption 2): The derivative of the score function satisfies a Lipschitz condition controlled by a sub-exponential random variable $\bar{H}(X)$. This assumption is crucial for establishing high-moment bounds and controlling the second-order Taylor expansion of the likelihood.
3. Smoothness of the Estimator Density (Assumption 3): To remove a smoothing step required for the main entropy result, the paper assumes either that the Fisher information of the density of $Z_n$ is uniformly bounded or that the first derivative of the density is uniformly bounded. This rules out singular behaviors, such as discrete atoms on the boundary of the parameter space, which would otherwise cause the relative entropy to be infinite.

Key Contributions and Results
The paper establishes three main theorems:

1. Sub-Gaussian Concentration and Moment Convergence (Theorem 1): Under Assumptions 1 and 2, the normalized error $Z_n$ is shown to be $O(1)$ sub-Gaussian. Consequently, the paper proves that all moments of $Z_n$ converge to the corresponding moments of the limiting Gaussian distribution ($E|Z_n|^m \to E|Z|^m$ for all $m \ge 1$). This is a significant strengthening of the classical Central Limit Theorem (CLT), which only guarantees convergence in distribution.

2. Entropic CLT for Smoothed Estimator (Theorem 2): The authors prove that a smoothed version of the estimator, defined as $\tilde{Z}_n = \sqrt{1-\epsilon}Z_n + \sqrt{\epsilon}Z$ (where $Z$ is an independent standard normal), converges in relative entropy to the Gaussian limit. That is, $\lim_{n \to \infty} D(\tilde{Z}_n \| Z) = 0$. This result holds under Assumptions 1 and 2. The smoothing is necessary because the raw MLE may place probability mass on the boundary of the compact parameter space, leading to a distribution that is not absolutely continuous with respect to the Gaussian limit.

3. Entropic Normality of the MLE (Corollary 1): By invoking Assumption 3 (bounded Fisher information or bounded density derivative), the authors demonstrate that the smoothing parameter $\epsilon$ can be removed. This yields the main result: the unsmoothed MLE error $Z_n$ itself converges in relative entropy to the standard Gaussian, i.e., $\lim_{n \to \infty} D(Z_n \| Z) = 0$.

Significance and Scope
The paper claims that convergence in relative entropy is a significantly stronger mode of convergence than convergence in distribution. By Pinsker's inequality, it implies convergence in total variation distance, and by Talagrand's inequality, it implies convergence in Wasserstein distance.

The authors note that while the entropic CLT is well-known for sample means and central order statistics, extending these results to the MLE is non-trivial because the MLE is not generally a sum of independent variables. The paper provides a framework for establishing entropic normality for a broad class of statistical models, including those outside the exponential family (e.g., Pearson Type IV, Logistic, and Cauchy models with unknown scale), provided the score function and its derivatives satisfy the specified boundedness and tail conditions.

The work distinguishes itself from prior literature on MLEs in learning theory, which often focuses on the KL divergence between distributions induced by estimated and true parameters over the sample space. In contrast, this paper focuses on the convergence of the distribution of the estimator itself to a Gaussian law. The proofs rely on auxiliary tools, including exponential consistency bounds and high-moment estimates, which the authors suggest may be of independent interest.