📊 statistics

Existence of penalised likelihood estimates and posterior propriety of separable prior distributions for Gaussian precision matrices

This paper establishes specific tail conditions on diagonal and off-diagonal penalty functions that guarantee the existence of penalised likelihood estimates for Gaussian precision matrices with positive semidefinite sample covariance, and extends these findings to derive conditions ensuring the propriety of posterior distributions under separable priors.

Original authors: Jack Storror Carter

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jack Storror Carter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data science, researchers often face a puzzle that looks like a giant, tangled web of connections. Imagine trying to understand how hundreds of different variables—perhaps stock prices, weather patterns, or gene expressions—relate to one another. To map these relationships, statisticians use a mathematical tool called a precision matrix. Think of this matrix as a master blueprint that reveals which variables are truly connected and which are merely coincidental. The challenge arises when the number of variables is larger than the number of observations available. In such high-dimensional situations, the data becomes too sparse to build a standard blueprint; the usual mathematical methods break down, and the answer simply vanishes. This is a common hurdle in modern science, where datasets grow faster than the ability to collect enough samples to measure them reliably.

To solve this, scientists have developed a technique called penalized likelihood. Instead of just looking for the most likely blueprint based on the data, they add a "penalty" to the calculation. This penalty acts like a rule that discourages the model from creating unnecessary or overly complex connections, effectively forcing the blueprint to be sparse and manageable. It is a bit like a sculptor who, rather than carving every possible detail, is given a rule to remove excess stone, ensuring the final statue stands firm even if the raw material is imperfect. This approach has become a standard way to find structure in noisy, high-dimensional data. However, a critical question remained: does this method actually work when the data is so sparse that the standard blueprint cannot be built at all?

Jack Storror Carter, working at the Universitat Pompeu Fabra and the Barcelona School of Economics, set out to answer this question with mathematical precision. The paper investigates the conditions under which these penalized estimates can actually exist when the data is insufficient to form a complete picture. The researcher focused on a specific type of penalty that treats the diagonal elements of the matrix (which represent the strength of individual variables) differently from the off-diagonal elements (which represent the connections between variables). By analyzing the behavior of these penalties as the numbers involved grow very large or very small, Carter mapped out exactly when a solution is guaranteed to exist and when it is mathematically impossible.

The findings reveal a delicate balance required to keep the solution alive. When the data is so sparse that the standard method fails, the penalty applied to the diagonal elements must grow fast enough to counteract the instability caused by the missing information. Specifically, the paper proves that if the penalty on the diagonal grows faster than the logarithm of the value itself, a solution is guaranteed to exist for any type of sparse data. If the penalty grows too slowly, the mathematical model collapses, and no valid blueprint can be found. This is a strict requirement; the paper shows that without this specific growth rate, the estimate simply does not exist for certain types of sparse data, regardless of how clever the algorithm might be.

The study also explored what happens when penalties are applied only to the connections between variables, ignoring the individual strengths. In this scenario, the paper demonstrates that a solution can only exist if the data has strictly positive values on its diagonal. If even a single variable in the dataset has a value of zero, the entire estimation process fails. This is a significant constraint, as it means that methods relying solely on penalizing connections are fragile and cannot handle the most extreme cases of missing data. However, the research offers a path forward: by combining a strong penalty on the individual variables with a penalty on the connections, researchers can ensure a solution exists even when the data is extremely sparse. The paper provides a precise formula for how these two penalties must work together, showing that their combined strength must exceed a specific threshold determined by the number of missing pieces in the data.

Beyond the existence of the estimate, the paper extends these findings into the realm of Bayesian statistics, where the goal is not just to find a single best answer but to understand the entire range of possible answers. In this framework, the penalty functions correspond to prior beliefs about the data. The author establishes conditions under which these Bayesian models produce a "proper" posterior distribution, meaning the total probability of all possible outcomes adds up to a finite, sensible number. If the penalties are too weak, the model becomes unmoored, and the probabilities spread out infinitely, rendering the analysis useless. The paper proves that by choosing penalties that grow sufficiently fast, researchers can ensure their Bayesian models remain grounded and mathematically sound, even in the most difficult high-dimensional settings.

The implications of this work are practical and immediate for anyone working with complex data. The paper does not propose a new algorithm to replace existing ones but rather provides a rigorous safety net. It tells data scientists exactly which penalty functions are safe to use and which will lead to mathematical dead ends. For instance, it clarifies that popular methods designed to create sparse models, such as those using specific non-convex penalties, can fail silently if the data is too sparse and the diagonal penalty is not strong enough. By following the conditions laid out in the paper, researchers can select penalty functions that guarantee a solution will be found, ensuring their models are robust enough to handle the realities of modern, high-dimensional data collection. The work essentially draws a map of the mathematical terrain, showing where the ground is solid and where it is too shaky to build a model, allowing scientists to navigate the complexities of sparse data with confidence.

Technical Summary: Existence of Penalised Likelihood Estimates and Posterior Propriety of Separable Prior Distributions for Gaussian Precision Matrices

Problem Statement
In high-dimensional settings where the dimension of a Gaussian precision matrix $\Theta$ ( $p$ ) exceeds the sample size ( $n$ ), the sample covariance matrix $S$ is not of full rank (it is positive semidefinite, PSD, but not positive definite, PD). Consequently, the standard maximum likelihood estimate (MLE) of $\Theta$ , given by $S^{-1}$ , does not exist. Penalised likelihood methods are frequently employed to induce sparsity and ensure estimation stability in such scenarios. However, the existence of penalised likelihood estimates $\hat{\Theta}_{Pen}$ is not guaranteed for all PSD $S$ , particularly when the penalty function lacks sufficient growth at the boundaries of the parameter space. Furthermore, in the Bayesian framework, while the Maximum A-Posteriori (MAP) estimate corresponds to the penalised likelihood estimate, the existence of the MAP does not automatically imply that the posterior distribution is proper (i.e., has a finite integral). This paper addresses the specific tail conditions required on separable penalty functions to ensure the existence of $\hat{\Theta}_{Pen}$ for any PSD $S$ and establishes conditions for the propriety of the resulting posterior distribution.

Methodology
The paper analyzes the optimisation problem:
$\hat{\Theta}_{Pen} = \arg\max_{\Theta \succ 0} \left( \log \det \Theta - \text{tr}(S\Theta) - \text{Pen}(\Theta) \right)$
where the penalty function is assumed to be separable: $\text{Pen}(\Theta) = \sum_{i} \text{pen}_D(\theta_{ii}) + \sum_{i<j} \text{pen}(\theta_{ij})$ . The analysis focuses on the limiting behavior of the objective function as $\Theta$ approaches the boundary of the cone of positive definite matrices (where eigenvalues approach 0) or as $\|\Theta\| \to \infty$ .

The methodology relies on spectral decomposition, expressing the log-likelihood in terms of the eigenvalues $\lambda_k$ and eigenvectors $v_k$ of $\Theta$ . The author constructs specific sequences of matrices $\Theta(t)$ to test for non-existence (by showing the objective function tends to $+\infty$ ) and prove existence (by showing the objective function tends to $-\infty$ ). Key to the proofs is the characterization of the null space of $S$ , denoted $N(S)$ , with dimension $m_0$ . The analysis distinguishes between sequences where eigenvectors converge to the null space of $S$ and those that do not, deriving necessary and sufficient conditions on the growth rates of $\text{pen}_D$ and $\text{pen}$ relative to the logarithmic term $\log x$ .

Key Contributions and Results

Existence Conditions for Diagonal Penalties:
- If the diagonal penalty $\text{pen}_D$ satisfies $\lim_{x\to\infty} (\text{pen}_D(x) - \log x) = \infty$ , the estimate $\hat{\Theta}_{Pen}$ exists for any PSD $S$ , regardless of the off-diagonal penalty (provided it is non-negative).
- If $S$ is a Gaussian sample covariance matrix with $m_0$ zero eigenvalues, a weaker condition suffices for existence with probability 1: $\liminf_{x\to\infty} \frac{\text{pen}_D(x)}{\log x} > \frac{m_0}{p}$ .
- Conversely, if $\text{pen} = 0$ and $\liminf_{x\to\infty} (p \cdot \text{pen}_D(x) - \log x) = -\infty$ , the estimate does not exist for any only-PSD $S$ .
Existence Conditions for Off-Diagonal Penalties:
- If the diagonal penalty is absent ( $\text{pen}_D = 0$ ), existence requires that $S$ has strictly positive diagonal entries. If $S$ has a zero diagonal entry, the estimate does not exist.
- When $S$ has strictly positive diagonals and $m_0$ zero eigenvalues, existence is guaranteed if $\liminf_{|x|\to\infty} \frac{\text{pen}(x)}{\log |x|} > m_0$ .
- For existence across all PSD $S$ with positive diagonals (worst-case rank), the condition is $\liminf_{|x|\to\infty} \frac{\text{pen}(x)}{\log |x|} > p-1$ .
- If $\text{pen}_D = 0$ and $\liminf_{|x|\to\infty} (q \cdot \text{pen}(x) - \log |x|) = -\infty$ (where $q = p(p-1)/2$ ), the estimate does not exist for any only-PSD $S$ .
Joint Penalties:
- The paper demonstrates that diagonal and off-diagonal penalties can compensate for each other. If $S$ has $m_0$ zero eigenvalues and strictly positive diagonals, existence is ensured if:
  $\liminf_{x\to\infty} \frac{\text{pen}_D(x)}{\log x} + \liminf_{|x|\to\infty} \frac{\text{pen}(x)}{\log |x|} > m_0$
- This allows for the use of off-diagonal penalties with weak tail growth (e.g., bounded penalties like SCAD or MCP, or $L_0$ approximations) provided they are combined with a sufficiently strong diagonal penalty.
Posterior Propriety:
- The paper extends these findings to the Bayesian setting. While the MAP estimate exists under the conditions above, posterior propriety (finite integral) requires stronger conditions.
- For PD $S$ , propriety holds if the off-diagonal prior $\pi$ is bounded and the diagonal prior $\pi_D$ satisfies specific integrability conditions near zero and infinity.
- For PSD $S$ , a sufficient condition for propriety is that the off-diagonal prior $\pi$ is proper (integrable) and the diagonal prior $\pi_D$ has a finite $n/2$ moment.
Generalisations:
- The results are shown to extend to unequal penalties across entries, non-symmetric off-diagonal penalties, and penalties that are not strictly increasing in magnitude.
- The paper also addresses penalties on partial correlations, noting that unlike standard separable penalties, penalising partial correlations alone is insufficient to ensure existence for PSD $S$ ; a diagonal penalty remains necessary.

Significance and Claims
The paper claims to provide a "road map" for determining the existence of penalised likelihood estimates for almost any choice of separable penalty functions. Its primary significance lies in resolving the tension between statistical model selection (which often favors off-diagonal penalties with weak tails, such as SCAD, MCP, or seamless $L_0$ , to reduce bias) and mathematical existence in high-dimensional settings ( $n \le p$ ).

The author argues that while weak off-diagonal penalties alone fail to guarantee existence when $S$ is only PSD, they can be successfully utilized in high-dimensional settings if combined with a suitably strong diagonal penalty. This combination ensures the existence of the estimate for any PSD $S$ without requiring the sample size to exceed the dimension. The paper modestly notes that while these conditions ensure existence, the choice of diagonal penalty also impacts estimation performance and sparsity, suggesting that the derived conditions serve as minimal requirements for existence rather than optimal tuning for statistical performance. Additionally, the work clarifies the distinction between the existence of the MAP estimate and the propriety of the full posterior distribution, providing explicit conditions for the latter in both PD and PSD scenarios.

More like this