📊 statistics

Asymptotics for estimating a diverging number of parameters -- with and without sparsity

This paper establishes a general asymptotic theory for estimating equations with a diverging number of parameters, providing conditions for existence, consistency, uniqueness, and asymptotic normality for both unpenalized and sparse penalized estimators under diverse data structures and complex penalty functions.

Original authors: Jana Gauss, Thomas Nagler

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jana Gauss, Thomas Nagler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for a single clue, you are sifting through a mountain of evidence that keeps growing every time you blink. In the world of statistics, this is the challenge of "high-dimensional data." Traditionally, scientists assumed they had a few suspects (parameters) and a huge pile of evidence (data points) to prove their case. But in the modern world, the number of suspects can sometimes explode, even outnumbering the evidence itself. This happens in everything from predicting stock market crashes to figuring out which genes cause a disease. The big question for statisticians is: when the number of variables gets huge, can we still trust our math to find the truth, or does the whole system collapse into chaos?

To make sense of this, we need to understand a few tools. First, there are "estimating equations," which are like a set of balance scales. You add up all your clues, and the goal is to find the setting where the scales balance perfectly at zero. If the scales balance, you've found your answer. Second, there is the concept of "sparsity." In a messy room with a thousand items, usually only a few are actually important, and the rest are just clutter. Sparsity is the idea that even if you have a million variables, only a tiny handful are the real "suspects," and the rest should be ignored. Finally, there are "penalties," which act like a strict librarian. If you try to include too many variables in your solution, the librarian slaps a fine on your hand, forcing you to keep your list short and focused.

For years, statisticians have had great rules for when there are few variables, and some rules for when there are many but the math is simple. But what happens when you have a million variables, the data is messy, the variables are connected in complex ways, and you are using a very strict librarian to keep things simple? That is the exact storm this paper sets out to navigate.

The authors, Jana Gauss and Thomas Nagler, have built a new, super-flexible map for this territory. They developed a general theory that tells us exactly when our statistical detective work will succeed, even when the number of variables grows as fast as the amount of data. They didn't just look at one specific type of problem; they created a universal framework that works for "unpenalized" problems (where we just balance the scales) and "penalized" problems (where we use the strict librarian).

Here is what they found. First, they proved that under certain conditions, a solution actually exists and is unique. It's not just a guess; they showed that if the data behaves in a specific way, there is one and only one correct answer hiding in the noise. Second, they showed that this answer gets closer and closer to the truth as we gather more data. This is called "consistency." Third, and perhaps most importantly, they proved that when we use these "penalties" to find the sparse truth, our method can correctly identify which variables are the real suspects and which are just noise. This is called "selection consistency." They even showed that for certain types of penalties, the method becomes as efficient as if we had known the answer all along (a property called the "oracle property").

However, the paper also explicitly rules out some old ideas that people used to rely on. For a long time, statisticians thought a condition called "Restricted Strong Convexity" (RSC) was necessary to guarantee these results. The authors found a simple example where this old condition fails completely, yet their new, weaker conditions still work perfectly. They showed that the old, stricter rules were too demanding and missed many real-world scenarios where the math still works. They also clarified that while some penalties (like the Lasso) are great at finding the right variables, they might not be the most efficient at estimating the exact size of those variables, whereas other penalties (like SCAD) can do both jobs perfectly.

The beauty of this work is that it doesn't just work for clean, perfect data. The authors extended their theory to handle data that is dependent, like a chain of events where one thing influences the next, or data that comes from different sources with different rules. They even applied this to "stepwise" procedures, where you solve a problem in many small steps, and showed that even if the number of steps grows huge, the math still holds up. They demonstrated this with real-world examples, like analyzing networks of connected people, estimating causal effects in medicine, and optimizing investment portfolios.

In short, this paper provides the rigorous mathematical backbone for trusting our statistical tools in the most complex, messy, and high-stakes scenarios imaginable. It tells us that as long as we use the right kind of "librarian" (penalty) and the data isn't too chaotic, we can find the needle in the haystack, even if the haystack is the size of a planet and keeps getting bigger. The authors didn't just suggest this might work; they proved it with theorems, giving us a solid foundation to build the next generation of data science.

Technical Summary: Asymptotics for Estimating a Diverging Number of Parameters — with and without Sparsity

Problem Statement

Modern statistical applications frequently involve high-dimensional data where the number of parameters $p_n$ grows with the sample size $n$ , and in sparse settings, $p_n$ may even exceed $n$ . Traditional fixed- $p$ large- $n$ asymptotic theory is inadequate for these scenarios. While existing literature has established asymptotic results for specific models (e.g., linear models, generalized linear models) or specific penalty structures (e.g., convex, coordinate-separable), a unified theoretical framework is lacking for:

General estimating equations that are not necessarily gradients of a single loss function.
Procedures involving a diverging number of estimation steps (e.g., stepwise estimation).
Data that are independent but non-identically distributed (i.i.d. is not assumed) or dependent (e.g., time series, network data).
Penalized estimation using penalties that are simultaneously nonconvex, non-coordinate-separable, and group-structured (e.g., Group SCAD).

The authors address the fundamental question: Under what conditions on the data-generating process, the estimating function $\phi$ , and the growth of $p_n$ does an estimator solving the estimating equation exist, and when is it consistent, unique, and asymptotically normal?

Methodology

The paper develops a general asymptotic theory for estimators $\hat{\theta}$ solving estimating equations of the form:
$\frac{1}{n} \sum_{i=1}^n \phi(X_i; \hat{\theta}) = 0$
and their penalized counterparts:
$\frac{1}{n} \sum_{i=1}^n \phi(X_i; \hat{\theta}) \in \partial p_{\lambda_n}(\hat{\theta})$
where $\partial p_{\lambda_n}$ denotes the generalized gradient of a penalty function.

Core Approach

Instead of deriving separate arguments for specific models, the authors formulate verifiable population-level conditions on the estimating equation and its curvature. Key methodological features include:

Population-Level Curvature Conditions: The theory relies on conditions (A1, A4) that impose a form of concavity (or negative definiteness) on the expected curvature of the estimating equation. This is controlled by a sequence of symmetric matrix-valued functions $H_n(x)$ that act as envelopes for the symmetrized Jacobians. Unlike previous works that often require strong Restricted Strong Convexity (RSC) on the realized sample Hessian, this approach uses weaker conditions on the population expectation.
Generalized Gradients: For penalized problems, the authors utilize Clarke's generalized gradient to handle non-differentiable and nonconvex penalties, allowing for a unified treatment of Lasso, SCAD, MCP, and group-structured penalties.
Dependent Data Extension: The framework extends to dependent data (e.g., time series, network dependence) via a blocking and coupling argument. By partitioning data into blocks and assuming weak dependence (measured by $\beta$ -mixing coefficients) between retained blocks, the authors show that the dependent problem can be approximated by an independent one with negligible error.
Stepwise and Multi-Sample Procedures: The theory treats sequential estimation steps or multi-sample problems as a single high-dimensional estimating equation, allowing the number of steps or samples to diverge with $n$ .

Key Contributions and Results

1. Unpenalized Estimation

The authors establish conditions for:

Existence and Consistency (Theorem 1): Solutions exist in a shrinking neighborhood of the true parameter $\theta^*$ with probability tending to 1. The convergence rate is $r_n = \sqrt{\text{tr}(I_p(\theta^*))/n}$ , typically of order $\sqrt{p_n/n}$ .
Uniqueness (Theorem 2): Under strengthened curvature conditions, the solution is unique within the neighborhood.
Asymptotic Normality (Theorem 3): Finite-dimensional projections of the estimator are asymptotically normal.
Growth Rates: For Generalized Linear Models (GLMs), the theory yields the condition $p_n \ln p_n / n \to 0$ for consistency, matching the sharpest known results (Portnoy, 1984). For asymptotic normality, it requires $p_n^2/n \to 0$ (or slightly weaker $p_n^{3/2} \ln n / n \to 0$ under specific linear structures), which is an improvement over the $p_n^5/n \to 0$ required by Fan and Peng (2004) for general nonlinear MLE.

2. Penalized Estimation

For sparse problems where $p_n > n$ :

Consistency and Selection (Theorems 4 & 5): The authors prove that a solution exists which is both estimation consistent and selection consistent (i.e., $\hat{\theta}^{(2)} = 0$ ). The convergence rate depends on the sparsity level $s_n$ and the bias induced by the penalty.
Generalized Mutual Incoherence: Assumption (A6) generalizes the mutual incoherence and irrepresentable conditions to non-coordinate-separable and group-structured penalties. It ensures that the influence of active variables on inactive equations is sufficiently small to allow the penalty to zero out the inactive coefficients.
Oracle Property (Theorem 7): For nonconvex penalties like SCAD and MCP, the estimator achieves the "oracle property": it behaves asymptotically as if the true support were known in advance, attaining the same efficiency as the unpenalized estimator on the active set.
Uniqueness (Theorem 6): Conditions are provided under which the penalized solution is unique, relying on a generalized version of the $\mu$ -amenability condition.

3. Dependent Data and Stepwise Procedures

Dependent Data (Theorem 8): The results are extended to dependent data (e.g., $\beta$ -mixing time series, network-dependent data) by introducing a blocking scheme. The theory shows that if the dependence decays sufficiently fast, the asymptotic behavior mirrors the independent case, with the covariance matrix adjusted for the dependence structure.
Diverging Steps: The framework accommodates stepwise procedures where the number of steps $K_n \to \infty$ , provided the cumulative error from previous steps is controlled.

4. Counter-Example to Strong RSC

The paper provides a simple quadratic allocation problem (Section 5.2) where the commonly used strong RSC condition (requiring a lower bound on the quadratic form for all sparse vectors) fails with high probability due to the rank structure of the sample Hessian. However, the authors' weaker population-level condition holds, demonstrating that their framework covers scenarios where existing non-asymptotic theories (e.g., Negahban et al., 2012; Loh and Wainwright, 2015) do not apply.

Significance and Claims

The authors claim that their work provides a unified asymptotic theory that substantially extends the applicability of existing results in three directions:

Generality: It applies to general estimating equations not derived from risk minimization, including stepwise and multi-sample procedures.
Penalty Flexibility: It covers penalties that are simultaneously nonconvex and group-structured (e.g., Group SCAD), a combination not previously covered by general theory.
Data Flexibility: It handles independent non-identically distributed observations and various forms of dependence without requiring the strong sample-level RSC conditions that often fail in high-dimensional settings.

The paper emphasizes that by formulating regularity conditions at the population level, the applicability of the theory is more transparent, and the resulting distributional limits provide a tight characterization of estimation uncertainty. The results are illustrated through applications including Group SCAD in GLMs, distributed inference under network dependence, and penalized stepwise estimation in causal inference, showing that the framework recovers known results while providing new guarantees for complex, high-dimensional procedures.