High-dimensional bootstrap and asymptotic expansion

This paper develops an asymptotic expansion formula for bootstrap coverage probabilities in high dimensions to theoretically explain why third-moment matching wild bootstrap methods outperform normal approximations, demonstrating that such methods achieve second-order accuracy without studentization under specific covariance structures or via a double wild bootstrap approach.

The Gist

Here is an explanation of the paper "High-dimensional bootstrap and asymptotic expansion" by Yuta Koike, translated into simple, everyday language using analogies.

The Big Picture: The "Too Many Variables" Problem

Imagine you are a detective trying to solve a mystery. You have a list of suspects (data points), but the list is incredibly long. In fact, the number of suspects ($d$) is much larger than the number of clues you have ($n$).

In statistics, this is called the high-dimensional setting. Usually, when you have more variables than data points, standard statistical tools break down. It's like trying to bake a cake with 1,000 ingredients but only 5 eggs; the recipe just doesn't work.

However, a famous group of researchers (Chernozhukov, Chetverikov, and Kato) discovered a "magic trick" called the Bootstrap. This trick allows us to estimate the behavior of our data even when $d > n$. They showed that if you simulate the process many times (resampling), you can get a good approximation of the truth, similar to how a Gaussian (bell curve) distribution behaves.

The Mystery: Why Do Some Tricks Work Better?

The paper starts with a puzzle. The standard "magic trick" (Gaussian Wild Bootstrap) works, but it's not perfect. It has a small error.

Then, researchers noticed something weird in computer experiments: If you tweak the trick to match the third moment (a fancy way of saying "skewness" or the asymmetry of the data), the results become much more accurate. It's as if the new trick is a master chef, while the old one is just a decent cook.

The Problem: The existing math theories couldn't explain why this new trick was so much better. The theories said they should be roughly the same.

The Solution: A New Map (Asymptotic Expansion)

Yuta Koike, the author of this paper, decided to build a more detailed map of the statistical landscape. Instead of just looking at the destination (the final answer), he looked at the journey step-by-step using something called an Asymptotic Expansion.

Think of it like this:

• Normal Approximation: "The destination is roughly 10 miles away." (Good enough for a rough guess).
• Asymptotic Expansion: "The destination is 10 miles, 3 feet, and 2 inches away, and there's a slight hill on the left." (Extremely precise).

By using this detailed map, Koike could see exactly where the errors were coming from.

The "Blessing of Dimensionality"

Here is the most surprising discovery in the paper. Usually, having too many variables is a curse. But Koike found that in this specific high-dimensional context, having more variables actually helps the "third-moment matching" trick.

The Analogy:
Imagine you are trying to balance a long, wobbly pole.

• Low Dimension (Short pole): If you try to balance a short stick, it's easy to tip over if you don't account for every tiny wobble.
• High Dimension (Long pole): If you have a massive, long pole with many sections, the wobbles in one section tend to cancel out the wobbles in another. The sheer size and complexity of the system actually make it more stable if you use the right technique.

Koike proved that if the data has certain properties (like equal variance across all variables), the "third-moment matching" bootstrap becomes second-order accurate. This is a technical way of saying: "The error is so small it's almost invisible, even without doing extra complex adjustments."

The "Double Bootstrap" Safety Net

The paper also introduces a backup plan called the Double Wild Bootstrap.

The Analogy:
Imagine you are playing a game where you have to guess the weight of a mystery box.

1. First Bootstrap: You guess the weight based on your first set of simulations.
2. Double Bootstrap: You realize your first guess might be slightly off. So, you run a second set of simulations to check your first guess. You are essentially "simulating the simulation."

Koike showed that this "double-check" method works perfectly, no matter how messy or weird the data structure is. It guarantees high accuracy even when the "Blessing of Dimensionality" doesn't apply.

The Secret Weapon: Stein Kernels

To prove all this, the author had to use a very advanced mathematical tool called a Stein Kernel.

The Analogy:
Imagine you are trying to describe a complex 3D shape (like a cloud) to someone who has never seen it.

• Standard Math: Tries to describe every single point on the cloud. Impossible.
• Stein Kernel: Instead of describing the points, it describes the "force" or "tension" inside the cloud. It tells you how the shape reacts if you push it.

Koike used this tool to handle the fact that in high dimensions, the data doesn't always behave like a smooth, perfect bell curve. The Stein Kernel allowed him to prove that his detailed map (the expansion) was valid even when the data was "rough" or "singular."

Summary of the Takeaways

1. The Puzzle: Why does a specific type of statistical simulation work better than expected in high dimensions?
2. The Discovery: It works better because high dimensions can actually stabilize the error, provided the data is balanced (a "Blessing of Dimensionality").
3. The Proof: The author built a highly detailed mathematical map (Asymptotic Expansion) to show exactly how the errors cancel out.
4. The Backup: If the data is messy, a "Double Bootstrap" (checking the check) guarantees accuracy.
5. The Tool: He used "Stein Kernels" (a way of measuring internal tension in data) to make the math work where traditional methods failed.

In a nutshell: This paper explains why a specific statistical "super-trick" works so well when dealing with massive amounts of data, proving that sometimes, having too much information is actually a superpower, not a problem.

Technical Summary

Here is a detailed technical summary of the paper "High-dimensional bootstrap and asymptotic expansion" by Yuta Koike.

1. Problem Statement

The paper addresses the statistical challenge of approximating the distribution of the maximum of a sum of independent random vectors in high-dimensional settings where the dimension $d$ is potentially much larger than the sample size $n$ ($d \gg n$).

• The Statistic: Let $X_1, \dots, X_n$ be independent centered random vectors in $\mathbb{R}^d$. The statistic of interest is $T_n = \max_{1 \le j \le d} S_{n,j}$, where $S_n = n^{-1/2} \sum_{i=1}^n X_i$.
• The Context: Previous seminal work by Chernozhukov, Chetverikov, and Kato (CCK) established that the Gaussian approximation for $T_n$ is valid under mild conditions, even when $d \gg n$. This allows for the construction of simultaneous confidence intervals and hypothesis tests.
• The Gap: While the Gaussian approximation (and its feasible version, the Gaussian wild bootstrap) provides a first-order approximation, numerical experiments suggest that third-moment matching bootstrap methods (e.g., using weights with skewness matching the data) significantly outperform the Gaussian bootstrap in terms of coverage probability accuracy. However, existing theoretical results could not explain why this improvement occurs, nor could they explain why the Gaussian bootstrap fails to improve in the same way. Furthermore, standard Edgeworth expansions (the tool used to analyze second-order accuracy) were not available for $T_n$ in high dimensions due to the lack of a non-degenerate limit distribution and the degeneracy of the sample covariance matrix when $d \ge n$.

2. Methodology

To explain the superior performance of specific bootstrap methods and establish second-order accuracy, the author develops a rigorous asymptotic expansion framework for the coverage probability $P(T_n \ge \hat{c}_{1-\alpha})$.

• Stein's Method: Instead of relying on Fourier analysis (which requires Cramér's condition and struggles with high-dimensional geometry), the paper utilizes Stein's method. This approach assumes the underlying random vectors admit Stein kernels. This is a crucial technical choice because Stein kernels exist for a broad class of distributions (including log-concave and Gaussian copula models) even when the covariance matrix is singular or the distribution is not absolutely continuous in the traditional sense required for Fourier methods.
• Edgeworth Expansion for High Dimensions: The author derives valid Edgeworth expansions for both the original statistic $S_n$ and the bootstrap statistic $S_n^*$ (Gaussian wild bootstrap). A key innovation is defining the expansion "around" the population Gaussian density $\phi_\Sigma$ rather than the sample covariance $\hat{\Sigma}_n$, which is degenerate when $d \ge n$.
• New Inequalities: Two novel inequalities are developed to overcome specific high-dimensional hurdles:
  1. Anti-concentration inequality for higher-order terms: A new bound for the integral of the gradient of the Gaussian density over the boundary of rectangles. Unlike previous bounds that grow polynomially with $d$, this new bound depends only poly-logarithmically on $d$, making it suitable for ultra-high dimensions.
  2. Isoperimetric inequality for Gaussian maxima: A bound on the inverse of the distribution function of $Z^\vee = \max Z_j$. This is necessary to justify the Cornish–Fisher expansion (which relates quantiles of the statistic to quantiles of the Gaussian limit) in a setting where the limit distribution depends on $n$ and $d$.
• Double Wild Bootstrap: To handle cases where the covariance structure prevents second-order accuracy in standard bootstraps, the paper proposes a double wild bootstrap (nested resampling) method, adapted from Beran's prepivoting technique.

3. Key Contributions and Results

A. Asymptotic Expansion Formula (Theorem 2.3)

The paper derives an explicit asymptotic expansion for the coverage error:
$$P(T_n \ge \hat{c}_{1-\alpha}) = \alpha - (1-\gamma)Q_n(c^G_{1-\alpha}) - E[R_n(\alpha)] + \text{Remainder}$$
where $\gamma = E[w_1^3]$ is the skewness of the bootstrap weights, $Q_n$ represents the third-moment correction, and $R_n$ involves the covariance structure. This formula quantifies exactly how the choice of bootstrap weights affects the error.

B. The "Blessing of Dimensionality" (Corollary 2.2)

The most significant theoretical finding is that third-moment matching wild bootstrap is second-order accurate (error rate $O(n^{-1})$ up to log factors) in high dimensions without studentization, provided:

1. The weights match the third moment ($\gamma = 1$).
2. The covariance matrix $\Sigma$ has identical diagonal entries and bounded eigenvalues.
   In this specific regime, the error term involving the third moment vanishes or becomes negligible due to the high dimensionality ($d \ge n$). This explains the numerical phenomenon where third-moment matching bootstraps outperform Gaussian bootstraps.

C. Limitations and Covariance Structure (Corollary 2.4)

The paper shows that this "blessing" is not universal. If the covariance matrix has a common factor structure (e.g., an equicorrelation matrix where $\Sigma_{jk} = \rho$ for $j \neq k$ with $\rho$ close to 1), the third-moment matching bootstrap may actually perform worse than the Gaussian bootstrap. The improvement relies heavily on the specific geometry of $\Sigma$.

D. Double Wild Bootstrap (Theorem 2.4)

To achieve second-order accuracy regardless of the covariance structure, the paper introduces a double wild bootstrap method. By resampling the bootstrap weights themselves (a nested procedure), the method effectively estimates the distribution of the studentized statistic without needing to invert the singular sample covariance matrix. Theorem 2.4 proves this method achieves second-order accuracy ($O(n^{-1})$) universally.

E. Validity under Stein Kernels

The results are established under the assumption that the data admits Stein kernels. The paper provides examples showing this condition holds for:

• Log-concave distributions.
• Gaussian copula models (with specific regularity conditions).
• Affine transformations and multiplicative perturbations of such variables.

4. Simulation Study

The author conducts a Monte Carlo study ($n=200, 400; d=400$) using Gamma marginals and Gaussian copulas.

• Findings:
  • In designs with independent components (or weak correlation), the Beta wild bootstrap (third-moment matching) significantly outperforms the Gaussian wild bootstrap, confirming the "blessing of dimensionality."
  • In designs with strong equicorrelation ($\rho=0.8$), the Gaussian bootstrap sometimes outperforms the third-moment matching bootstrap, aligning with Corollary 2.4.
  • The Double Wild Bootstrap uniformly outperforms other methods in high dimensions, particularly as $n$ increases, validating the theoretical claim of universal second-order accuracy.

5. Significance

• Theoretical Explanation: This paper provides the first rigorous theoretical explanation for the empirical success of third-moment matching bootstraps in high-dimensional inference, a phenomenon previously unexplained by existing literature.
• Methodological Advancement: It extends the applicability of Edgeworth expansions and Cornish–Fisher expansions to the high-dimensional regime ($d \gg n$), a setting where classical tools fail.
• Practical Implication: It offers a concrete solution (Double Wild Bootstrap) for constructing highly accurate simultaneous confidence intervals and hypothesis tests in high-dimensional data analysis, even when the covariance structure is complex or unknown.
• Tool Development: The new anti-concentration and isoperimetric inequalities developed for this paper are likely to be valuable tools for future research in high-dimensional probability and statistics.

In summary, Yuta Koike's work bridges the gap between numerical observations and theoretical justification in high-dimensional bootstrap inference, revealing that dimensionality can be a "blessing" for specific bootstrap methods while providing a robust fallback (double bootstrap) for general cases.