📊 statistics

Adaptive L-statistics for high dimensional test problem

This paper proposes an adaptive L-statistic framework for high-dimensional one-sample location testing that combines fixed and diverging parameter statistics via a Cauchy combination test to achieve robust performance across varying sparsity levels, supported by theoretical asymptotic results and empirical validation.

Original authors: Huifang Ma, Long Feng, Zhaojun Wang

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huifang Ma, Long Feng, Zhaojun Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for a single clue, you are staring at a massive wall of thousands of tiny, flickering lights. Some of these lights might be glowing because a secret signal is there, while most are just flickering randomly due to background noise. In the world of statistics, this is called a "high-dimensional" problem. The challenge is that when you have more lights (variables) than you have snapshots of the scene (samples), the usual tools for finding the signal break down. It's like trying to find a needle in a haystack, but the haystack is so huge that your compass spins wildly.

To solve this, statisticians have developed two main strategies. The first is the "Sum-It-Up" approach: you add up the brightness of every single light. This works great if the signal is spread out, like a soft, glowing fog covering the whole wall. The second is the "Spotlight" approach: you ignore the dim lights and only look at the single brightest one. This is perfect if the signal is a tiny, blinding laser pointer hidden among the darkness. But here's the tricky part: in real life, we rarely know if the signal is a fog or a laser. If we guess wrong and use the wrong tool, we might miss the signal entirely. This paper tackles the problem of how to build a detective's toolkit that works whether the signal is a fog, a laser, or something in between.

The researchers, Huifang Ma, Long Feng, and Zhaojun Wang from Nankai University, propose a clever new method called "Adaptive L-statistics." Think of their solution as a smart, shape-shifting flashlight. Instead of forcing the data to fit a "fog" or "laser" model, they create a family of tests that can look at different numbers of the brightest lights at once. They discovered that if you know exactly how many lights are actually glowing (the "sparsity level"), you can tune your flashlight to look at just that many, and it becomes incredibly powerful. However, since we usually don't know the exact number, they combined all these different flashlights into one super-tool using a mathematical trick called a "Cauchy combination."

Here is how their new method works and what they found. First, they proved mathematically that their different flashlights (tests with different settings) don't interfere with each other; they are "asymptotically independent." This means you can safely mix their results together without the math getting messy. They then showed that by blending a test that looks at a few bright lights (good for sparse signals) with a test that looks at many lights (good for dense signals), they created a single test that is robust against almost any scenario.

In their simulations, which are like running thousands of practice detective cases on a computer, they found that their new method consistently outperformed existing tools. When the signal was very sparse (only a few lights), their method was just as good as the best "laser" detectors. When the signal was dense (many lights), it was just as good as the best "fog" detectors. Most importantly, when the signal was somewhere in the middle—a situation where other methods often struggle—their adaptive method remained strong and reliable. They even tested it on real-world data: weekly stock returns from the S&P 500. In this real-life scenario, their method successfully identified that some stocks were earning returns that couldn't be explained by chance alone, outperforming other popular tests in the process.

The authors are careful to note that their results are based on rigorous mathematical proofs for the theory and extensive computer simulations for the performance. They didn't just guess; they showed that under a wide range of conditions, their method holds up. They also demonstrated that their approach works even when the lights (variables) are somewhat connected to each other, which is a common complication in real data. By combining the strengths of looking at the "top few" and the "top many," they have built a statistical tool that doesn't need to know the secret of the signal in advance to do its job well. It's a bit like having a detective who can instantly switch from a magnifying glass to a wide-angle lens, ensuring that no matter how the mystery is hidden, the truth is likely to be found.

Technical Summary: Adaptive L-statistics for High Dimensional Test Problems

Problem Statement
This study addresses the high-dimensional one-sample location test problem, specifically testing the null hypothesis $H_0: \mu = 0$ against $H_1: \mu \neq 0$ where the dimension $p$ is comparable to or exceeds the sample size $n$ . Traditional methods like Hotelling's $T^2$ fail in this regime due to the non-invertibility of the sample covariance matrix. Existing high-dimensional approaches generally fall into two categories: sum-type tests (effective for dense alternatives where many variables have non-zero means) and max-type tests (effective for sparse alternatives where only a few variables are non-zero). While adaptive tests have been proposed to combine these approaches, they often rely on a limited set of power parameters (e.g., $r = 2, 4, 6, \infty$ ) or suffer from high computational costs for large $r$ . Furthermore, many existing adaptive procedures lack rigorous theoretical justification regarding the limiting distributions of their component statistics or the independence required for combination.

Methodology
The authors propose an adaptive testing procedure based on L-statistics, which are linear combinations of functions of order statistics. The core of the method involves constructing test statistics $T_k$ that sum the squares of the $k$ largest order statistics of the individual $t$ -test statistics:
$T_k = \sum_{i=1}^k t^2_{(p+1-i)}$
where $t^2_{(1)} \leq \dots \leq t^2_{(p)}$ are the ordered squared $t$ -statistics.

The methodology proceeds in three theoretical stages:

Fixed Parameter Regime: For a fixed integer $k$ , the authors derive the limiting null distribution of $T_k$ . They establish that $T_k$ converges to a Gumbel-type distribution (specifically related to the maximum of $k$ variables). This statistic is designed to detect sparse alternatives.
Diverging Parameter Regime: For $k$ diverging with $p$ (specifically $k = \lceil \gamma p \rceil$ for fixed $\gamma \in (0,1)$ ), the authors establish the asymptotic multivariate normality of $T_k$ . This statistic is designed to detect dense alternatives.
Adaptive Combination: Recognizing that the true sparsity level is unknown, the authors propose a Cauchy combination test ( $T_C$ ) that integrates $T_k$ statistics with fixed $k$ (specifically $k=5$ ) and $T_k$ statistics with diverging $k$ (specifically $k = \lceil 2^{-i}p \rceil$ ).

A critical theoretical component is the proof of asymptotic independence between the $T_k$ statistic with fixed $k$ and the $T_k$ statistic with diverging $k$ under the null hypothesis. This independence allows for the valid application of the Cauchy combination method, which aggregates the $p$ -values of the individual tests. The authors utilize the wild bootstrap method (using Rademacher distributions) to calculate $p$ -values, as the convergence of max-type statistics to their asymptotic distributions can be slow in finite samples.

Key Contributions
The paper makes three primary theoretical and methodological contributions:

Limiting Distribution for Fixed $k$ : The authors determine the limiting null distribution of the L-statistic with a fixed parameter $k$ . They show this statistic aligns with max- $k$ type tests but provides a rigorous theoretical foundation for its use, demonstrating superior performance over the classic max-type test ( $k=1$ ) under sparse alternatives.
Asymptotic Normality for Diverging $k$ : The paper establishes the asymptotic multivariate normality of the L-statistic when the parameter $k$ diverges with the dimension. Power analysis indicates that the optimal choice of $k$ corresponds to the true number of non-zero mean components, maximizing detection power for dense alternatives.
Asymptotic Independence and Adaptive Procedure: The authors prove the asymptotic independence between L-statistics with fixed parameters and those with diverging parameters. This theoretical result justifies the use of the Cauchy combination test to merge these distinct statistics into a single adaptive procedure ( $T_C$ ) that is robust across varying sparsity levels.

Results

Simulation Studies: Extensive simulations under various distributions (Normal, $t$ -distribution, Mixture Normal) and covariance structures (including strong correlations and non- $\alpha$ -mixing structures) demonstrate that the proposed adaptive test $T_C$ effectively controls the empirical size. In terms of power, $T_C$ consistently outperforms existing adaptive methods, including the COM test (Fan et al., 2024), the adaQ test (Xu et al., 2017), the Power Enhancement (PE) test (Fan et al., 2015), the Random Projection (RP) test (Lopes et al., 2011), and the Higher Criticism (HC) test (Zhong et al., 2013). The proposed method maintains high power in both sparse and dense regimes, whereas competitors often excel in only one regime.
Real Data Application: The method was applied to weekly excess returns of S&P 500 constituents from 2009 to 2018. The test rejected the null hypothesis that the mean vector of excess returns is zero, providing evidence of abnormal returns. The adaptive procedure $T_C$ achieved the strongest or nearly strongest rejection rates compared to competing methods, and the analysis identified a small set of securities with significant excess returns.

Significance and Claims
The paper claims that the proposed adaptive L-statistic procedure offers a more flexible and broadly robust solution for high-dimensional mean testing compared to existing methods. By utilizing L-statistics with a continuous range of parameters (via the diverging $k$ approach) rather than a discrete, limited set of power indices, the method avoids the suboptimal power issues that arise when the true sparsity level deviates from predefined values. The rigorous establishment of asymptotic independence between fixed and diverging parameter statistics provides a theoretical basis for combining Gumbel-type and Gaussian-type statistics via the Cauchy method, a combination previously lacking theoretical justification in the literature. The authors conclude that their method outperforms existing adaptive procedures in both simulation studies and real-world data applications, offering a reliable tool for high-dimensional inference under diverse sparsity and dependence structures.

More like this