📊 statistics

Zero Variance Portfolio

This paper demonstrates that a novel "Ridgelet" estimator enables zero-variance portfolios to achieve optimal out-of-sample risk and exhibit the double descent phenomenon in high-dimensional settings where the number of assets exceeds the sample size, outperforming traditional "Ridgeless" pseudoinverse methods that fail to generalize.

Original authors: Jinyuan Chang, Yi Ding, Zhentao Shi, Bo Zhang

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinyuan Chang, Yi Ding, Zhentao Shi, Bo Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect, low-calorie soup. You have a recipe book (the Market) with thousands of ingredients ( Assets like stocks), but you only have a tiny amount of time to taste-test them before you have to serve the soup to your customers ( Out-of-Sample Data).

This is the core problem of Portfolio Optimization: How do you mix thousands of stocks to get the lowest possible risk, when you don't have enough historical data to know exactly how they behave?

The Old Problem: The "Zero Variance" Trap

Traditionally, if you try to mix 500 ingredients using only 22 days of taste tests, your math breaks. The computer gets confused and says, "I found the perfect mix! It has zero risk!"

But this is a lie. It's a hallucination. The computer just found a way to perfectly cancel out the noise in your tiny sample, like a magician making a coin disappear. If you use this "perfect" mix tomorrow, it will likely fail spectacularly because it memorized the training data instead of learning the rules of the game. This is called overfitting.

The "Ridgeless" Mistake

Some modern AI researchers tried to fix this by using a "magic eraser" (called a Pseudoinverse) to solve the math when data is scarce. They thought, "If we just ignore the impossible parts, we can still find a solution."

The paper shows this is a disaster. In the soup analogy, it's like the chef saying, "I'll just pretend the salt doesn't exist." The result? The soup tastes fine in the test kitchen (in-sample), but the moment you serve it to a customer, it's inedible. The risk explodes.

The New Solution: The "Ridgelet" Estimator

The authors propose a surprisingly simple fix they call Ridgelet.

Imagine you are trying to balance a stack of 500 plates on a tiny table (your limited data).

The Old Way: You try to balance them perfectly. They fall over immediately.
The Ridgelet Way: You add a tiny, almost invisible drop of glue (a tiny number, like 0.00000001) to the bottom of every plate.

This "glue" is mathematically called a Ridge penalty. It's so small it doesn't change the flavor of the soup, but it stops the plates from wobbling and falling over.

Why does this work?

It prevents the "Zero Risk" lie: The glue ensures the math never claims the risk is zero. It forces the model to admit, "I don't know everything, so I'll add a tiny bit of safety."
The Double Descent: The paper shows a fascinating phenomenon called Double Descent.
- Phase 1: As you add more ingredients (stocks), the soup gets better.
- Phase 2: As you add too many ingredients for your small sample size, the soup gets terrible (the risk spikes).
- Phase 3: But if you keep adding even more ingredients (going way past the limit), the soup suddenly gets better again.
- The Ridgelet method is the key that unlocks this third phase. It allows you to use massive amounts of data (thousands of stocks) with very little history, and it actually works better than traditional methods.

The "Refined" Version: Ridgelet2

The authors realized that not all ingredients are the same. Some move together because of the weather (Market Factors), and some move together because they are in the same neighborhood (Industry/Idiosyncratic factors).

Ridgelet1 puts the same tiny drop of glue on every plate. It works well, but it's a bit blunt.
Ridgelet2 is the "Smart Chef." It looks at the specific neighborhood of each ingredient and applies the glue exactly where it's needed. It uses a special technique (called POET) to figure out which ingredients are neighbors and which are strangers.

The Real-World Result

The authors tested this on the S&P 500 (500 US stocks) and even added Nikkei 225 (Japanese stocks) to make a giant global soup.

Traditional methods struggled when they tried to mix 700+ stocks with only one month of data.
The Ridgeless method failed completely.
Ridgelet2 succeeded. It managed to construct a portfolio with 700+ stocks using only 22 days of data, and it performed better than the experts' best methods.

The Takeaway

In a world where we have too much data (thousands of stocks) but too little time (short history), the old rule of "simplify the model" is wrong.

Sometimes, the best way to handle a complex, high-dimensional problem is to embrace the complexity, add a tiny bit of "glue" (regularization) to keep things stable, and let the sheer volume of data do the heavy lifting. The paper proves that more is better, provided you have the right mathematical "glue" to hold it all together.

1. Problem Statement

The paper addresses the Minimum Variance Portfolio (MVP) problem in a high-dimensional setting where the number of assets ( $N$ ) significantly exceeds the sample size ( $T$ ), i.e., $N \gg T$ . This is a common scenario in modern finance (e.g., constructing portfolios with hundreds of stocks using only a few months of daily data).

The Challenge: In the classical $N < T$ regime, the sample covariance matrix ( $S_0$ ) is invertible, and the standard MVP estimator converges to the oracle (population) solution. However, when $N > T$ , $S_0$ is rank-deficient.
The Pathological Case: When $N > T$ , there exist infinitely many portfolio weights that achieve Zero In-Sample Variance (Zero Variance Portfolio or ZVP). The standard approach to handle this singularity is the Ridgeless estimator (using the Moore-Penrose pseudoinverse, $S_0^+$ ), which selects the minimum $\ell_2$ -norm solution among those interpolating the data.
The Failure: The authors argue that the Ridgeless estimator fails to generalize to out-of-sample (OOS) data in the $N \gg T$ regime. While it perfectly fits the training data, its OOS risk diverges to infinity, performing even worse than a naive equal-weight portfolio.

2. Methodology

The authors propose a novel estimator called Ridgelet, which leverages insights from modern machine learning theory (specifically "benign overfitting" and "double descent") but adapts them to portfolio optimization.

A. The Ridgelet Estimator (Ridgelet1)

Instead of using the pseudoinverse ( $S_0^+$ ), the authors propose adding a tiny, fixed ridge penalty $\tau$ to the sample covariance matrix:
$S_\tau = S_0 + \tau I_N$
The portfolio weights are then calculated as:
$\hat{\omega}_\tau = \frac{S_\tau^{-1} \mathbf{1}}{\mathbf{1}^\top S_\tau^{-1} \mathbf{1}}$

Key Distinction: Unlike traditional Ridge regression where $\tau$ is a tuning parameter optimized via cross-validation to balance bias and variance, the Ridgelet uses a fixed, infinitesimal constant (e.g., $\tau = 10^{-8}$ ).
Theoretical Justification: The authors prove that as $\tau \to 0$ , this estimator converges to the minimum $\ell_2$ -norm solution of the ZVP problem (solving $R^\top \omega = 0$ and $\mathbf{1}^\top \omega = 1$ ). Crucially, they show that the Ridgelet solution is discontinuous at $\tau=0$ relative to the Ridgeless solution ( $S_0^+$ ), meaning the Ridgeless estimator is not the limit of the Ridgelet estimator in this specific constrained optimization context.

B. The Refined Ridgelet (Ridgelet2)

Recognizing that asset returns often follow a factor model ($R = BF + E$) where $E$ represents idiosyncratic returns with a sparse covariance matrix $\Omega$ , the authors propose a refined version.

Mechanism: Instead of adding $\tau I_N$ , they add $\tau \hat{\Omega}$ , where $\hat{\Omega}$ is a consistent estimator of the idiosyncratic covariance matrix (using the POET estimator by Fan et al., 2013, which applies thresholding to PCA residuals).
Goal: This aligns the ridge penalty with the true structure of the noise, allowing the estimator to achieve oracle optimality in the high-dimensional limit.

3. Key Contributions

Resolution of the ZVP Generalizability Paradox: The paper demonstrates that while ZVPs are traditionally viewed as pathological overfits, a specific ZVP (the minimum $\ell_2$ -norm one) can possess excellent out-of-sample generalizability if estimated correctly.
Ridgelet vs. Ridgeless: The authors rigorously distinguish between the Ridgelet and Ridgeless estimators. They prove that:
- Ridgeless ( $S_0^+$ ): Fails in-sample interpolation (does not yield zero variance) and suffers from divergent OOS risk ( $RV \to \infty$ ) when $N \gg T$ .
- Ridgelet ( $S_\tau^{-1}$ ): Achieves zero in-sample variance (up to numerical precision) and exhibits double descent behavior, where OOS risk decreases significantly once $N$ surpasses $T$ .
Double Descent Phenomenon in Finance: The paper provides theoretical and empirical evidence of the double descent curve in portfolio risk. As $N$ $N$ increases:
- Risk decreases (under-parameterized).
- Risk peaks near $N \approx T$ (interpolation threshold).
- Risk drops again and converges to the oracle risk as $N \to \infty$ (over-parameterized regime).
Optimality under Factor Models: The paper proves that Ridgelet2 achieves the asymptotic oracle OOS variance when $N \gg T$ , provided the idiosyncratic covariance matrix is estimated consistently. This establishes a "blessing of dimensionality" for portfolio optimization.

4. Results

Theoretical Results

Proposition 1: Ridgelet1 is the limiting solution to the minimum $\ell_2$ -norm ZVP, whereas Ridgeless is orthogonal to this solution.
Proposition 2: In the $N \gg T$ regime, the relative OOS variance of Ridgelet remains bounded (comparable to the oracle), while the relative variance of Ridgeless diverges to infinity.
Theorem 1 & 2: Under approximate factor models, Ridgelet2 achieves the oracle OOS variance asymptotically. The error depends on the consistency of the idiosyncratic covariance estimator $\hat{\Omega}$ .

Empirical and Simulation Results

Simulations: Extensive Monte Carlo simulations confirm the double descent pattern. Ridgelet1 outperforms Linear Shrinkage (LS) and Factor-based Nonlinear Shrinkage (FNLS) in high-dimensional settings ( $N \ge 400$ ). Ridgelet2 further improves performance when idiosyncratic correlations are non-zero.
Robustness: The method remains robust under heavy-tailed distributions (t-distribution with 5 degrees of freedom).
Real-World Data:
- S&P 500: Using daily returns (2000–2023), Ridgelet2 consistently outperforms LS, FNLS, and Equal Weight (EW) portfolios, achieving lower annualized risk.
- Multinational Portfolio: Combining S&P 500 and Nikkei 225 stocks ( $N \approx 725$ ), Ridgelet2 demonstrated superior risk reduction compared to all benchmarks, validating the "blessing of dimensionality" in cross-asset diversification.

5. Significance

This paper challenges the conventional wisdom in portfolio management that high-dimensional covariance estimation requires complex shrinkage or dimension reduction to avoid overfitting. Instead, it posits that:

Overfitting can be beneficial: In the $N \gg T$ regime, interpolating the training data (ZVP) is not only possible but optimal for OOS performance, provided the correct estimator (Ridgelet) is used.
Simplicity wins: A simple, tuning-free addition of a tiny ridge ( $10^{-8}$ ) to the sample covariance matrix is sufficient to unlock the benefits of high dimensionality, outperforming sophisticated shrinkage estimators that require parameter tuning.
Machine Learning Integration: It successfully bridges the gap between modern ML theory (double descent, benign overfitting) and classical financial theory (Markowitz MVP), offering a practical, theoretically grounded tool for institutional investors managing large asset universes with limited historical data.