📊 statistics

Bidirectional Random Projections

This paper establishes an expected excess loss bound for ordinary least squares regression using bidirectional random projections on both the design matrix and response vector under a fixed design setting, demonstrating that this approach offers a performance gap of approximately $O(p_1 + C/p_1)$ compared to unidirectional projections, with the constant $C$ potentially yielding improvements for small projection dimensions.

Original authors: Chao Lan, Luyuan Yang

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chao Lan, Luyuan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, complex puzzle (predicting an outcome based on data). You have two ways to make this puzzle easier to solve:

Simplify the pieces: You have thousands of puzzle pieces (features/variables), but many are redundant. You can throw away the extras and keep only the most important ones. In the paper, this is called projecting the data sideways (reducing the number of columns).
Simplify the picture: You have thousands of photos of the puzzle (data points/samples), but looking at all of them takes forever. You can pick a smaller, representative handful of photos to work with. In the paper, this is called projecting the data up and down (reducing the number of rows).

Usually, statisticians do one or the other. This paper asks: What happens if we do both at the same time? They call this "Bidirectional Random Projection."

Here is the breakdown of their findings using simple analogies:

The Setup: Two Ways to Shrink the Problem

The authors are looking at a standard method called Ordinary Least Squares (OLS), which is like trying to draw the best possible straight line through a cloud of dots to predict a trend.

Method A (The Standard Shortcut): You keep all your photos (rows) but simplify the puzzle pieces (columns). You throw away some details to make the math faster.
Method B (The Bidirectional Shortcut): You simplify the puzzle pieces AND you throw away some photos. You are working with a tiny, simplified version of the puzzle.

The Big Discovery: Doing Too Much Hurts

The paper's main conclusion is that Method B (doing both) is generally worse than Method A (doing just one).

Think of it like trying to listen to a song.

Method A is like turning down the volume slightly so you can hear the melody better without losing the lyrics.
Method B is like turning down the volume and putting on noise-canceling headphones that block out half the instruments. You get a faster, smaller file, but the quality of the song (the accuracy of the prediction) suffers more than if you had just turned down the volume.

The math shows that the "error" (how far off your prediction is) is higher when you shrink the data in both directions compared to just shrinking the features.

The "Goldilocks" Zone: Finding the Sweet Spot

The paper also explores a very specific, interesting behavior regarding how much you should shrink the data.

Imagine you are adjusting the zoom on a camera.

If you zoom in too little, the picture is too big and slow.
If you zoom in too much, the picture becomes blurry and you lose important details.
There is a "sweet spot" in the middle where the picture is clear and fast.

The authors found that when you shrink the data in both directions, this "sweet spot" happens sooner (with less data) than when you only shrink in one direction.

The Analogy: If you are trying to find the perfect temperature for a cup of coffee by adding ice, the "Bidirectional" method (adding ice and stirring) makes the coffee get too cold faster than just adding ice. You have to stop adding ice earlier, or the coffee becomes undrinkable.

The "Pivot" Point

The paper describes a "pivot" point. As you increase the size of your simplified dataset (adding more puzzle pieces back in):

First: The error goes down (the prediction gets better).
Then: The error goes up (the prediction gets worse again because you've added too much noise).

The surprising finding is that for the "Bidirectional" method, the error starts going back up much earlier than it does for the standard method. It's like a car that accelerates quickly but runs out of gas sooner.

Real-World Check

The authors tested this on a real dataset of handwritten digits (recognizing numbers 0-9).

They confirmed that the "Bidirectional" method (shrinking both rows and columns) was indeed less accurate than the standard method.
They confirmed that the "sweet spot" where the error starts to rise again happened earlier for the Bidirectional method, exactly as their math predicted.

Summary

If you are trying to speed up a statistical model:

Don't try to cut corners in two directions at once. It sounds efficient, but it actually introduces more error.
If you must do it: Be very careful not to shrink your data too much, because the "tipping point" where your results get bad happens much faster than you might expect.

Technical Summary: Bidirectional Random Projections for OLS Regression

Problem Statement

This paper addresses the statistical properties of Ordinary Least Squares (OLS) regression when applied to large-scale, high-dimensional data. While random projection is a well-established technique for reducing data dimensionality (feature space) to scale OLS, and separately for reducing the number of samples (data points) to accelerate estimation, the statistical behavior of applying random projections in both directions simultaneously remains unexplored.

The authors investigate the Bidirectional Random Projection (bOLS) estimator, which operates on a dataset $(WXR, WY)$, where $X \in \mathbb{R}^{n \times p}$ is the feature matrix, $Y \in \mathbb{R}^n$ is the label vector, $R \in \mathbb{R}^{p \times p_1}$ projects features to a lower dimension $p_1$ , and $W \in \mathbb{R}^{n_1 \times n}$ projects samples to a reduced count $n_1$ . The goal is to derive an expected excess loss bound for this bOLS estimator and compare it against the established bound for the standard projected OLS (pOLS) estimator, which only applies the feature projection $(XR, Y)$.

Methodology

The analysis is conducted under a fixed design setting. The data generation model is defined as $Y = X\beta^* + E$ , where $\beta^*$ is a fixed parameter vector and $E$ consists of independent random noises with zero mean and variance $\sigma^2$ .

Two random projection matrices are introduced:

Feature Projection ( $R$ ): Entries are i.i.d. Gaussian $\mathcal{N}(0, 1/p_1)$ .
Sample Projection ( $W$ ): Entries are i.i.d. Gaussian $\mathcal{N}(0, 1/n)$ .

The study defines two estimators:

pOLS ( $\hat{\theta}_R$ ): Minimizes $\frac{1}{n}\|XR\theta - Y\|^2$ .
bOLS ( $\hat{\theta}_{RW}$ ): Minimizes $\frac{1}{n_1}\|WXR\theta - WY\|^2$ .

The performance metric is the expected excess loss, defined as the difference between the expected squared loss of the estimator and the optimal loss $L(\beta^*)$ . The authors derive a theoretical upper bound for the bOLS excess loss and analyze the "gap" between this bound and the known bound for pOLS. The derivation relies on properties of the singular values of the random projection matrix $W$ , specifically utilizing asymptotic results regarding the concentration of the largest ( $\lambda_{w\uparrow}$ ) and smallest ( $\lambda_{w\downarrow}$ ) singular values of Gaussian random matrices.

Key Contributions

1. Theoretical Bound for Bidirectional Projection

The paper establishes a new expected excess loss bound for the bOLS estimator (Theorem 3.2). The bound is expressed as:
$B_{RW} = C_1 \frac{\sigma^2 p_1}{n} + C_2 \frac{1}{p_1} \|\beta^*\|_M^2 + b$
where $C_1$ and $C_2$ are constants dependent on the ratio $r = \sqrt{n_1/n}$ and the singular values of $W$ , and $b$ is a bias term. This bound generalizes the existing result for pOLS (Theorem 3.1), recovering it when $W$ is the identity matrix ( $n_1 = n$ ).

2. Analysis of the Performance Gap

A primary contribution is the analytical characterization of the gap between the bOLS bound ( $B_{RW}$ ) and the pOLS bound ( $B_R$ ). The authors identify three key implications regarding this gap as the projected dimension $p_1$ increases:

Performance Degradation: Generally, the gap is positive, indicating that the bOLS estimator performs worse than the pOLS estimator.
Non-Monotonic Gap Behavior: The gap does not simply increase or decrease monotonically. It initially decreases and then increases as $p_1$ grows. The "pivot" point (where the trend reverses) occurs earlier when the sample reduction ratio ( $n_1/n$ ) is smaller.
Early Loss Recovery: While both estimators exhibit a "U-shaped" loss curve (decreasing then increasing) as $p_1$ increases, the bOLS estimator's loss "bounces back" (begins to increase) earlier than that of the pOLS estimator.

3. Empirical Validation

The theoretical implications are validated using real-world data from the Digit dataset (1797 images, 64 features). The numerical results confirm that:

bOLS generally yields higher expected excess loss than pOLS.
The loss gap decreases as $p_1$ increases but eventually bounces back.
The bounce-back point occurs at a lower $p_1$ when the sample reduction is more aggressive (e.g., $n_1/n = 0.25$ vs. $0.5$), aligning with the theoretical predictions.

Significance and Claims

The paper claims to fill a gap in the literature by providing the first theoretical analysis of bidirectional random projections for OLS regression. The significance lies in quantifying the trade-offs of reducing both sample size and feature dimension simultaneously.

The authors modestly conclude that while bidirectional projection is desirable for computational efficiency in large-scale settings, it comes at a statistical cost. The derived bounds and the identified "pivot" behavior provide practitioners with theoretical guidance on how the interplay between reduced sample size ( $n_1$ ) and reduced dimension ( $p_1$ ) affects estimation accuracy. Specifically, the results suggest that aggressive sample reduction ( $small \ n_1/n$ ) can cause the estimation error to deteriorate at lower feature dimensions compared to standard dimensionality reduction alone.