📊 statistics

Risk reversal for least squares estimators under nested convex constraints

This paper demonstrates that in constrained stochastic optimization, the intuition that restricting the feasible set to a smaller nested convex set containing the true parameter always reduces statistical risk fails under sufficiently large noise, a phenomenon termed "risk reversal" where tighter constraints paradoxically degrade the performance of least squares estimators.

Original authors: Omar Al-Ghattas

Published 2026-07-28

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Omar Al-Ghattas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a lost hiker in a dense forest. You have a map, but it's a bit fuzzy, and you only have a noisy radio signal to guide you. In the world of statistics, this is a common problem: trying to guess a hidden truth (like the hiker's location) based on messy data. Usually, statisticians have a hunch about where the truth might be—maybe they know the hiker is somewhere inside a specific park, or perhaps even closer, inside a small clearing within that park. This "hunch" is called a constraint. The standard rule of thumb is that if you add more rules to your map (like narrowing the search from the whole forest to just the park, or the park to the clearing), you should get a better guess. It feels logical: more information should mean less error. This is the foundation of "constrained estimation," a field where mathematicians use geometry to figure out how to make the best guesses possible when the data is noisy.

But what if your extra rules actually made your guess worse? What if narrowing your search area, instead of helping you, accidentally pushed your estimate further away from the truth? This sounds like a paradox, a glitch in the matrix of logic. Yet, a new paper by Omar Al-Ghattas suggests that in the world of noisy data, this counter-intuitive "risk reversal" is not only possible but can happen in very simple, everyday scenarios. The paper explores a specific type of math problem called the "Gaussian sequence model," which is just a fancy way of saying: "We have a true number, and we see a version of it that has been jumbled up by random static." The goal is to find the true number by projecting the noisy observation onto a shape (like a triangle or a line) that we believe contains the answer. The paper asks a simple question: If we shrink our shape to a smaller one that still definitely contains the answer, does our error always go down?

The answer, surprisingly, is no. The paper proves that under certain conditions, specifically when the noise (the static) is loud enough, tightening the constraints can actually increase the error. The authors show this isn't just a fluke of a weird, broken example; it happens with standard shapes like triangles and lines, and it holds true even when the "true" answer is right in the middle of the shape, not just on the edge. They demonstrate that while adding constraints usually helps when the noise is tiny, there is a "sweet spot" of high noise where the geometry of the shapes interacts with the randomness in a way that makes the smaller, tighter constraint perform worse than the larger, looser one. This isn't just a theoretical curiosity; the authors prove it mathematically, showing that the error can strictly increase, and they even show that this can happen for the "worst-case" scenario, meaning it's a genuine flaw in how these estimators work, not just a one-off glitch.

The Story of the Shrinking Trap

Let's dive into the mechanics of this paradox using a story about a dartboard and a very confused thrower.

Imagine you are trying to hit a bullseye (the true parameter, $\theta^\star$ ) on a dartboard. But you are blindfolded, and every time you throw, the wind (the noise, $\sigma Z$ ) blows the dart off course. You know the bullseye is somewhere inside a large, triangular tent ( $\Theta_L$ ). To help yourself, you decide to put up a smaller, tighter tent ( $\Theta_S$ ) inside the big one, believing that if you restrict your search to this smaller area, you'll be more accurate. You use a "Least Squares Estimator," which is just a fancy name for a rule that says: "Look at where the dart landed, and draw a straight line to the closest point inside your tent."

In a calm, quiet world (low noise), this works perfectly. If the wind is barely blowing, your dart lands very close to the bullseye. Whether you are inside the big tent or the small one, the closest point to your dart is almost exactly the bullseye. In fact, if the bullseye is in the middle of the small tent, the small tent is slightly better because it cuts off some of the "wasted" space of the big tent that the dart could never reach anyway.

But now, crank up the wind. Make it a hurricane. Suddenly, your dart can land anywhere on the field, far away from the bullseye. This is where the magic (or the trap) happens.

The paper constructs a specific shape for the tents to show how this goes wrong. Imagine the big tent is a triangle, and the small tent is just a single line segment running through it. When the wind is howling, your dart might land in a spot that is far away from the bullseye.

The Big Tent: Because it's wide and has a corner pointing in a specific direction, the "closest point" inside the big tent might be a corner that happens to be relatively close to where your dart landed.
The Small Tent: Because it's just a thin line, the "closest point" on that line to your dart might be a spot that is actually farther away from the bullseye than the corner of the big tent was.

It's like trying to find a lost key in a room. If you have a huge room (the big tent), and you drop the key in a corner, you might find it quickly. But if you shrink the room to a narrow hallway (the small tent) that doesn't quite reach that corner, you might be forced to look in a spot that is actually further from where the key really is, simply because the hallway forces you to look in a different direction.

The paper shows that when the noise is high enough, the "small tent" forces the estimator to make a "bad guess" more often than the "big tent" does. The big tent, with its extra space, has a better chance of having a "safe harbor" (a point on its boundary) that is closer to the noisy dart than the forced point on the small tent.

The Two Faces of Noise

The authors explain that this phenomenon depends entirely on how loud the noise is. They split the story into two chapters:

Chapter 1: The Whisper (Vanishing Noise)
When the noise is tiny (the wind is a gentle breeze), the paper proves that the small tent always wins or ties. The error is dominated by the local shape of the tent right around the bullseye. If the bullseye is inside the small tent, the small tent's local shape is "tighter," which usually means less wiggle room for error. In this regime, the intuition holds: more constraints = better accuracy. The paper proves mathematically that you cannot have a risk reversal here; the error of the small tent is always less than or equal to the big tent.

Chapter 2: The Roar (Diverging Noise)
When the noise is huge (the hurricane), the local shape doesn't matter as much as the global shape. The dart is so far away that the estimator is essentially looking at the "silhouette" of the tent from the direction of the wind. The paper shows that the estimator ends up picking a specific "face" or "corner" of the tent based on the direction of the noise.
Here is the twist: The small tent might have a geometry that, when viewed from a random noisy direction, points to a corner that is far from the bullseye. The big tent, however, might have an extra corner that "catches" the noise better, landing closer to the bullseye.
The paper provides a concrete example with a parameter $c$ (which controls the shape of the triangle). They show that if $c$ is small (making the small tent very skinny), and the noise is large, the error of the small tent becomes strictly larger than the big tent. They even plot graphs showing that as you increase the noise, the error curves cross over. At first, the small tent is better, but then, as the noise gets louder, the lines cross, and the small tent becomes worse. This is the "Risk Reversal."

It's Not Just a Fluke

You might wonder, "Is this just a weird trick with a specific triangle?" The authors say no. They show that this happens even if the bullseye is right in the middle of the shape, not just on the edge. They show it happens with smooth shapes like ellipses, not just sharp triangles. They even show that it doesn't matter if the noise is perfectly Gaussian (the standard bell curve); it happens with other types of "radial" noise too, as long as the noise is loud enough.

Furthermore, they prove that this isn't just a one-time bad guess at a single point. They show that the worst-case error (the maximum possible error you could get) also flips. This means that if you are a statistician trying to design a system that works well even in the worst scenarios, adding a constraint could actually make your system less robust, not more.

The Takeaway

The paper reveals a hidden failure mode in one of the most basic tools of statistics: the projection estimator. We often think that "more structure" (more constraints) is always better. But this paper shows that in a noisy world, adding structure can sometimes trap your estimator in a geometry that is ill-suited for the chaos of the noise.

It's a reminder that in the face of high uncertainty, sometimes having a little more room to maneuver (a larger constraint set) is safer than being forced into a tight, specific corner. The "risk reversal" is a genuine statistical phenomenon where the very act of trying to be more precise can backfire, leading to a larger error than if you had just been a little more vague. The authors have proven this mathematically, showing that for sufficiently large noise, the intuition that "smaller is better" can be completely turned on its head.

Technical Summary: Risk Reversal for Least Squares Estimators under Nested Convex Constraints

Problem Statement
In constrained stochastic optimization, it is a standard intuition that restricting the feasible set of an estimator, provided the true parameter remains within the set, should improve or at least not degrade statistical performance. This paper investigates the validity of this intuition within the Gaussian sequence model. Specifically, given a true parameter $\theta^\star$ and two nested, compact, convex constraint sets $\Theta_S \subsetneq \Theta_L$ (where $\theta^\star \in \Theta_S$ ), the authors examine the behavior of the Least Squares Estimator (LSE), which coincides with the Maximum Likelihood Estimator (MLE) in this setting. The LSE is defined as the Euclidean projection of the observation $Y = \theta^\star + \sigma Z$ onto the constraint set. The central question is whether the squared-error risk of the estimator constrained to the smaller set $\Theta_S$ can strictly exceed that of the estimator constrained to the larger set $\Theta_L$ , a phenomenon the authors term "risk reversal."

Methodology
The paper employs a combination of explicit geometric construction, asymptotic analysis, and probabilistic limit theorems to characterize the risk of the LSE across different noise regimes.

Explicit Construction: The authors construct a specific two-dimensional example involving nested convex sets (a line segment $\Theta_S$ and a triangle $\Theta_L$ ) to demonstrate risk reversal. They derive exact closed-form expressions for the risk functions $R_\sigma(\theta^\star; \Theta)$ to track their dependence on the noise level $\sigma$ and geometric parameters.
Asymptotic Regimes: The analysis is divided into two complementary regimes:
- Vanishing-noise regime ( $\sigma \to 0$ ): The risk is analyzed using the statistical dimension of the tangent cone at the true parameter, following established results by Oymak and Hassibi (2016) and Amelunxen et al. (2014).
- Diverging-noise regime ( $\sigma \to \infty$ ): The authors develop a new geometric characterization where the estimator's behavior is governed by the global geometry of the constraint set rather than local properties. They show that as noise diverges, the LSE concentrates on the "exposed face" of the constraint set selected by the random direction of the noise vector.
Generalization: The framework is extended beyond the Gaussian squared-error setting to a class of radial location models (where noise density depends only on the norm) and radial loss functions, demonstrating that the phenomenon relies on the geometry of the projection and the direction of noise rather than specific distributional assumptions.

Key Results

Failure of Intuition in Diverging Noise: The paper proves that risk reversal is possible. In the diverging-noise regime, there exist nested sets $\Theta_S \subsetneq \Theta_L$ and a parameter $\theta^\star \in \Theta_S$ such that $R_\sigma(\theta^\star; \Theta_S) > R_\sigma(\theta^\star; \Theta_L)$ for sufficiently large $\sigma$ . This occurs because the "misalignment" of the constraint sets causes the projection onto the smaller set to land on a face (or vertex) that is, on average, farther from the true parameter than the projection onto the larger set when noise directions are random.
Vanishing Noise Monotonicity: Conversely, in the vanishing-noise regime, risk reversal cannot occur at the leading order ( $\sigma^2$ ). The leading term of the risk is proportional to the statistical dimension of the tangent cone. Since the statistical dimension is monotone under inclusion of convex cones, the risk of the smaller constraint is asymptotically no larger than that of the larger set as $\sigma \to 0$ .
Worst-Case Risk Reversal: The phenomenon is not merely a pointwise anomaly. The authors construct examples where the worst-case risk (the supremum of the risk over the constraint set) also reverses: $\sup_{\theta \in \Theta_S} R_\sigma(\theta; \Theta_S) > \sup_{\theta \in \Theta_L} R_\sigma(\theta; \Theta_L)$ for large $\sigma$ . This is established via uniform convergence of the risk function to its limiting form in the diverging-noise regime.
Robustness: The results extend to radial noise models and radial loss functions. The mechanism depends on the uniform distribution of noise directions on the sphere and the geometry of the exposed faces, which remains invariant under these generalizations.

Significance and Claims
The paper claims to reveal a previously unrecognized failure mode of the Least Squares Estimator and projection-based estimators more broadly. The authors emphasize that this is a genuine statistical phenomenon arising from the global geometry of the constraint set, not an artifact of model misspecification, non-convexity, or boundary effects.

Paradoxical Performance: The work demonstrates that in sufficiently noisy settings, tightening a correctly specified constraint can paradoxically degrade statistical performance.
Limitations of Existing Theory: The authors note that existing risk characterizations, such as those by Chatterjee (2014), are too coarse to detect this phenomenon. Chatterjee's bounds rely on local approximations or scale parameters that do not capture the non-local, global geometric interactions responsible for risk reversal in the diverging-noise regime.
Distinction from Misspecification: Unlike related work (e.g., Fang and Guntuboyina, 2019) where risk reduction in misspecified settings is driven by bias, this phenomenon occurs in well-specified models where the true parameter is strictly contained within both constraint sets.

In summary, the paper establishes that while adding valid structural constraints generally reduces complexity, it can introduce global geometric misalignments that, under high noise, increase estimation error. This challenges the assumption that "more constraints" always equate to "better estimation" in projection-based methods.

The Story of the Shrinking Trap

The Two Faces of Noise

It's Not Just a Fluke

The Takeaway

More like this