📈 economics

Compound Selection Decisions: An Almost SURE Approach

This paper introduces ASSURE, an almost unbiased estimator inspired by Stein's unbiased risk estimate, to optimize compound selection decisions in Gaussian sequence models by borrowing strength across noisy estimates to maximize expected utility, with applications demonstrated in economic opportunity selection, firm identification, and A/B testing.

Original authors: Jiafeng Chen, Lihua Lei, Timothy Sudijono, Liyang Sun, Tian Xie

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiafeng Chen, Lihua Lei, Timothy Sudijono, Liyang Sun, Tian Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a manager at a large company with hundreds of different projects running simultaneously. For each project, you have a noisy estimate of how well it will perform (maybe based on a small pilot test), but you don't know the true future success. Your goal is to pick the best projects to fund while avoiding the ones that will fail.

The problem is that your estimates are "noisy." Sometimes a bad project looks good just by luck, and sometimes a great project looks bad because of a fluke. If you just pick every project that looks good in the pilot, you'll waste money on false alarms. If you set the bar too high, you'll miss out on hidden gems.

This paper introduces a new tool called ASSURE (Almost SURE) to help you make these decisions smarter. Here is how it works, using simple analogies:

1. The Problem: The "Noisy Crystal Ball"

In the past, statisticians tried to solve this by guessing the "true shape" of all the projects combined. They might say, "I bet 80% of projects are average, 10% are amazing, and 10% are terrible." They then used this guess to filter the data.

The Flaw: If your guess about the "shape" is wrong (which it often is in the real world), your filtering system breaks. It's like trying to tune a radio by guessing the frequency; if you guess wrong, you hear static.

2. The Solution: The "Honest Scorecard" (ASSURE)

Instead of guessing the shape of the future, the authors propose a method that acts like an honest scorecard.

They created a formula (based on a famous statistical trick called "Stein's Identity") that looks at your noisy data and calculates: "If we use this specific rule to pick projects, how much money will we actually make on average?"

The Magic: This scorecard is "robust." It doesn't care if your guess about the projects was right or wrong. It calculates the expected payoff directly from the data you have.
The "Almost" in ASSURE: The authors admit they can't make the scorecard perfectly unbiased (mathematically impossible for this specific problem), so they call it "Almost SURE." However, they show that the error is so tiny it vanishes as you get more data.

3. How It Works in Practice: The "Tuning Knob"

Imagine you have a radio with a tuning knob.

Old Way: You guess where the station is and set the knob there. If you guess wrong, you get static.
ASSURE Way: You have a device that tells you exactly how clear the signal is for any position of the knob. You simply turn the knob until the device says, "This is the clearest signal possible."

In the paper, the "knob" is the threshold you set for selecting projects (e.g., "Only fund projects with a score above 5"). ASSURE lets you test thousands of different thresholds and find the one that maximizes your actual profit, without needing to know the true nature of the projects beforehand.

4. Real-World Examples from the Paper

The authors tested this tool on three real-world scenarios:

Finding the Best Neighborhoods: Imagine a government wants to give housing vouchers to families moving to neighborhoods with high economic mobility. They have noisy estimates for hundreds of neighborhoods. Using ASSURE, they can find the best cutoff point to recommend neighborhoods, ensuring they don't pick areas that look good by chance but are actually struggling.
Spotting Discrimination: A study looked at 108 large companies to see which ones discriminated against job applicants. The data was noisy. The old method (guessing the distribution of discrimination) missed some companies or flagged the wrong ones. ASSURE adjusted the "bar" for what counts as discrimination, finding a better balance between catching real discrimination and avoiding false accusations.
Tech A/B Testing: Tech companies run thousands of experiments (like changing a button color on an app) to see what works. The standard rule is "only change the button if the test is statistically significant (p < 0.05)." The authors used ASSURE to show that for their specific data, this standard rule was too strict. By using ASSURE, they found a "looser" rule that would actually save the company more money by implementing more successful changes.

The Bottom Line

The paper argues that instead of trying to build a perfect model of the world (which is hard and often wrong), we should build a tool that directly measures the success of our decisions.

ASSURE is that tool. It allows researchers and managers to:

Evaluate: Check if their current way of making decisions is actually working.
Optimize: Automatically find the best decision rule for their specific situation.
Stay Safe: Even if their underlying assumptions are wrong, ASSURE ensures they don't make terrible decisions, often beating the traditional methods.

In short: Stop guessing the rules of the game. Use ASSURE to measure the score and play to win.

Technical Summary: Compound Selection Decisions: An Almost SURE Approach

1. Problem Formulation

The paper addresses compound selection problems within a Gaussian sequence model. In this setting, a researcher observes $n$ parallel units, each characterized by a noisy estimate $Y_i$ , a known standard error $\sigma_i$ , covariates $x_i$ , and a selection cost $k_i$ . The underlying parameters $\mu_i$ represent the true payoffs of selecting unit $i$ . The goal is to choose a set of decision thresholds $\delta_i$ (potentially dependent on $z_i = (\sigma_i, x_i, k_i)$ ) to select units where $Y_i > \delta_i$ , thereby maximizing the expected net utility (welfare):
$u(\delta_{1:n}, \mu_{1:n}) = \frac{1}{n} \sum_{i=1}^n \mathbb{1}(Y_i > \delta_i)(\mu_i - k_i)$
The statistical challenge is that the expected utility depends on the unknown $\mu_{1:n}$ and cannot be directly optimized. Standard approaches either treat decisions separately (ignoring collective structure) or rely on Empirical Bayes (EB) methods that assume a specific random-effects distribution for $\mu_{1:n}$ . However, these EB models are often misspecified, leading to suboptimal selection decisions because better prediction of $\mu_i$ does not necessarily imply better selection (Mehta, 2019; Manski, 2021).

2. Methodology: ASSURE and ASSURE*

Inspired by Stein's Unbiased Risk Estimate (SURE), the authors propose ASSURE (Almost SURE), a family of estimators for the expected welfare of a selection rule.

The Core Idea: Directly estimating the welfare of a step-function decision rule ( $\mathbb{1}(Y > \delta)$ ) is impossible with an unbiased estimator that grows slower than $e^{y^2}$ (Stefanski, 1989). To circumvent this, the authors smooth the indicator function using a kernel $k_h$ to create a differentiable approximation $F_h$ .
The Estimator: For a decision rule $\delta_i$ , the ASSURE estimator is defined as:
$\hat{W}_{ASSURE} = \frac{1}{n} \sum_{i=1}^n \left[ (Y_i - k_i) F_h\left(\frac{Y_i - \delta_i}{\sigma_i}\right) - \sigma_i k_h\left(\frac{Y_i - \delta_i}{\sigma_i}\right) \right]$
This form is derived via Stein's identity applied to the smoothed function.
ASSURE (Sinc Kernel):* The authors recommend the sinc kernel ( $k(x) = \frac{\sin x}{\pi x}$ ) for the smoothing function. This choice, termed ASSURE, yields an estimator with bias that vanishes exponentially fast in the bandwidth parameter $h$ . With $h = 1/\sqrt{2 \log n}$ , ASSURE achieves near-parametric convergence rates ( $O(\sqrt{\log n}/n)$ for mean-squared error).
ASSURE-CB (Gaussian Kernel): An alternative using the Gaussian kernel is proposed, which admits an interpretation as a Rao–Blackwellized version of coupled bootstrap. However, due to the Gaussian kernel being only second-order, ASSURE-CB converges more slowly ( $O(n^{-2/5})$ ) than ASSURE*.

3. Key Contributions and Theoretical Results

A. Robust Welfare Evaluation
The primary contribution is a method to evaluate selection decisions without requiring the random-effects model for $\mu_{1:n}$ to be correctly specified.

Validity: The estimator $\hat{W}$ converges to the true welfare $W$ for fixed $\mu_{1:n}$ , with randomness arising solely from the Gaussian draws $Y_i$ . This validity holds regardless of the distribution of $\mu_{1:n}$ .
Robustness to Misspecification: Unlike standard EB methods that integrate over a potentially misspecified prior $P$ , ASSURE evaluates decisions based on the realized parameters. This allows practitioners to use EB models merely as a source of decision rule classes (e.g., linear shrinkage rules) and then tune them via ASSURE to ensure robustness.

B. Regret Guarantees
The paper establishes theoretical bounds on the regret (the difference between the optimal welfare in a class $\mathcal{D}$ and the welfare of the rule selected by maximizing $\hat{W}$ ):

General Rates: Under weak assumptions (VC-subgraph classes), ASSURE* achieves a regret of $\tilde{O}(n^{-1/2})$ . This matches the minimax lower bound up to logarithmic factors.
Fast Rates: Under additional "margin conditions" (where the optimal decision is well-separated and the welfare function is curved), the regret improves to $\tilde{O}(n^{-1})$ .
Comparison to Alternatives: The paper demonstrates that while standard Empirical Success rules (e.g., $Y_i > 0$ ) minimize worst-case frequency regret, they fail to achieve vanishing regret in the compound setting for many parameter sequences. Conversely, ASSURE* achieves vanishing regret by "borrowing strength" across units.

C. Robustness to Non-Gaussianity
The authors analyze the performance of ASSURE* when $Y_i$ are sample means of $m$ i.i.d. non-Gaussian variables. They show that the estimator's performance degrades gracefully as the Gaussian approximation becomes less accurate. Specifically, the bias term scales with $1/\sqrt{m}$ , and ASSURE* strictly dominates simple plug-in estimators (which do not exploit the approximate Gaussianity) when $m \gg \log(n)^4$ .

D. Ensembling Complex Rules
The framework extends to ensembling complex decision rules (e.g., combining parametric EB, nonparametric EB, and regression models) via leave-one-out cross-fitting. ASSURE can tune the weights of such ensembles, ensuring the combined rule performs no worse asymptotically than its best constituent.

4. Empirical Applications and Simulation Results

The paper validates the methodology through simulations and three economic applications:

Simulation (Misspecified Priors): In a setup where the true data-generating process involves a misspecified Gaussian prior, ASSURE* significantly outperforms plug-in Empirical Bayes and SURE-based tuning. While SURE minimizes squared error (leading to good posterior mean estimates), it incurs high welfare loss due to false positives/negatives in selection. ASSURE* directly optimizes the selection objective, achieving welfare close to the oracle.
Opportunity Atlas (Census Tracts): Applied to selecting Census tracts for economic mobility (Bergman et al., 2024), ASSURE* improves upon plug-in Fay-Herriot and linear shrinkage methods, particularly when the decision class is less flexible. It certifies that flexible EB models perform well but provides gains when simpler models are used.
Discrimination in Large Firms: In the context of Kline et al. (2022), where linear shrinkage rules are known to be conservative due to skewness in the true distribution of discrimination, ASSURE* partially corrects the selection threshold, reducing the welfare gap relative to a nonparametric EB benchmark compared to standard plug-in methods.
A/B Testing (Experimentation): Using a Netflix dataset, the authors apply ASSURE* to select A/B tests. The method suggests a much more lenient threshold (near $t=0$ ) than the conventional $t=1.96$ when implementation costs are low, and it can be used to infer the implicit cost level that justifies conventional significance thresholds.

5. Significance and Claims

The paper claims that ASSURE provides a robust alternative to standard Empirical Bayes for compound selection. Its significance lies in:

Decoupling Estimation and Selection: It allows the use of EB models as "working approximations" to generate candidate decision rules without requiring the model to be correct.
Direct Welfare Optimization: By optimizing an estimator of the actual selection utility rather than a proxy (like squared error), it avoids the "better prediction $\neq$ better selection" pitfall.
Robustness: The method offers theoretical guarantees that hold for fixed parameters, making it robust to misspecification of the prior distribution of $\mu_{1:n}$ .
Practical Utility: It offers a data-driven way to tune thresholds in settings ranging from policy design (housing vouchers) to corporate experimentation, often outperforming standard ad-hoc or plug-in methods.

The authors emphasize that while ASSURE inherits the versatility of SURE, it is specifically tailored for the non-differentiable nature of selection decisions, filling a gap where standard SURE is not directly applicable.