Minimizing Type 2 Errors in an Experiment-Rich Regime via Optimal Resource Allocation

This paper addresses the challenge of allocating limited resources across multiple concurrent experiments to minimize worst-case Type II errors, proposing robust optimization frameworks and a practical "Surrogate-S" procedure that outperforms traditional mean-squared-error-based and naive plug-in methods in detecting meaningful treatment effects.

The Gist

Imagine you are the manager of a massive, high-tech factory. Your job is to test hundreds of new ideas every day—maybe a new button color, a faster checkout process, or a different ad layout. You have a limited number of workers (users) to test these ideas on. This is what the paper calls an "experiment-rich regime."

The big question is: How do you split your limited workers among all these different tests so you don't miss out on the winners?

The Old Way: The "Average Accuracy" Trap

For a long time, companies used a strategy based on estimation accuracy. Think of this like trying to measure the height of a group of people.

• If you have a very wobbly ruler (high variance), you need to measure that person many times to get a good average.
• If you have a steady ruler (low variance), you only need a few measurements.

So, the old rule was: "Give more workers to the tests that are 'wobbly' (have high variance), regardless of whether the idea is actually good."

The problem? This is like spending all your money measuring the height of a person who is already known to be short, just because your ruler was shaky. You might end up with a very precise measurement of a boring idea, while completely missing a brilliant, game-changing idea that was hard to detect because it was "noisy."

In statistical terms, this old method minimizes Mean Squared Error (MSE). But in business, you don't just want to measure things; you want to find the winners.

The New Goal: Avoiding the "Missed Opportunity"

The authors argue that in the early screening phase, the biggest risk isn't being slightly inaccurate; it's making a Type 2 Error.

• Type 1 Error (False Positive): You think a bad idea is good. (Cost: Wasted money).
• Type 2 Error (False Negative): You think a great idea is bad, so you throw it away. (Cost: Missing a million-dollar innovation).

The paper says: "Let's stop worrying about being perfectly precise. Let's worry about not missing the winners."

Their goal is to allocate workers so that every single test has a fair shot at proving it's a winner. They want to minimize the chance that any good idea gets rejected.

The Solution: The "Safety Margin" Strategy

Here is where it gets tricky. In the real world, you don't know how "wobbly" (variance) a test will be until you run it. So, you run a tiny "pilot" test first to guess the variance.

The Naive Mistake:
Most people just take the result of the pilot test and say, "Okay, the variance is 5.0, so I'll use 5.0 for the big test."

• The Analogy: Imagine you are packing for a trip. You check the weather forecast, and it says "50% chance of rain." You pack a light jacket. But forecasts are often wrong, and sometimes it rains harder than predicted. If you only pack for the average, you might get soaked.

The Paper's Fix: The "Inflation Factor"
The authors say: "Don't trust the pilot test blindly. Assume the worst-case scenario and pack a bigger umbrella."

They propose a method where you inflate (increase) the variance estimate from the pilot test by a specific "safety factor."

• If a test is already hard to detect (it's a "tough" idea), you give it extra workers.
• If a test is easy, you give it fewer.
• Crucially, you add a buffer to account for the fact that your pilot test might have underestimated the difficulty.

The Three "Risk Personalities"

The paper offers three different ways to calculate this safety buffer, depending on how risky your boss is:

1. The "Safety First" Boss (TOL - Tolerance):

   • Goal: "I want to be 90% sure that we don't miss a winner by more than a tiny bit."
   • Action: We calculate the buffer to guarantee that even in a bad luck scenario, we stay within a safe zone.

2. The "Confidence" Boss (CONF):

   • Goal: "I have a strict rule: We cannot miss a winner by more than 5%. How sure can we be that we follow this rule?"
   • Action: We adjust the buffer to make that 5% rule hold true as often as possible.

3. The "Average Case" Boss (EXP):

   • Goal: "I don't care about extreme bad luck. I just want the average number of missed winners to be as low as possible."
   • Action: We calculate the buffer to minimize the average cost of mistakes.

The "Surrogate-S" Magic Trick

Calculating these safety buffers is mathematically very hard (like solving a Rubik's cube while juggling). The authors created a clever shortcut called Surrogate-S.

• The Metaphor: Instead of trying to predict the exact weather for every single day of the year (which is impossible), they built a simplified model that uses the pilot data to create a "good enough" safety margin.
• The Result: This shortcut is so smart that it performs almost as well as if you knew the true weather (the "Oracle") from the start, but it's fast enough to run on a computer for thousands of tests.

Why This Matters

In the past, companies might have been so focused on "measuring things precisely" that they accidentally threw away their best ideas because they didn't give them enough traffic to prove themselves.

This paper gives managers a new rulebook:

1. Stop allocating resources just to get precise averages.
2. Start allocating resources to ensure you don't miss the next big thing.
3. Use a "safety margin" (inflation factor) to protect against bad luck in your pilot tests.

By doing this, companies can turn their limited traffic into a powerful engine for discovery, ensuring that the best innovations get the spotlight they deserve.

Technical Summary

1. Problem Statement

The paper addresses the challenge of allocating limited experimentation resources (a fixed pool of users $N$) across $M$ concurrent A/B tests in an "experiment-rich" regime (e.g., large tech platforms running thousands of tests annually).

• Context: Traditional resource allocation literature focuses on minimizing the Mean Squared Error (MSE) of estimated treatment effects. This approach allocates more samples to experiments with higher outcome variance to ensure estimation precision.
• The Gap: In the initial screening phase of experimentation, the primary managerial goal is not precise estimation but detection: determining whether an effect exceeds a meaningful threshold ($\theta_i$). In this binary decision context, the critical risk is a Type 2 error (false negative)—failing to detect a truly beneficial innovation.
• Objective: The authors propose shifting the optimization objective from minimizing worst-case MSE to minimizing the worst-case Type 2 error across the portfolio of experiments. This ensures a uniform baseline of statistical power, preventing any single experiment from being underpowered.
• Complication: In practice, the outcome standard deviations ($\sigma_i$) are unknown and must be estimated from pilot data ($S_i$). A naive approach of plugging these estimates directly into the allocation formula leads to significant power loss because sample variance estimates are right-skewed and often underestimate the true variance.

2. Methodology

The paper develops a framework for optimal allocation under both known and unknown variance scenarios, utilizing robust optimization principles to handle estimation uncertainty.

A. Benchmark Case: Known Standard Deviations ($\vec{\sigma}$)

When $\vec{\sigma}$ is known, the authors derive a closed-form Power-Optimal Allocation ($\vec{n}^*$) that minimizes the maximum Type 2 error ($\beta$).

• Result: The optimal sample size for experiment $i$ is proportional to the square of the ratio of its standard deviation to its minimum detectable gap ($\Delta_i$):
  $$n_i^* \propto \left( \frac{\sigma_i}{\Delta_i} \right)^2$$
• Insight: This allocation equalizes the Type 2 error across all experiments. Unlike MSE-optimal allocation (which depends only on $\sigma_i$), the power-optimal allocation explicitly accounts for the signal strength ($\Delta_i$). Experiments with small effect sizes or high noise receive more resources.

B. Realistic Case: Unknown Standard Deviations ($\vec{\sigma}$)

When $\sigma_i$ is unknown, the platform uses pilot data to estimate $S_i$.

• The Naive Plug-in Failure: Simply substituting $S_i$ for $\sigma_i$ in the power-optimal formula ignores the uncertainty in $S_i$. Since $S_i$ often underestimates $\sigma_i$, the resulting allocation is too small, leading to underpowered tests.
• Correction Factors: The authors introduce inflation factors ($k_i \ge 1$) to adjust the pilot estimates: $\tilde{\sigma}_i = \sqrt{k_i} S_i$. The goal is to select $k_i$ such that the realized Type 2 error remains close to the oracle optimum.

C. Optimization Frameworks

To select the optimal correction factors $\vec{k}$, the authors propose three distinct frameworks reflecting different risk preferences:

1. Tolerance-based (TOL): Minimizes the tolerance $\delta$ such that, with high probability ($\gamma$), the realized max Type 2 error is within $\beta^* + \delta$.
2. Confidence-based (CONF): Maximizes the probability ($\gamma$) that the realized error stays within a pre-specified tolerance $\delta$.
3. Expectation-based (EXP): Minimizes the expected value of the realized maximum Type 2 error (risk-neutral).

D. Surrogate Reformulations (Robust Optimization)

The original TOL, CONF, and EXP problems are computationally intractable for large $M$ due to complex stochastic constraints involving non-central distributions.

• Solution: The authors derive tractable surrogate reformulations (R-TOL, R-CONF, R-EXP) inspired by robust optimization.
• Mechanism: They replace the stochastic constraints with deterministic upper bounds derived from confidence intervals of the chi-squared distribution. This transforms the problems into convex optimization programs with separable structures.
• Surrogate-S: A fully data-dependent algorithm is proposed. It replaces the unknown true variances in the surrogate formulations with pilot estimates ($S_i$) and solves the convex program to determine the correction factors.

3. Key Contributions

1. Shift in Objective: The paper establishes that minimizing Type 2 error (detection power) is distinct from and often superior to minimizing MSE for screening phases. It proves that MSE-optimal allocations can be highly inefficient for detection under tight resource constraints.
2. Power-Optimal Allocation Characterization: Derives the closed-form solution for known variances, showing that resources should be allocated based on the "statistical difficulty" ($\sigma_i / \Delta_i$) rather than just variance.
3. Correction Factor Theory: Extends the concept of variance inflation (previously used for single experiments) to multi-experiment portfolios. It reveals that optimal inflation is asymmetric: easier experiments (lower difficulty index) should be inflated more than harder ones to stabilize the portfolio's worst-case performance.
4. Tractable Algorithms: Develops robust optimization-based surrogate formulations that convert intractable stochastic programs into solvable convex problems.
5. Surrogate-S Procedure: Proposes a practical, fully data-dependent implementation that achieves near-oracle performance without requiring knowledge of true variances.

4. Key Results

• MSE vs. Power: Numerical simulations show that under tight budgets, MSE-based allocation results in significantly higher Type 2 errors (e.g., ~0.75 vs. ~0.10 in specific scenarios) compared to the power-optimal approach.
• Naive vs. Corrected: The naive plug-in method suffers substantial power loss. The proposed correction factors significantly mitigate this, bringing performance close to the theoretical oracle (which knows true $\sigma$).
• Asymmetry of Inflation: In a two-experiment setting, the optimal inflation ratio $r = k_1/k_2$ deviates from 1. If Experiment 1 is statistically easier than Experiment 2, the optimal strategy inflates Experiment 1's variance estimate more ($r > 1$) to balance the volatility of the maximum Type 2 error.
• Performance of Surrogate-S:
  • TOL: Achieves the same confidence level with a 60% reduction in required error tolerance compared to the naive method.
  • CONF: Increases the probability of meeting a strict error bound from ~37% (naive) to ~98% (Surrogate-S).
  • EXP: Reduces the average excess error by over 60% compared to the naive approach.

5. Significance

• Managerial Impact: Provides a principled method for platforms to maximize the value of their testing resources. By explicitly controlling Type 2 errors, companies can avoid discarding valuable innovations due to underpowered tests.
• Theoretical Advancement: Bridges the gap between statistical decision theory (hypothesis testing power) and operations research (resource allocation), moving beyond the standard MSE minimization paradigm.
• Scalability: The proposed Surrogate-S method is computationally efficient (convex optimization), making it applicable to large-scale portfolios with hundreds or thousands of concurrent experiments, a common scenario in modern tech companies.
• Robustness: The framework explicitly handles the uncertainty of pilot data, offering a safeguard against the systematic underestimation of variance that plagues standard A/B testing practices.