📊 statistics

Blending Proxy Metrics with a North Star

This paper proposes an optimal blending framework for A/B testing that dynamically balances proxy metrics and a "north star" based on experiment power and proxy quality, offering actionable insights for experiment design and demonstrating its real-world application at Netflix.

Original authors: Winston Chou

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Winston Chou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a captain steering a massive ship (a company like Netflix) through foggy waters. Your ultimate goal is to reach a specific, distant island called "The North Star" (true business success, like keeping customers for years). However, the fog is so thick that you can't see the island clearly until you are almost there.

To navigate, you have two tools:

The North Star Compass: It points directly to your destination, but it's very weak and jittery. You can only trust it if you sail for a very long time (run a huge experiment).
The Proxy Radar: It detects small ripples in the water (user clicks, engagement) that happen before you reach the island. It's super sensitive and gives you a clear signal immediately, but it doesn't always mean you are actually heading toward the island.

The Dilemma
In the past, companies had to choose: "Do I trust the weak North Star, or the noisy Proxy?"

If you trust the Proxy too much, you might steer toward a ripple that looks promising but leads to a dead end.
If you wait for the North Star, you might sail for years without making any decisions, missing out on opportunities.

The Solution: The "Smart Blend"
Winston Chou's paper proposes a clever navigation system that blends these two tools. Instead of picking one, the ship's computer automatically adjusts how much it trusts each tool based on how much data it has collected.

Here is how the "Smart Blend" works, using simple analogies:

1. The "Volume Knob" Analogy

Imagine the decision-making process is a radio with two volume knobs: one for the Proxy (clicks) and one for the North Star (retention).

When the experiment is small (low data): The North Star signal is too quiet to hear. The system turns the Proxy volume up and the North Star volume down. You make decisions based on the immediate, sensitive signals because you don't have enough data to see the big picture yet.
When the experiment is huge (high data): You have sailed far enough that the North Star is finally visible. The system automatically turns the North Star volume up and the Proxy volume down. You stop guessing based on ripples and start steering by the actual destination.

The paper proves that there is a mathematically "perfect" way to turn these knobs at every stage of the journey.

2. The "Taste Test" Analogy

Imagine you are a chef trying to create a new dish.

The North Star is the final taste test by a food critic (did they love it?). This takes time and is rare.
The Proxy is the smell of the kitchen while cooking. It's immediate and tells you if something is burning or smelling good.

If you only have a tiny sample (one spoonful), you can't trust the critic's opinion yet, so you rely on the smell. But if you have cooked a thousand pots, the critic's opinion becomes the most important thing. The paper's method tells you exactly how much to weigh the "smell" versus the "critic's taste" based on how many pots you've cooked.

3. The "Traffic Jam" Analogy (Designing Experiments)

The paper also changes how you plan your experiments.

If your Proxy is excellent (like a very accurate smell), you don't need to cook huge batches to know if the dish is good. You can run many small, quick experiments. This is like taking many small side roads to get around traffic.
If your Proxy is poor (the smell is unreliable), you can't trust it. You have to wait for the critic. This means you need to run fewer, but much larger experiments to get a clear signal.

Real-World Proof: Netflix

The author tested this at Netflix.

The North Star: "Plays" (did the user actually watch the movie?). This is hard to measure quickly.
The Proxy: "Clicks" (did the user click the title?). This is easy to measure instantly.

In Netflix's data, "Clicks" were much more sensitive than "Plays."

For small tests, the system relied mostly on Clicks.
For massive tests (millions of users), the system shifted to rely mostly on Plays.
The Result: By blending them, Netflix got better results than if they had used only Clicks or only Plays. They made fewer mistakes and found more successful features.

The Bottom Line

You don't have to choose between "fast but noisy" data and "slow but accurate" data. By using a blended approach that automatically shifts its trust from the fast data to the accurate data as you gather more information, you can make better decisions faster.

Key Takeaways from the Paper:

Don't pick a side: Use both metrics, but weight them differently.
Size matters: The bigger your experiment, the more you should trust the "North Star" (the true goal).
Quality matters: If your "Proxy" is very good, you can run more, smaller experiments. If it's bad, you need fewer, bigger ones.
The math works: The paper provides a formula to calculate exactly how to mix these metrics to get the best possible outcome.

Technical Summary: Blending Proxy Metrics with a North Star

Problem Statement
In digital experimentation (A/B testing), experimenters frequently face a trade-off between precision and alignment with long-term business goals. "North star" metrics (e.g., customer retention, revenue) directly measure business success but are often statistically insensitive to product interventions, requiring prohibitively large sample sizes to detect effects. Conversely, "proxy" metrics (e.g., user engagement, clicks) are more sensitive to interventions and allow for faster, smaller experiments but may not perfectly align with the ultimate business objective.

This creates a practical dilemma: when should experimenters trust the sensitive proxy, and when should they trust the noisy north star? Specifically, the paper addresses three questions:

Should proxies be relied upon even in large, infrequent experiments powered to detect effects on the true north star?
How should decisions be made when the proxy and the north star yield conflicting signals?
How does the quality of a proxy metric influence the optimal design of experimentation programs (e.g., sample size and frequency)?

Methodology
The author proposes an optimal "blending" framework that constructs a decision metric as a weighted linear combination of the estimated treatment effects on the proxy ( $\hat{P}$ ) and the north star ( $\hat{Y}$ ). The blended statistic is defined as:
$\hat{Z}(A) = A^\top \hat{X} = a \hat{P} + (1-a) \hat{Y}$
where $A = (a, 1-a)^\top$ represents the blending weights.

The core of the methodology is the derivation of the optimal weights $A^*$ that maximize the expected returns to the north star metric ( $Y$ ) under a decision rule that "launches" a treatment if $\hat{Z}(A)$ exceeds a significance threshold.

Theoretical Framework: The paper assumes true treatment effects $X = (P, Y)$ follow a Gaussian distribution with zero mean and covariance $\Sigma_X$ . Estimated effects $\hat{X}$ are observed with sampling noise, modeled as $\hat{X} | X \sim N(X, 2\Omega/n)$ , where $\Omega$ is the unit-level sampling variance and $n$ is the sample size per arm.
Optimal Weights: The optimal weights $A^*$ are derived by solving a constrained optimization problem to maximize expected returns. Proposition 1 provides a closed-form approximation for sufficiently large experiments:
$\tilde{A} \propto (\Sigma_X + (1 + z^2)2\Omega/n)^{-1}\Gamma$
where $\Gamma = \text{Cov}(X, Y)$ and $z$ is the critical value for the significance level. This formula reveals that the weights act as a penalized linear regression, where the penalty term depends on the ratio of within-experiment sampling variance to between-experiment structural variance.
Estimation: The paper outlines a procedure to estimate these weights from historical experiment data by estimating $\Sigma_X$ (using nonparametric, empirical Bayes, or hierarchical methods) and $\Omega$ , then applying the plug-in estimator.
Program Design Implications: The analysis extends to the design of experimentation programs. It demonstrates that the average return per unit of allocation is decreasing in sample size $n$ . Consequently, if a proxy is high-quality (excellent signal-to-noise), experimenters should run smaller and more frequent experiments. If the proxy is poor, they should run larger and fewer experiments to rely more on the north star.

Key Results

Simulation Performance: Through simulations, the blended approach consistently yields higher cumulative returns than strategies relying exclusively on the north star ( $a=0$ $a = 0$ ) or the proxy ( $a=1$ $a = 1$ ).
- At small sample sizes, the optimal weight shifts heavily toward the proxy due to its superior signal-to-noise ratio.
- As sample size increases, the optimal weight smoothly shifts toward the north star, eventually concentrating exclusively on the north star as $n \to \infty$ .
Statistical Power and False Positives:
- Power: The blended metric maintains high power across varying correlations ( $\rho$ ) between proxy and north star. In contrast, a proxy-only strategy suffers from low power when the structural correlation is low, even with large sample sizes.
- False Positives: A proxy-only strategy exhibits a false positive risk that increases with sample size when the proxy is misaligned with the north star. The blended metric controls the Type I error rate, interpolating between the proxy and north star rules and converging to the north star's error control as $n$ grows.
Robustness: The core findings hold even when treatment effects follow heavy-tailed distributions (e.g., multivariate t-distribution with $\nu=3$ ), though the Gaussian assumption facilitates the closed-form derivation.

Empirical Application
The methodology was applied to a real-world experimentation program at Netflix, using "clicks" as a proxy ( $P$ ) and "plays" as the north star ( $Y$ ).

Data: Analysis of 15 A/B tests and 50 treatment arms.
Findings: Clicks were found to be approximately six times more sensitive than plays (higher SE-ratio). However, the optimal blending weights are dynamic:
- For small experiments ( $n \lesssim 2$ million), the optimal strategy places 100% weight on clicks.
- At the median experiment size ( $n \approx 5$ million), the optimal blend assigns roughly 52% weight to clicks and 48% to plays.
- For very large experiments ( $n = 25$ million), the weight shifts to approximately 90% on plays.
Outcome: The blended decision rule dominated both single-metric rules across the entire observed range of sample sizes, demonstrating that incorporating the north star information improves returns even when the proxy is highly sensitive.

Significance and Claims
The paper claims to provide a rigorous, decision-theoretic framework for integrating proxy metrics into A/B testing when a contemporaneous but insensitive north star is available. Its primary contributions are:

Optimal Blending: It establishes that the optimal decision rule is not static but depends on the experiment's power (sample size). The weight on the north star should increase as the experiment grows larger.
Program Design: It offers actionable guidance on experimentation program design, suggesting that the quality of the proxy metric dictates the optimal scale and frequency of experiments.
Practical Implementation: It provides a concrete, estimable method for calculating these weights using historical data, validated by a real-world application at Netflix.

The author notes that the work assumes experiments are independently and identically distributed (i.i.d.) and relies on Gaussian assumptions for treatment effects, though simulations suggest robustness to heavy tails. Future work is suggested to explore non-i.i.d. settings and the use of observational data when historical experiment data is unavailable.