🤖 machine learning

Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification

This paper presents a practical framework combining post-stratification and CUPED to reduce variance in heavy-tailed monetization metrics for ranking experiments, enabling ShareChat to achieve equivalent statistical confidence with approximately 45% less traffic while improving decision stability.

Original authors: Neeti Pokharna, Olivier Jeunen, Yatharth Saraf, Aleksei Ustimenko

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Neeti Pokharna, Olivier Jeunen, Yatharth Saraf, Aleksei Ustimenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Whale" Problem in A/B Testing

Imagine you are running a test to see if a new feature in a video app makes people spend more money. You split your users into two groups: Group A sees the old version, and Group B sees the new version.

Usually, you just add up all the money spent by everyone in Group A and compare it to Group B. But in apps like ShareChat (where users buy virtual gifts for creators), the spending habits are wildly uneven.

Think of it like a fishing trip:

99.9% of the fish you catch are tiny minnows that weigh almost nothing.
0.01% of the fish are massive "whales" (super-rich users) that weigh as much as a car.

In a standard test, if one of these "whales" happens to land in Group A by pure luck, Group A looks like a huge success. If that same whale lands in Group B, Group B looks like a failure. Because these whales are so heavy, they dominate the math. They create so much "noise" (randomness) that you can't tell if the new feature actually works or if you just got lucky with where the whales landed.

This means you have to run the test for a very long time or get a massive amount of traffic just to get a reliable answer. Often, you can't get enough traffic, so you can't make decisions.

The Solution: Sorting the Fish Tank

The authors propose a clever way to fix this called Post-Stratification, combined with a technique called CUPED.

Here is how it works, step-by-step:

1. Sorting the Fish (Stratification)

Instead of looking at the whole tank of fish as one big group, you sort them into separate buckets before you start the test based on how they behaved in the past.

Bucket 1: The "Whales" (the top 0.01% of spenders).
Bucket 2: The "Regulars" (everyone else).

2. Weighing the Buckets

Here is the magic trick. When you calculate the final result, you don't just add up the buckets equally. You give the buckets different weights based on how many people are actually in the real world.

Since "Whales" are rare, their bucket gets a tiny weight (like 0.0001).
Since "Regulars" are common, their bucket gets a huge weight.

The Analogy: Imagine you are judging a cooking contest. If one judge is a billionaire who gives a score of 1,000,000, and 999 other judges give scores of 5, the billionaire's score ruins the average.

Old Way: You take the average of all scores. The billionaire's score dominates.
New Way: You say, "The billionaire is one person, so their score only counts as 0.001 of the total. The 999 regular judges count for 99.9%." Now, the billionaire's wild score doesn't swing the result wildly.

3. Smoothing the Data (CUPED)

Inside each bucket, they also use a trick called CUPED. This is like using a user's past spending as a "baseline."

If a user usually spends \ $100, and in the test they spend \$ 110, CUPED says, "Okay, that's just their normal high spending, not necessarily because of the new feature."
It subtracts out the predictable parts of the spending, leaving only the real changes caused by the experiment.

The Results: Faster Decisions with Less Traffic

By using this method, the authors achieved some impressive results at ShareChat:

Less Noise: They reduced the "noise" (variance) in their data by 99%.
Faster Tests: They can now get the same level of confidence with 45% less traffic.
- Analogy: It's like being able to hear a whisper clearly in a noisy room without needing to shout. You don't need a bigger crowd to hear the truth; you just need a better way to listen.
Reliability: They tested this on over 40 real-world experiments. The method helped them spot small, real improvements that would have been invisible before.

Important Rules and Warnings

The paper also points out when NOT to use this method:

Don't use it if you are testing a feature specifically for Whales.
- Analogy: If you are testing a new "VIP Lounge" specifically for the rich whales, you don't want to down-weight their importance. If you do, you might miss the fact that the VIP Lounge is amazing for them. This method is for general improvements that help the whole user base.
Don't use it if the "Whales" are the only ones changing.
- If the new feature only affects the top 0.01% of users, this method will hide that effect because it treats the whales as a tiny, low-weight group.

Summary

The paper presents a practical tool for companies running A/B tests on money-related metrics. By sorting users into groups and down-weighting the rare, super-spending outliers, they can stop the "whales" from hijacking the results. This allows them to make faster, more confident decisions about their product without needing millions more users.

Technical Summary: Variance Reduction for Heavy-Tailed Monetization Metrics via Post-Stratification

1. Problem Statement

In large-scale Information Retrieval (IR) and ranking systems, downstream monetization metrics (e.g., Gross Merchandise Value or GMV, creator earnings) are critical for evaluating system performance. However, these metrics typically exhibit heavy-tailed distributions, where a minute fraction of users (e.g., the top 0.01%) dominate both the mean and the variance.

This distributional characteristic creates significant challenges for standard A/B testing:

Low Statistical Power: Under realistic traffic allocations (e.g., 1–20%), the heavy tail prevents the Central Limit Theorem (CLT) from converging quickly. Empirical analysis shows that $z$ -statistics deviate significantly from the standard normal distribution at low traffic levels.
Unreliable Inference: The resulting tests are often severely underpowered, requiring prohibitively long durations or massive traffic increases to detect true treatment effects. Conversely, standard variance reduction techniques like CUPED (Controlled Experiments Using Pre-Experiment Data) offer limited gains because pre-experiment signals alone cannot fully predict the extreme variance driven by "whale" users.
Conservatism: The failure of normal approximations leads to conservative hypothesis testing, where the False Positive Rate (FPR) drops well below the nominal 5% (observed at 1–2%), indicating a lack of sensitivity rather than robustness.

2. Methodology

The authors propose a production-ready framework that combines Post-Stratification with CUPED to address these challenges. The core idea is to partition the user population into homogeneous strata based on pre-experiment behavior, thereby isolating the high-variance tail users and reducing their disproportionate influence on the overall variance.

Key Components of the Framework:

Stratum Definition: Users are partitioned into discrete strata based on pre-period GMV (e.g., 30-day historical revenue). A critical distinction is that strata are defined before the experiment begins, ensuring treatment assignment does not affect stratum membership (preserving unbiasedness).
- Typical strata include: "Tail" (high spenders), "Non-tail" (regular users), and "New/Non-spenders."
Winsorization: To mitigate the impact of extreme outliers (e.g., the top 0.001% of users) that may dominate variance even within strata, values beyond the 99.999th percentile are winsorized. This step is conservative, affecting fewer than 5 users per experiment.
CUPED within Strata: Within each stratum, CUPED is applied to adjust the post-experiment GMV using pre-experiment GMV as a covariate. This removes predictable variation within the homogeneous groups.
- Formula: $GMV_{CUPED} = GMV - \frac{cov(GMV, PreGMV)}{\sigma^2_{PreGMV}} \cdot (PreGMV - \mu_{PreGMV})$
Post-Stratified Estimation: The treatment effect is estimated by aggregating stratum-level CUPED-adjusted effects using population-level weights ( $w_s = N_s/N$ $w_{s} = N_{s} / N$ ), where $N_s$ $N_{s}$ is the total number of users in stratum $s$ $s$ across the entire platform (not just the experiment sample).
- Estimator: $\hat{\tau}_{PS} = \sum_{s \in S} w_s \cdot \hat{\tau}_{CUPED, s}$

Theoretical Properties:

Unbiasedness: The estimator remains unbiased because strata are defined using pre-treatment information, and CUPED is unbiased.
Variance Reduction: The variance of the estimator is reduced by the squared weight factor ( $w_s^2$ ). For a tail stratum with a tiny population weight (e.g., $w \approx 0.0001$ ), its contribution to the total variance is suppressed by a factor of $\sim 10^{-8}$ , even if the internal variance of that stratum is enormous.

3. Experimental Validation and Results

The framework was deployed at ShareChat and Moj across >40 production A/B tests involving ranking and recommendation systems, with each experiment covering over 1 million users.

Key Findings:

Variance Reduction: The combined approach achieved a 99.3% to 99.7% reduction in variance compared to raw GMV. In contrast, CUPED alone achieved only ~48% reduction.
Statistical Sensitivity: The median relative $z$ -score increased to 1.35–1.36. This implies that the method requires approximately 45% less traffic to achieve the same statistical confidence as standard metrics.
Minimum Detectable Effect (MDE): At 10% traffic allocation, the MDE dropped from ~136% of the mean (raw GMV) to ~10% (post-stratified), enabling the detection of realistic, subtle treatment effects previously masked by noise.
Type-I Error:
- In simulation (A/A tests on static snapshots), the False Positive Rate was 0%.
- In production (live traffic), the empirical FPR was 6.1%, slightly above the nominal 5%. The authors attribute this to finite-sample variance estimation in strata and real-world noise, noting that raw metrics were severely underpowered (FPR 1–2.6%). They accept this slight elevation as a trade-off for a massive reduction in Type-II errors (False Negatives).
Robustness:
- Threshold Sensitivity: Variance reduction remained above 95% across various tail cutoff thresholds (99.9th to 99.995th percentiles).
- Temporal Consistency: Effects detected in Week 1 maintained 91% directional agreement in Week 2.
- Failure Modes: In one instance, variance increased when a treatment effect was concentrated entirely within the tail stratum, reinforcing that this method is not suitable when the primary business goal is to optimize specifically for "whale" users.

4. Key Contributions

Production Framework: A practical, deployable system for variance reduction in heavy-tailed monetization metrics that integrates seamlessly into existing experimentation pipelines.
Methodological Synthesis: Demonstrates that combining post-stratification (to handle population heterogeneity) with CUPED (to handle within-stratum correlation) yields superior variance reduction compared to either technique alone.
Empirical Evidence: Provides large-scale validation showing that this approach enables reliable decision-making with 40–50% less traffic, significantly accelerating iteration cycles for ranking improvements.
Guidance on Estimands: Clarifies the trade-off in estimand interpretation. By down-weighting tail users, the method optimizes for the "broad user experience" rather than "whale-specific" outcomes. The authors recommend surfacing both raw and post-stratified metrics to detect directional disagreements (which signal tail-concentrated effects).

5. Significance and Limitations

The paper argues that for IR systems where monetization is driven by rare, high-impact events, standard A/B testing is often insufficient. The proposed framework offers a solution that preserves the original metric scale and population estimand while drastically improving statistical power.

Limitations and Guardrails:

Not for Tail-Specific Goals: The method is explicitly not recommended for experiments targeting tail users (e.g., VIP programs), as the post-stratified estimand systematically down-weights the population of interest.
Data Requirements: Requires robust pre-period data pipelines to calculate population weights and frozen stratification thresholds to prevent p-hacking.
Stratum Drift: Monitoring is required to detect significant changes in stratum sizes that could corrupt the metric.

In conclusion, the authors present a validated approach that transforms heavy-tailed monetization evaluation from a low-power, slow process into a sensitive, stable mechanism, enabling faster and more reliable product iteration in large-scale recommendation systems.