📊 statistics

Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments

This paper presents a cost-effective, system-level solution for Pinterest's experimentation platform that scales LLM-based content prevalence measurement across hundreds of concurrent A/B tests by implementing a single, continuously refreshed global calibration of ML score buckets, thereby enabling daily, high-fidelity tracking for over 20 times more experiment arms than traditional per-experiment LLM labeling within the same budget.

Original authors: Zehao Xu, Tony Paek, Kevin O'Sullivan, Attila Dobi

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zehao Xu, Tony Paek, Kevin O'Sullivan, Attila Dobi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, bustling city where millions of people walk through the streets every day. In this city, there are thousands of different shops, parks, and billboards, and the city planners want to know exactly how much time people spend looking at specific types of things—like "gardening tools" or "AI-generated art." This is the world of prevalence measurement: figuring out what fraction of the total "impressions" (or glances) belong to a certain category.

To make decisions about the city, planners often run A/B experiments. This is like testing two different versions of a street sign: one version for half the people, another for the other half, to see which one makes people happier or more engaged. Usually, to know if a sign is working, you need to count the results. But here's the catch: sometimes the thing you want to count is hard to measure. You can't just ask every single person what they saw; it would take forever and cost a fortune. In the digital world, this "hard to measure" task is often done by Large Language Models (LLMs)—super-smart AI computers that can read a picture or a post and tell you, "Yes, this is about gardening," or "No, this is not." While these AI judges are incredibly accurate, asking them to look at every single item in a city with billions of daily visitors is too expensive and slow to do for every single experiment.

This is where the paper by Zehao Xu and his team at Pinterest comes in. They faced a problem: they were running hundreds of experiments every day, and they needed to know the "prevalence" of certain content types (like how many pins were about AI or safety issues) for every single experiment group. They couldn't afford to pay the AI to label every single item for every single test. So, they built a clever system that acts like a "global translator." Instead of asking the expensive AI to check every single item in every new experiment, they ask the AI to check a representative sample of the whole city once a day. They use that sample to teach a cheaper, faster model how to guess the prevalence. Then, they use that fast model to measure everything else instantly. The result? They can track changes in content exposure for hundreds of experiments every day, catching tiny shifts that a single, expensive check would miss, all without spending extra money on AI labels.

The Problem: The "Gold Standard" is Too Expensive

In the world of online media, companies like Pinterest need to balance keeping users engaged with making sure they aren't seeing too much of certain types of content (like unsafe images or low-quality spam). To do this, they run A/B tests. In these tests, they tweak the algorithm to see if it changes how much of a specific type of content users see.

To measure this, the "gold standard" is to use an AI (an LLM) to look at a sample of content and label it perfectly. This is like having a team of expert art critics examine every painting in a gallery to count how many are landscapes. It's accurate, but it's slow and incredibly expensive. If you have hundreds of experiments running at the same time, and you need to check the content for every single group in every single experiment, the cost would be astronomical. You simply can't afford to hire the "experts" to look at everything, every day.

Furthermore, the changes companies care about are often very small. If a new algorithm reduces the amount of "AI-generated art" users see by just 2% or 5%, a single, expensive check might not be precise enough to tell if that change is real or just random noise. It's like trying to hear a whisper in a noisy room with a single, expensive microphone; you might miss it entirely.

The Solution: A Global Calibration Map

The team at Pinterest realized they didn't need to ask the expensive AI to label everything for every experiment. Instead, they needed a way to "calibrate" a cheaper, faster method using the expensive AI's wisdom.

Think of it like this: Imagine you have a super-accurate but slow thermometer (the LLM) and a cheap, fast thermometer (the ML model score). The cheap thermometer gives you a number, but it's not perfectly accurate. However, if you use the expensive thermometer to check the temperature in a few specific spots every day, you can create a map that tells you exactly how to correct the cheap thermometer's reading for any location.

The paper describes a system that does exactly this:

Global Calibration: Instead of labeling items for each experiment separately, they run a daily, global sampling process. They use the expensive LLM to label a representative sample of the entire platform's traffic.
Bucketing: They take the scores from the cheap, fast model (which predicts how likely an item is to be, say, "AI-generated") and group them into "buckets" (e.g., 0–10%, 10–20%, etc.).
The Translation: Using the daily LLM labels, they calculate exactly what percentage of items in each bucket are actually "AI-generated." This creates a lookup table: "If the cheap model says an item is in the 80–90% bucket, there is a 95% chance it is actually AI-generated."
Instant Measurement: Now, for any new experiment, they don't need to call the expensive AI. They just look at the cheap model's scores for the items in that experiment, see which buckets they fall into, and use the pre-calculated map to instantly know the prevalence.

The Results: Catching the Whispers

The team tested this system against the "gold standard" LLM measurements in real-world experiments. The results were impressive:

Scale: The system is currently serving about 100 experiments and 250 different groups (arms) every single day.
Efficiency: Compared to the old way of doing a single, expensive check per experiment, this new system provides daily measurements for over 20 times more concurrent experiments using the same labeling budget.
Accuracy: In about 300 production audits, the system's 95% confidence interval (a statistical range of certainty) contained the expensive LLM's answer 92% of the time. This means the cheap, fast method is almost as reliable as the expensive one.
Sensitivity: The most exciting finding was about detecting small changes. In one experiment, the new system detected a small but consistent 4.2% relative reduction in a specific content type. A single, expensive LLM check on that same experiment would have missed this entirely because the noise was too high. By aggregating daily data, the new system could hear the "whisper" that the expensive microphone missed.

Why It Matters

This isn't just about saving money; it's about making better decisions. Before this system, teams might have to skip measuring content prevalence in many experiments because it was too costly. Or, they might make decisions based on a single, noisy data point that missed small but important trends.

By "calibrating globally and measuring everywhere," the team created a system where the cost of the expensive AI is paid once (or amortized) and then reused thousands of times. It turns a slow, expensive process into a fast, daily routine. This allows product teams to see the full picture of how their changes affect content exposure, ensuring that the platform remains safe, high-quality, and engaging for everyone, without breaking the bank.

The paper emphasizes that this is a system-level achievement, not just a new math formula. It's about building a pipeline that takes a recurring, expensive task and turns it into a reusable asset. While the paper notes that LLM costs might drop in the future, making direct labeling cheaper, the logic of this system—using a global calibration to scale measurements—remains a powerful tool for managing complex, large-scale experiments today.

Technical Summary: Calibrate Globally, Measure Everywhere

Problem Statement
Online media platforms like Pinterest rely on A/B experiments to evaluate product changes. A critical metric for these experiments is prevalence: the fraction of impressions associated with specific content attributes (e.g., "Food-Recipe" or "Gen-AI Generated"). While Large Language Models (LLMs) provide high-fidelity labeling for these attributes, running LLM-based labeling for every experiment, every arm, every segment, and every day is cost-prohibitive. Directly applying LLM labeling to thousands of concurrent experiments is operationally infeasible. Furthermore, single-shot LLM measurements per arm often lack the statistical power to detect small but meaningful treatment effects (2–5% relative shifts) due to sampling uncertainty.

Methodology: The ML Score Bucket Surrogate
The authors propose a system-level solution rather than a new statistical estimator. The core methodology involves using a Machine Learning (ML) model score as a surrogate outcome, calibrated against a recurring, global stream of LLM-labeled data.

Global Calibration: Instead of labeling per experiment, the system maintains a single global calibration of ML score buckets.
- Bucketization: Model scores ( $m_{i,k}$ ) for a category $k$ are discretized into $B$ buckets (e.g., $[0, 0.1), [0.1, 0.2), \dots$ ).
- Calibration Sample: A daily, platform-wide LLM-labeled stream is used to estimate the bucket-level prevalence ( $\hat{P}_{k,b}$ ), defined as the probability that an item in bucket $b$ truly belongs to category $k$ .
- Sampling Design: To ensure sufficient data in high-score buckets (which are often rare), the sampling weights are proportional to $I_i \times m_{i,k}$ (impressions $\times$ model score), rather than just impressions. This balances the sample distribution across buckets.
- Estimation: The bucket-level prevalence is estimated using the Hansen–Hurwitz estimator on the PPSWOR (Probability Proportional to Size Without Replacement) sample.
Surrogate Application:
- For any specific experiment arm $S$ , the system calculates the share of impressions falling into each score bucket ( $c_{k,b}(S)$ ) directly from impression logs.
- The arm-level prevalence is computed as a weighted sum: $\hat{P}_k(S) \approx \sum_{b} c_{k,b}(S) \cdot \hat{P}_{k,b}$ .
- This computation is deterministic and executed via SQL, requiring no new LLM calls for the specific experiment.
Variance and Inference:
- The system propagates variance from the bucket-level calibration to the final estimate.
- Delta-Focused Dashboard: To detect small shifts, the system aggregates daily deltas ( $\Delta(d)$ ) between treatment and control arms. It uses a sign test on these daily deltas to determine statistical significance, effectively filtering out the noise inherent in absolute prevalence estimates caused by calibration uncertainty.

Key Contributions

System-Level Architecture: The primary contribution is a deployed measurement system that amortizes the cost of expensive LLM labeling across hundreds of concurrent experiments.
Zero Incremental Cost: In Pinterest's deployment, the system reuses an existing daily LLM workflow used for platform-level monitoring, incurring zero incremental labeling cost for per-experiment metrics.
Scalability: The system serves ~100 experiments and ~250 arms daily across six content categories. It provides daily per-arm prevalence for over 20x more concurrent arms than would be possible with a per-experiment LLM budget.
Sensitivity: By aggregating daily deltas, the system recovers small (2–5%) relative shifts that single-shot LLM measurements would miss due to wide confidence intervals.

Results and Validation
The system was validated through production audits against the LLM-based reference estimator on live A/B experiments:

Accuracy: Across ~300 audits, the surrogate's 95% confidence interval contained the LLM-based reference point estimate in 92% of evaluations. The intervals overlapped in 96% of cases.
Case Studies:
- Experiment A (Filtering): The surrogate correctly detected significant reductions in prevalence (approx. 19–36% drops) for filtered categories, with estimates falling within the LLM reference's confidence intervals.
- Experiment B (UI Change): The surrogate correctly identified no significant shift in prevalence, aligning with the null hypothesis.
- Experiment C (Small Shift): A single per-arm LLM measurement failed to detect a ~4% relative reduction (p-value 0.31). The surrogate, via daily delta aggregation, detected this shift as statistically significant (p-value < 0.001).
Latency: The surrogate provides results in <10 seconds (interactive latency) via SQL, compared to 10–24 hours for LLM-based processing.

Significance and Claims
The paper claims its significance lies in scaling LLM-based measurement. It argues that while LLMs are excellent "judges," they are too expensive to be the default metric for every experiment. By treating the ML score as a surrogate and calibrating it globally, the system enables:

High-Fidelity Measurement at Scale: Extending LLM-quality insights to billions of daily impressions without proportional cost increases.
Operational Feasibility: Making prevalence a standard, daily metric for product teams rather than a rare, one-off analysis.
Cost Amortization: Shifting the cost model from "pay per experiment/arm" to "pay once for global calibration," making high-quality measurement accessible for all downstream experiments.

The authors note that while LLM costs may eventually decrease, the system pattern of separating absolute levels from deltas and amortizing calibration across concurrent experiments remains a valuable architectural pattern for production measurement systems.

The Problem: The "Gold Standard" is Too Expensive

The Solution: A Global Calibration Map

The Results: Catching the Whispers

Why It Matters

Technical Summary: Calibrate Globally, Measure Everywhere

More like this