📊 statistics

Measuring the Prevalence of Policy Violating Content with ML Assisted Sampling and LLM Labeling

This paper presents a design-based measurement system that combines ML-assisted probability sampling with LLM labeling to efficiently and accurately estimate the prevalence of policy-violating content across various user segments while maintaining statistical unbiasedness and cost-effectiveness.

Original authors: Attila Dobi, Aravindh Manickavasagam, Benjamin Thompson, Xiaohan Yang, Faisal Farooq

Published 2026-08-18

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Attila Dobi, Aravindh Manickavasagam, Benjamin Thompson, Xiaohan Yang, Faisal Farooq

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

On the vast digital landscapes where billions of people share photos, ideas, and stories every day, a quiet but critical challenge persists: how do you know what your users are actually seeing? For years, platforms have relied on user reports to flag harmful content, but this method is like trying to measure the depth of a river by counting only the people who shout for help. Many dangers go unreported because users do not notice them, do not know how to report them, or simply do not want to engage with the system. To truly understand the safety of a platform, teams need to measure prevalence—the actual fraction of times a user encounters content that breaks the rules. This requires looking at the entire stream of content, not just the parts that have already been flagged. The problem is that harmful content is often rare, and checking every single item by hand is too slow and expensive to do every day.

A team of researchers at Pinterest has developed a new way to solve this, creating a system that combines smart sampling with advanced artificial intelligence to measure safety with unprecedented speed and accuracy. Instead of waiting for complaints, they take a daily, representative snapshot of everything shown to users. They use a statistical method that focuses their limited checking budget on the most likely candidates for violation, ensuring they find enough examples to be sure of their numbers without checking everything. Then, they use a large language model—an AI trained to understand text and images—to label these samples, acting as a tireless, consistent reviewer. By combining these techniques, they can produce a daily report on how often users see harmful content, complete with a measure of how precise that number is. This allows safety teams to spot emerging problems immediately, test whether their fixes are working, and understand exactly where and when risks appear, all without waiting weeks for human reviewers to catch up.

The core of this system is a clever way of choosing which pieces of content to examine. In a sea of billions of daily views, finding a rare violation is like looking for a specific type of leaf in a forest. If you pick leaves at random, you might walk for miles and find nothing. The researchers instead use a "weighted" approach. They look at two main clues: how many times a piece of content has been shown to people, and a risk score generated by the platform's existing safety models. They combine these clues to create a list where items that are both popular and risky are more likely to be picked for review. This does not mean they only check risky items; they still pick from the whole population, but they tilt the odds slightly to ensure they get enough examples of violations to make a solid statistical estimate. This method allows them to use a fixed number of checks each day and still get a clear picture of the whole platform, even when violations are extremely rare.

Once the system has selected its daily sample, it moves to the labeling phase. Here, the researchers replaced the traditional human review process with a multimodal large language model. This AI can look at an image, read the text, and understand the context, just like a human moderator. However, the researchers did not simply let the AI decide; they built a rigorous quality control system around it. Before the AI is allowed to label the daily samples, it is tested against a "gold set" of content that has been carefully reviewed by human experts. The AI must match the human experts' decisions within a very tight margin of error before it is turned on. Once live, the system continues to check the AI's work by sending a random selection of its daily labels back to human experts for verification. This ensures that the AI does not drift or make systematic mistakes over time. The result is a labeling process that is roughly fifteen times faster than human-only review and costs more than ten times less, while maintaining a similar level of accuracy.

The power of this approach lies in its ability to provide a single, unified view of the platform that can be sliced and diced in many different ways. Because the researchers draw one global sample each day, they can immediately answer questions about how safety varies by region, by the type of content surface, or by how old the content is, without needing to run a new study for each question. They can see if a new policy update is working by comparing the numbers before and after, or if a specific type of harmful content is spiking in a particular country. The system also calculates a confidence interval for every number it produces, which tells the safety team how much trust to place in the result. If the number is based on very few examples of a rare violation, the confidence interval will be wide, signaling that the team should wait for more data before making a major decision.

The researchers tested this system extensively, both in simulations and in their actual production environment at Pinterest, where it has been running daily for over a year. They found that by using their ML-assisted sampling method, they could find six to eleven times more violations in their daily sample compared to a standard random sampling method. This efficiency meant they could achieve the same level of statistical precision with far fewer checks, or get much tighter, more reliable numbers with the same amount of work. They also demonstrated that even when the underlying risk scores used to guide the sampling changed or became less accurate over time, the system's final estimates remained unbiased. The errors showed up as wider confidence intervals rather than wrong answers, which is a crucial distinction for decision-makers who need to know when a trend is real and when it is just noise.

One of the most significant findings was how the system handles the inevitable errors that come with automated labeling. The researchers discovered that for very rare violations, the biggest risk is not missing a violation, but falsely flagging safe content as harmful. A single false positive in a sea of safe content can make it look like a problem is ten times worse than it actually is. To manage this, the system constantly monitors the AI's false positive rate. If the rate gets too high, the system can apply a mathematical correction to the final numbers to account for the error, or it can trigger a human review of the specific items causing the spike. This ensures that the daily reports reflect reality rather than the quirks of the AI model. The team also found that day-to-day fluctuations in the numbers were often driven by changes in the types of content being posted or subtle shifts in how the AI interpreted the rules, rather than actual changes in safety levels. By smoothing out these short-term variations and focusing on weekly trends, they could distinguish between normal noise and genuine emerging threats.

This work represents a shift in how digital platforms monitor safety, moving from a reactive model based on user complaints to a proactive, data-driven approach. By treating safety measurement as a statistical science rather than just an enforcement task, the researchers have created a tool that can detect problems the moment they start to appear. The system is designed to be flexible, allowing teams to quickly add new policies or adjust parameters as the landscape of online content changes. It relies on a continuous loop of sampling, AI labeling, human verification, and statistical analysis, creating a feedback loop that is fast enough to keep up with the speed of the internet. The researchers emphasize that this system is not a replacement for human judgment or policy enforcement, but a complementary tool that provides the clarity needed to make those decisions effectively. It turns the invisible into the visible, giving safety teams the eyes they need to understand the true state of their platforms and protect their users with greater precision and speed.

Technical Summary: Measuring the Prevalence of Policy-Violating Content with ML-Assisted Sampling and LLM Labeling

1. Problem Statement

Content safety teams on user-generated content platforms face a critical gap in measurement: relying solely on user reports provides an incomplete view of safety, as many harms are under-reported or rare categories yield insufficient data for statistical monitoring. To address this, teams require prevalence metrics—the fraction of user views (impressions) exposed to policy-violating content on a given day.

Accurate prevalence measurement is historically difficult due to two factors:

Rarity of Violations: Policy violations are often rare events, making simple random sampling inefficient and requiring prohibitively large sample sizes to achieve statistical power.
Cost and Latency of Human Labeling: Traditional human review is too slow and expensive to support daily, platform-representative studies, preventing rapid feedback loops for detecting emerging harms or evaluating interventions.

The goal is to construct a system capable of daily, unbiased, exposure-weighted prevalence estimation that supports granular drill-downs (by surface, geography, content age) without re-sampling, while maintaining statistical rigor and operational feasibility.

2. Methodology

The authors propose a three-stage measurement system combining design-based sampling, ML-assisted weighting, and LLM-driven labeling.

A. ML-Assisted Probability Sampling

To address the rarity of violations, the system employs Probability Proportional to Size (PPS) sampling, augmented by auxiliary risk scores from production safety models.

Sampling Weights: Each content unit $j$ is assigned a weight $w_j$ based on its total impressions ( $C_j$ ) and an auxiliary risk score ( $s_j$ ):
$w_j = C_j^\nu (s_j^\gamma + \epsilon)$
Here, $\nu$ controls exposure weighting, $\gamma$ controls the influence of the risk score, and $\epsilon$ ensures a non-zero probability floor for unscored items.
Implementation: The system uses weighted reservoir sampling (Efraimidis–Spirakis algorithm) to efficiently draw a fixed-size sample ( $m$ ) from billions of daily impressions in a streaming fashion ( $O(m)$ memory).
Unbiasedness: Crucially, the risk score $s_j$ is used only for sampling prioritization, not as a label. The final estimator reweights samples by $1/p_j$ (where $p_j$ is the draw probability) to ensure the prevalence estimate remains design-consistent and unbiased, regardless of the underlying model's threshold evolution.

B. LLM-Assisted Labeling

Sampled items are labeled by a multimodal Large Language Model (LLM) rather than human reviewers for the bulk of the workload.

Input: The LLM receives the content (images/video keyframes, text metadata) and an SME-reviewed policy prompt.
Output: Structured JSON containing a binary violation label, a rationale, and a confidence score.
Governance: To ensure quality, the system employs a gold-set validation workflow. Before launch, the LLM configuration must meet specific decision-quality thresholds (bias and F1 score) relative to human SMEs. Post-launch, a random subsample of daily items is routed to human validation to monitor for drift (policy changes, content shifts, or model updates).

C. Design-Consistent Estimation and Uncertainty Quantification

The system calculates exposure-weighted prevalence ( $\theta$ ) using the Hansen–Hurwitz ratio estimator (for PPS with replacement) or Horvitz–Thompson variants (for without replacement).

Single-Sample Drill-Downs: A key design goal is that a single daily sample supports multiple segments (e.g., geography, surface). By persisting segment-specific impression counts ( $C_{jg}$ ) for sampled items, the system can compute prevalence for any segment without re-sampling.
Uncertainty Quantification: The system reports 95% Confidence Intervals (CIs) using Taylor linearization for the ratio estimator. It also tracks the Kish Effective Sample Size (ESS) to monitor the impact of weight variance on precision.
Label Error Correction: Recognizing that LLMs have false positive rates ( $f$ ) and sensitivity ( $r$ ), the system models the measured prevalence $\theta^{(L)}$ as a function of true prevalence $\theta$ . If error rates are homogeneous, a Rogan–Gladen correction is applied. The system monitors for label error-induced upward fluctuations (common in rare-event regimes) and triggers human adjudication for high-influence anomalies.

3. Key Contributions

The paper details four primary contributions:

A Design-Consistent Estimator: A method for daily prevalence estimation that supports multiple segment drill-downs from a single global probability sample, ensuring unbiasedness even as enforcement models evolve.
Practical Uncertainty Quantification: A comprehensive framework for CIs, effective sample size diagnostics, and a sensitivity analysis of how label errors (specifically false positives) propagate into prevalence estimates, including guidance on when to activate bias corrections.
Hyperparameter Configuration Workflow: A concrete, simulation-backed workflow for tuning sampling parameters ( $\nu, \gamma, \epsilon$ ) to balance positive sample counts against effective sample size for new policies.
Configurable Engineering Workflow: An end-to-end system where policy definitions, prompts, and gold sets are declarative configurations, enabling the rapid onboarding of new policy metrics without bespoke engineering pipelines.

4. Results and Empirical Evidence

The system has been in daily production at Pinterest for over a year across multiple policy areas. Due to confidentiality regarding absolute prevalence levels, results are reported in relative terms and via synthetic simulations.

Sampling Efficiency: In production, ML-assisted sampling (using risk scores) increased the sample positive fraction by 6× to 11.4× compared to impression-only PPS sampling at a fixed label budget. This significantly reduces the label budget required to achieve a target CI width.
Labeling Efficiency: The LLM-assisted workflow reduced end-to-end labeling latency by approximately 15× and operational costs by >10× compared to human-only workflows, while maintaining comparable decision quality.
Statistical Robustness: Simulations demonstrate that design-based reweighting keeps point estimates unbiased even when auxiliary risk scores drift; the primary effect of drift is an increase in variance (wider CIs) rather than bias.
Hyperparameter Sensitivity: Experiments show that increasing the risk score exponent ( $\gamma$ ) improves precision up to a point (e.g., $\gamma=1$ ), but aggressive tilting ( $\gamma \ge 2$ ) collapses the effective sample size, degrading performance.
Minimum Detectable Effects (MDE): The system enables the detection of relative changes as small as 5–6.5% (relative to the mean) on a weekly basis, depending on the policy area and label volume.

5. Significance and Claims

The paper claims that this approach fundamentally shifts the operating point of content safety measurement:

From Infrequent Audits to High-Frequency Feedback: By reducing latency and cost, the system enables daily, platform-representative monitoring, allowing teams to detect emerging patterns, perform rapid root-cause analysis, and evaluate interventions (e.g., model launches, policy updates) in near real-time.
Complementarity to Reporting: The system provides visibility into harms that are under-reported, offering a complementary metric to user reports.
Scalable Governance: The configurable workflow allows safety teams to scale measurement to new policies rapidly, governed by gold-set validation rather than bespoke engineering efforts.
Ethical Safety: The system is designed to minimize human exposure to harmful content by using LLMs for bulk labeling and reserving human review for targeted validation and adjudication of edge cases.

The authors emphasize that while absolute prevalence levels for sensitive policies are confidential, the methodology is fully specified and verified through synthetic simulations, ensuring the approach is independently verifiable and robust to the challenges of rare-event estimation in large-scale systems.