Hierarchical Bayesian Experimental Estimation of Incremental Local Advertising Effects with Secure Aggregation and Differential Privacy

Technical Summary: Hierarchical Bayesian Experimental Estimation of Incremental Local Advertising Effects with Secure Aggregation and Differential Privacy

1. Problem Statement

The paper addresses the challenge of measuring the incremental effectiveness of local commercial advertising in the United States under strict privacy constraints. Traditional measurement models face three primary hurdles:

• Data Sparsity and Bias: Local businesses (e.g., restaurants, retailers) often have low advertising investment, few conversion indicators, and short campaign cycles. Standard "last-click" attribution models fail to distinguish between ad-driven conversions and organic demand, often overstating incremental effects.
• Privacy Restrictions: Increasing regulations limit access to user-level behavioral data, necessitating the use of aggregated, privacy-preserving statistics.
• Statistical Identification: While Secure Aggregation (SA) and Differential Privacy (DP) protect individual data, they introduce noise that weakens statistical identification capabilities, particularly for small-sample campaigns where signal-to-noise ratios are low.

The core problem is to construct a framework that can accurately estimate incremental conversions and Return on Ad Spend (ROAS) using privacy-perturbed regional data without treating the noise as negligible or discarding the privacy protections.

2. Methodology

The authors propose a computational pipeline integrating regional randomized experiments, secure aggregation, differential privacy, and hierarchical Bayesian inference.

A. Experimental Design and Data Collection

• Scope: The study covers 1,860 local businesses across 12 U.S. states, involving 4,720 campaigns, 1.26 billion impressions, and 2.18 million purchase/appointment records.
• Paired-Region Randomization: A zip-code-level paired randomized experiment is employed. Businesses are matched into treatment and control groups based on a 4-dimensional vector: population size, historical sales, industry category, and ad spend.
• Matching Algorithm: The system uses Z-score normalization for continuous variables, KD-tree candidate retrieval (top 5), and Mahalanobis distance for final selection (1:1 matching without substitution). A SHA-256 hash of the campaign ID serves as a random seed for assignment.

B. Privacy-Preserving Data Aggregation

• Secure Aggregation (SA): Local nodes encode impressions, visits, conversions, and ad spend into fixed-point vectors. Using 256-bit keys, nodes generate paired masks that cancel out during summation, preventing the server from accessing individual shard data.
• Differential Privacy (DP): Post-aggregation, a Gaussian mechanism adds noise to the regional weekly statistics. The system operates with a privacy budget of $\epsilon = 2$ and $\delta = 10^{-6}$. The noise variance is explicitly preserved and passed to the inference model.

C. Hierarchical Bayesian Inference Model

• Likelihood Construction: The model treats the observed noisy count ($\tilde{Y}_{ipt}$) as a realization of a latent true count ($Y_{ipt}$) perturbed by Gaussian noise.
  • Observation Layer: $\tilde{Y}_{ipt} \sim N(Y_{ipt}, \sigma_{DP}^2)$
  • Count Layer: $Y_{ipt} \sim \text{NegativeBinomial}(\mu_{ipt}, \phi)$
• Hierarchical Structure: The latent mean ($\mu_{ipt}$) follows a 4-level random-effects structure modeling:
  1. Firm (Business level)
  2. Industry
  3. Metropolitan Area
  4. Ad Placement Week
• Inference: The model uses the recorded privacy-noise variance ($\sigma_{DP}^2$) within the likelihood function, allowing the inference to account for the perturbation rather than treating perturbed counts as noise-free observations. Parameters are estimated using NUTS sampling (4 chains, 4,000 iterations).

D. Decision Framework

• Counterfactual Generation: Posterior samples are used to generate counterfactual conversion volumes by setting the treatment variable to 1 (advertising) and 0 (control).
• Budget Optimization: Decisions (Increase, Maintain, Reduce, Stop) are made by minimizing a posterior risk function based on the probability of achieving return targets and the 5th, 50th, and 95th percentiles of the posterior distribution.

3. Key Results

The proposed framework was evaluated against a standard Difference-in-Differences (DiD) baseline and a non-hierarchical model.

• Estimation Accuracy:
  • The Mean Absolute Error (MAE) for incremental conversions decreased from 71.4 (standard DiD) to 43.6 (Hierarchical Bayesian), a 38.9% reduction.
  • The absolute error for incremental ROAS decreased from 0.39 to 0.17, a 56.4% reduction.
• Interval Calibration: The 90% posterior confidence interval achieved 91.2% coverage, slightly exceeding the nominal 90% level.
• Impact of Differential Privacy:
  • Introducing DP noise ($\epsilon=2$) added an additional error of 6.3 conversions compared to a non-noise scenario (MAE rose from 37.3 to 43.6).
  • Despite this, the error remained significantly lower than the standard DiD baseline.
  • Decision Stability: 83% of companies kept their original budget retention or adjustment decisions, while 17% changed their actions, indicating the noise correction in the likelihood function effectively mitigated decision volatility.
• Small-Sample Performance:
  • For campaigns with fewer than 500 conversions, hierarchical information sharing reduced the Minimum Detectable Effect (MDE) from 8.6% to 5.9% (a 31.4% relative reduction).
  • The median posterior standard deviation decreased from 0.084 to 0.061, and posterior variance decreased by approximately 47.3%.

4. Significance and Claims

The paper claims to provide a statistically rigorous, privacy-preserving pipeline for local advertising measurement that overcomes the limitations of traditional attribution models.

• Statistical Innovation: The primary contribution is the construction of a hierarchical Bayesian negative binomial model that explicitly incorporates privacy-noise variance into the likelihood. This allows for the inference of incremental effects without discarding privacy protections or treating noisy data as ground truth.
• Practical Utility: The framework enables stable budget optimization decisions even in low-conversion, small-sample scenarios where traditional methods fail due to high variance. By leveraging multi-level parameter sharing (firm, industry, metro, week), the model effectively borrows strength across the hierarchy to reduce estimation variance.
• Privacy-Utility Trade-off: The study demonstrates that with appropriate noise correction and hierarchical modeling, the utility loss from Differential Privacy (an additional 6.3 conversion error) is manageable and does not significantly alter the majority of business decisions (83% stability).

The authors conclude that this approach offers an interpretable statistical alternative to user-level tracking, balancing data availability, statistical reliability, and individual privacy. They acknowledge limitations regarding external market changes, cross-regional spillovers, and industry heterogeneity, suggesting future work could incorporate dynamic time-varying causal structures.