Sensitivity and Early Detection of Bayesian Causal Impact Models for Marketing Interventions

Technical Summary: Sensitivity and Early Detection of Bayesian Causal Impact Models for Marketing Interventions

Problem Statement
In dynamic e-commerce environments, marketing teams frequently implement structural changes to automated journeys (e.g., abandoned cart flows), such as software upgrades or logic modifications. Unlike isolated A/B tests, these system-level interventions affect the entire user population, making controlled experimentation impractical. While Bayesian Causal Impact models are standard for retrospective effect estimation, they offer limited guidance on early detection and sensitivity. Decision-makers require answers to operational questions: How many days of post-change data are needed to confidently conclude that performance has deteriorated? How early can a negative impact be detected to enable timely rollbacks or adjustments? Current methods often fail to quantify the trade-offs between statistical conservatism, detection speed, and the magnitude of performance degradation.

Methodology
The paper proposes a simulation-based framework to evaluate the sensitivity and detection capabilities of Bayesian Causal Impact models under controlled performance degradations. The methodology utilizes daily traffic data from an abandoned cart marketing journey at Despegar, where the outcome variable is the daily volume of automated messages sent.

1. Data Perturbation and Subsampling: To assess stability against data variability, the authors employ a repeated subsampling strategy. For each iteration, the original time series is rescaled by a factor ($\alpha = 0.7$) and augmented with stochastic noise proportional to the series magnitude. This generates multiple plausible realizations of the data while preserving temporal structure.
2. Synthetic Impact Injection: Within the validation period, observed outcomes are artificially reduced by multiplicative factors ($r \in \{1.0, 0.9, 0.8, 0.7, 0.5\}$) to simulate varying magnitudes of efficiency loss.
3. Modeling: A Bayesian Structural Time Series (BSTS) model is fitted on a training period (5 months) to generate counterfactual predictions for the validation period (1 month). The model leverages covariates such as user entry into abandoned cart flows and product search volume.
4. Alarm Criteria: Two distinct detection criteria are evaluated across varying confidence levels ($c \in \{60\%, 70\%, 80\%, 90\%\}$) and detection horizons ($h \in \{5, 7, 10, 15\}$ days):
   • Proportion-Based: An alarm triggers if the proportion of days where the observed value falls below the predictive lower bound exceeds a threshold ($\tau = 0.4$).
   • Persistence-Based: An alarm triggers if the cumulative impact remains negative for three consecutive days.
5. Evaluation: The probability of triggering an alarm is computed across 100 simulation runs for each configuration, quantifying the likelihood of detecting a specific degradation magnitude within a given timeframe.

Key Results
The study reveals that detection performance is highly dependent on the interaction between effect size, confidence level, and evaluation horizon.

• Limitations of Proportion-Based Criteria: The proportion-based rule (requiring >40% of observations to be below the bound) exhibits a structural bias as the horizon increases. For longer horizons (e.g., 10 or 15 days), the criterion becomes increasingly difficult to satisfy, requiring very strong reductions or low confidence levels to trigger an alarm. This limits its utility for early detection of moderate degradations.
• Advantages of Persistence-Based Criteria: The alternative criterion, based on three consecutive days of negative cumulative impact, demonstrates more stable and operationally meaningful behavior. By focusing on temporal persistence rather than aggregate proportions, it avoids dilution effects in longer horizons and responds more quickly to sustained negative trends.
• Optimal Configuration: The results suggest a balanced operational configuration of a 10-day detection horizon combined with an 80% confidence level. This setup offers a reasonable trade-off between robustness and sensitivity, minimizing false alarms while enabling the detection of relevant performance deteriorations.
• Confidence Level Trade-offs: Lower confidence levels (e.g., 60%) increase sensitivity but significantly raise the risk of false positives, even under mild reductions.

Significance and Contributions
The paper claims to bridge the gap between retrospective causal estimation and the operational needs of marketing teams. Its primary contributions are:

1. Operational Framework: It extends causal impact analysis beyond simple effect estimation to provide a methodology for quantifying detection sensitivity and supporting monitoring decisions in dynamic environments.
2. Criterion Evaluation: It identifies that standard proportion-based monitoring rules may be suboptimal for early detection in marketing systems, proposing persistence-based criteria as a more stable alternative.
3. Decision Support: The framework provides a principled mechanism for teams to determine how much data is required to reliably detect a specific level of performance degradation, enabling better balancing of uncertainty, timeliness, and business risk.

Future Directions
The authors modestly note that the current study relies on a fixed validation window. They suggest that future research should evaluate the temporal stability of these alarm criteria by incorporating rolling or shifting validation windows. This would assess whether alarm signals remain consistent across different temporal realizations of the data, distinguishing persistent deterioration from window-specific fluctuations.