Technical Summary: Estimating the Effect of Timing on Coupon Effectiveness

Problem Statement

Coupon incentives are a staple of marketing strategies, yet businesses often struggle to maximize their return on investment (ROI). While previous research has focused on optimizing the value of coupons (e.g., discount percentage) or the segmentation of users (e.g., uplift modeling), the impact of timing remains under-explored in many contexts.

The core challenge is that verifying the hypothesis that "coupons are more effective when delivered at critical moments of user engagement" typically requires a real-time, event-triggered distribution system. Implementing such a system involves significant engineering effort and cost. Consequently, companies face a "catch-22": they need evidence of the benefits to justify building the infrastructure, but they cannot generate that evidence without the infrastructure to run a dedicated Randomized Control Trial (RCT) comparing immediate vs. delayed delivery.

This paper addresses the problem of estimating the causal effect of coupon timing on conversion rates without requiring a dedicated AB test or a pre-existing real-time distribution service.

Methodology

The authors propose a framework that applies causal inference to pre-existing "natural randomized control trial" data. The approach leverages data from standard batch-based campaigns where users are randomly assigned to control and treatment groups, but the exact timing of coupon delivery is determined by the user's registration or activity time relative to the batch processing schedule.

1. Causal Modeling and Identification

The framework utilizes Directed Acyclic Graphs (DAGs) to identify confounding variables and determine the correct conditioning set for unbiased estimation.

• User Onboarding Case: The authors model the relationship between user attributes ($X$), treatment ($T$), a pre-treatment purchase event ($P$), and the outcome ($Y$, future purchase). They identify a critical pitfall: conditioning on $P$ (users who hadn't purchased yet) blocks the forward path between the timing covariate and the outcome, leading to biased interaction estimates. To estimate the interaction between timing and treatment ($P(Y|do(T), do(M))$), the model must not condition on $P$ when analyzing the interaction, or must carefully account for the selection bias introduced by the batch process.
• User Retention Case: A simpler causal model is applied to a public dataset (Lenta), where the goal is to estimate the interaction between the time elapsed since the last purchase and the treatment effect.

2. Uplift Modeling with Interaction Terms

Once the causal structure is defined, the authors employ Uplift Modeling to estimate the Conditional Average Treatment Effect (CATE).

• Model Architecture: Instead of complex algorithms like neural networks or decision trees, the authors utilize a logistic regression model with interaction terms. This choice prioritizes interpretability.
• Formulation: The model predicts the log-odds of a future purchase ($Y$) as a function of baseline covariates ($X$) and an uplift component specific to the treatment group ($S$). Crucially, interaction terms are created by multiplying the treatment indicator with time-related covariates (e.g., minute of registration, time since last purchase).
  $$\ln\left(\frac{y_i}{1-y_i}\right) = \sum_{j \in C} baseline_j x_{ij} + s_i \sum_{j \in C} uplift_j x_{ij} + \epsilon_i$$
• Feature Engineering: Time-related features (hour, minute, recency) are standardized. In the onboarding case, the authors note that standardizing the minute of registration avoids issues with cyclical discontinuities (e.g., midnight) because coupons are not distributed at night, and delays are consistent within the hour.

3. Experimental Setup

• Internal Data (Mercari): An analysis of a user onboarding campaign where users were randomly assigned to control/treatment groups via hourly batch jobs. The "natural experiment" aspect arises because the delay between registration and coupon receipt varies based on the minute of registration, independent of the marketer's control.
• Public Data (Lenta): A user retention campaign dataset from a Russian hypermarket chain, used to validate the framework's generalizability and reproducibility.

Key Results

User Onboarding (Mercari)

• Statistical Significance: The uplift coefficient for the minute of registration was found to be positive and statistically significant ($p \approx 0.05$). This indicates that users who registered closer to the top of the hour (receiving coupons sooner) had a higher incremental conversion rate compared to those who registered later in the hour.
• Interpretation: The time elapsed between account creation and coupon delivery significantly impacts the effectiveness of the coupon.
• Counterfactual Analysis: Visualizations of the model's predictions show that while baseline conversion rates vary slightly by registration time, the treatment effect (uplift) is clearly higher for users receiving coupons immediately. This provides the necessary evidence to justify the engineering investment in a real-time distribution system.

User Retention (Lenta)

• Baseline Effect: The baseline coefficient for "latest purchase time" was strongly positive, confirming that users with recent purchases are more likely to convert (the "zone" hypothesis).
• Timing Interaction: The uplift coefficient for the time elapsed since the last purchase was weakly positive but not statistically significant.
• Conclusion: Unlike the onboarding scenario, delivering coupons immediately after a purchase (vs. a delayed batch) does not yield a considerable business impact for retention in this specific dataset.

Contributions and Significance

The paper makes the following specific contributions:

1. Highlighting Timing: It establishes that the timing of coupon distribution is a critical, often overlooked variable in marketing effectiveness.
2. Methodological Framework: It proposes a framework to estimate the causal effect of timing using natural RCTs and causal inference (specifically do-calculus and uplift modeling). This allows companies to validate the need for expensive real-time infrastructure without first building it.
3. Empirical Validation:
   • It demonstrates the framework's success in an internal onboarding campaign, providing data-driven justification for implementing real-time event-triggered coupons.
   • It validates the framework's generalizability using a public dataset for user retention, showing that the method can distinguish between scenarios where timing matters (onboarding) and where it may not (retention in this specific context).
4. Reproducibility: By applying the method to a public dataset and releasing source code, the authors aim to make the research reproducible and accessible to other practitioners.

Limitations

The authors acknowledge that the validity of their unbiased estimations relies entirely on the correctness of their causal diagrams.

• Unobservable Confounders: In the onboarding case, they assume no unobservable confounders (e.g., income, hobbies) exist that affect both registration time and conversion. They argue that controlling for the hour of registration mitigates this risk.
• Data Specificity: The retention results are specific to the Lenta dataset; the lack of significant timing effects there does not universally negate the importance of timing in all retention contexts, but rather suggests the framework can identify where timing is not a primary lever.

Conclusion

The paper concludes that earlier is generally better for coupon delivery, but the magnitude of the benefit depends on the user journey stage. For onboarding, the delay caused by batch processing significantly reduces coupon efficacy. For retention, the immediate post-purchase window may not offer the same marginal gain. The proposed framework offers a cost-effective, data-driven path for businesses to determine whether investing in real-time distribution infrastructure is warranted for their specific use cases.