🤖 machine learning

From Prediction to Incrementality: Causal Optimization for Large-Scale Targeting and Recommendation

This paper presents a decision-centric framework that replaces traditional predictive targeting with causal optimization for large-scale recommendation systems, integrating causal neural networks, Bayesian exploration, and constrained allocation to achieve a statistically significant 7.20% lift in long-term value during an online A/B test on LinkedIn.

Original authors: Changshuai Wei, John Bencina, Phuc Nguyen, Andre Assuncao Silva T Ribeiro, Benjamin Zelditch

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Changshuai Wei, John Bencina, Phuc Nguyen, Andre Assuncao Silva T Ribeiro, Benjamin Zelditch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive cruise ship, and your job is to hand out free ice cream to the passengers. In the old way of doing things, you'd look at a list of who usually eats ice cream and hand it out to them. But here's the catch: some of those people were going to eat ice cream anyway, even if you didn't give it to them. You wasted your scoops on people who didn't need a nudge, while the passengers who were on the fence about getting a treat got nothing. This is the problem of "prediction" versus "incrementality." Prediction asks, "Who will eat ice cream?" while incrementality asks, "Who will eat ice cream only because I gave it to them?"

This paper dives into the world of Causal Machine Learning, a branch of artificial intelligence that tries to figure out cause-and-effect rather than just spotting patterns. It's like moving from a weather forecaster who just says "it usually rains on Tuesdays" to a scientist who can prove that "if we release a cloud-seeding rocket, it will actually make it rain." The authors are working on a system used by huge online platforms (like LinkedIn) to decide who gets to see marketing messages, job ads, or notifications. The big question is: how do we stop wasting money on people who would have clicked the ad anyway, and instead focus our limited budget on the people who truly need that extra push to take action?

The researchers at LinkedIn built a new, smarter system to solve this puzzle. They call it a "decision-centric framework," which is a fancy way of saying they stopped just guessing who would click and started mathematically calculating who would be changed by the message. Their system has three main parts working together like a dream team. First, a "causal neural network" (powered by a Transformer, the same kind of brain behind many modern AI tools) acts as a detective. It looks at a user's history and estimates the "treatment effect"—basically, how much more likely they are to buy something or apply for a job specifically because they saw the ad.

Second, the system includes a "neural bandit" layer. Think of this as a curious explorer. In the past, AI systems often got stuck in a loop, only showing ads to people they were already sure would click, missing out on new opportunities. This explorer part of the system is willing to take small risks to try new things, ensuring it doesn't miss out on potential customers who haven't been tested yet. It uses a clever math trick called "Laplace approximation" to guess how uncertain it is about its predictions, allowing it to explore safely.

Third, and perhaps most importantly, there is a "large-scale linear programming" layer. This is the strict but fair manager. Even if the detective finds a perfect candidate and the explorer wants to try a new strategy, the manager has to make sure they don't break the rules. The company has a limited budget, a limit on how many times they can email a person, and a need to reach different types of people fairly. This layer solves a giant math puzzle to distribute the ads perfectly within those limits, ensuring no money is wasted and no one is ignored unfairly.

To test if this new system actually works, the team ran simulations and then a real-world experiment on LinkedIn's marketing traffic. In the simulations, they compared their new "causal" method against older methods that just predicted who would click. The results were clear: the new system found more valuable targets and spent less money to get the same results. But the real magic happened in the live test. When they let the new system run for eight weeks against the old standard system, it delivered a statistically significant +7.20% lift in the primary long-term value metric. This means the company got noticeably more value from their marketing budget without spending more money.

The paper also highlights some practical lessons learned from building this in the real world. They found that simply having a smart model isn't enough; you need to be incredibly careful about how you prepare your data and control how the ads are actually delivered. For instance, if the new system decides not to show an ad to someone (because it thinks they wouldn't change their mind), but the old system did show one, you have to make sure you aren't comparing apples to oranges. They built special tools to ensure the test groups were perfectly matched so the results were fair.

In short, this paper shows that by shifting focus from "who will act?" to "who will act because of us?", and by combining smart AI with strict budget management, companies can make their marketing much more effective. It's not just about shouting louder at everyone; it's about whispering the right message to the right person at the right time, saving resources and creating more value for everyone involved.

Technical Summary: From Prediction to Incrementality: Causal Optimization for Large-Scale Targeting and Recommendation

1. Problem Statement

Large-scale targeting and recommendation systems in modern online platforms (e.g., advertising, marketing, notifications) traditionally rely on predictive modeling. These systems train models to estimate user response probabilities (e.g., click-through or conversion likelihood) and rank items based on these predictions.

However, the authors argue that this paradigm is fundamentally misaligned with business goals in marketing and incentive-driven settings where the objective is incremental impact (e.g., "Would this user convert because of the intervention, or would they have converted anyway?").

The Misalignment: High predicted response does not necessarily equate to high causal value. Users who would act regardless of an intervention (the "always-takers") are systematically over-targeted, wasting resources.
The Gap: Existing causal approaches often focus on local decision-making (ranking) or policy evaluation without addressing global constraints (budgets, frequency caps, fairness). Conversely, constrained optimization systems typically use non-causal predictive scores as inputs, leading to suboptimal resource allocation.

2. Methodology

The paper proposes a decision-centric framework that jointly optimizes causal effects under global constraints. The system integrates three core components under a single objective:

A. Causal Neural Network (The "DragonBandit" Head)

The system estimates Individual Treatment Effects (ITE) using a modified DragonNet architecture:

Architecture: A shared encoder $\Phi$ feeds into three heads: two outcome heads ( $\mu_1, \mu_0$ ) for potential outcomes under treatment and control, and one propensity head ( $e$ ) for treatment assignment probability.
Transformer Backbone: To handle sequential user interactions (marketing touchpoints), the model incorporates a Transformer encoder. This processes temporal features (recency, engagement cadence) using learned embeddings for interaction types, day-of-week, and sinusoidal positional encodings.
Multi-Outcome Extension: To support multiple products without retraining, the model uses outcome embeddings. A shared embedding matrix encodes product attributes (structured or LLM-derived), allowing the model to score new, unseen products via dot-product similarity with the user's state representation.
Training Objective: The model minimizes a composite loss including outcome prediction (BCE), propensity prediction, targeted regularization based on the Efficient Influence Function (EIF) for ATE, and an autoencoder reconstruction loss for outcome embeddings.

B. Uncertainty-Aware Exploration (Neural Bandit)

To address feedback loops and improve causal identification (overlap), the system employs Neural Thompson Sampling:

Mechanism: Instead of using deterministic point estimates, the system samples from the posterior distribution of the network parameters.
Approximation: It uses a Linearized Laplace Approximation (LLA) centered at the Maximum A Posteriori (MAP) solution. This computes the curvature of the log-likelihood to derive a Gaussian distribution over logits, enabling uncertainty-aware exploration with minimal serving overhead compared to methods like Bayes by Backprop.
Role: This ensures that the system explores actions with high uncertainty, preventing the "rich-get-richer" dynamic and improving the quality of data for future causal estimates.

C. Constrained Allocation (Large-Scale Linear Programming)

The sampled incremental scores are fed into a dual-based large-scale linear programming (LP) solver to coordinate decisions across millions of users and items:

Formulation: The problem maximizes the sum of sampled incremental rewards subject to global constraints (budgets, volume caps, frequency limits).
Scalability: To handle tens of millions of variables, the authors use Dual Decomposition. They relax binary constraints to probabilities and solve the dual problem using Nesterov-accelerated ascent. The primal minimizer decomposes per user, making the per-iteration cost linear in the number of users and items ( $O(|U| \cdot |I|)$ ), rather than cubic.
Delivery Control: A feedback controller adjusts cost targets based on realized delivery rates (e.g., auction wins, user return rates) to ensure the system meets pacing requirements without over- or under-delivering.

3. Key Contributions

Unified Decision Framework: A formulation that aligns causal estimation, exploration, and constrained allocation under a single objective, moving beyond the standard pattern of optimizing predictive signals with heuristic allocation.
Architectural Extensions:
- A Transformer-augmented DragonNet for modeling temporal marketing sequences.
- A shared outcome-embedding head enabling multi-outcome and attribute-conditioned scoring for cold-start products.
Production Pipeline: A fully integrated pipeline combining the causal head, neural-bandit exploration, and a dual-based LP solver.
Empirical Validation:
- Offline: Simulations on the Open Bandit Dataset (OBD) showing that incremental modeling + constrained optimization outperforms propensity-based and ranking-only baselines.
- Online: An eight-week A/B test on LinkedIn Feed marketing traffic.

4. Results

Offline Simulations:
- Single-turn: The "Incremental Modeling + Constrained Optimization" approach achieved the highest net return ($0.46 vs. $0.39 for propensity + constrained) and significantly reduced costs by identifying users likely to convert organically (increasing "no-recommendation" rates).
- Multi-turn: The "Bandit Incremental Model" overcame initial data biases and outperformed greedy variants after ~50 rounds, demonstrating the long-term value of exploration.
Online A/B Test:
- Deployed on LinkedIn Feed marketing traffic against a Business-As-Usual (BAU) two-tier propensity stack.
- Primary Metric: The treatment arm achieved a statistically significant +7.20% lift in the primary long-term-value metric ( $p=0.041$ , 95% CI: [0.31%, 14.09%]).
- Operational Success: The system successfully managed budget pacing and delivery constraints, avoiding the "under-delivery" pitfalls observed in early tests without the feedback controller.

5. Significance and Claims

The paper claims to demonstrate the feasibility of production-scale causal optimization under complex business constraints. Its significance lies in:

Shifting the Paradigm: Moving from "ranking high-response users" to "choosing users with high incremental impact" via globally coordinated allocation.
Practical Deployment: Providing a blueprint for integrating causal inference into industrial systems, addressing critical real-world hurdles such as training data construction (context/treatment/outcome windows), cost control, and delivery estimation.
Scalability: Proving that complex causal models combined with constrained optimization can run efficiently at the scale of hundreds of millions of members using dual decomposition and lightweight uncertainty approximations.

The authors emphasize that the success of the system relies not just on the model architecture, but on the pragmatic enhancements in data construction and delivery control, which are often invisible in offline simulations but critical for real-world ROI.