🤖 machine learning

Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning

This paper presents a unified, model-agnostic framework for optimizing long-term user engagement in large-scale recommendation systems by identifying early predictive session behaviors and deriving downstream reward signals, which have been successfully deployed across multiple Pinterest surfaces to improve retention metrics.

Original authors: Dingsu Wang, Filip Ryzner, Kelly He, Armando Ordorica, David Woo, Aditya Mantha, Liyao Lu, Usha Amrutha Nookala, Haoran Guo, Jiacong He, Olafur Gudmundsson, Matt Chun, Krystal Benitez, Dhruvil Deven B

Published 2026-07-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Dingsu Wang, Filip Ryzner, Kelly He, Armando Ordorica, David Woo, Aditya Mantha, Liyao Lu, Usha Amrutha Nookala, Haoran Guo, Jiacong He, Olafur Gudmundsson, Matt Chun, Krystal Benitez, Dhruvil Deven Badani, Yijie Dylan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're walking through a massive, endless library where the shelves rearrange themselves based on what you look at. This is the world of recommender systems, the invisible engines behind apps like Pinterest, TikTok, and Netflix. Their job is to guess what you'll like next. For a long time, these systems were like impatient tour guides: they only cared if you stopped to look at one exhibit right now. If you clicked a picture, they'd show you ten more just like it immediately. But here's the problem: if you only show people exactly what they already know, they get bored and stop visiting the library altogether. The big question scientists are trying to solve is: How do we recommend things that keep people coming back for months or years, not just for the next five minutes?

To answer this, researchers have to deal with a tricky reality: we can't easily see the future. We can't wait to see if a user returns in a month to decide what to show them today. So, instead of guessing the future directly, they look for "clues" in the present—small behaviors that usually happen right before someone decides to stick around. This paper is about finding those clues and teaching the library's tour guide to follow them, turning a short, boring visit into a long, exciting adventure.

The Problem: The "Click-Trap"

Imagine you're at a carnival. A game booth offers you a free, instant candy if you spin a wheel. You spin, get your candy, and leave. The booth owner is happy because you spun the wheel, but you never come back. This is what happens when recommendation systems only care about short-term signals like a quick click or a one-second glance. They get a lot of "spins," but the users get bored and leave the carnival forever.

The authors of this paper, working at Pinterest, realized that optimizing for immediate clicks was actually hurting the long-term health of their platform. They wanted to build a system that didn't just say, "Here is something you might click," but rather, "Here is something that will make you stay, explore, and come back tomorrow."

The Solution: The "Rabbit Hole" Detective

The team came up with a clever idea: instead of waiting to see if a user returns next week (which is rare and hard to track), they would look for Downstream Rewards. Think of this like a detective looking for footprints. If a user takes a deep dive into a "rabbit hole" of related content, saves a picture, or spends a long time exploring, those are strong footprints that say, "This person is having a great time and will likely return."

The paper proposes a model-agnostic framework. "Model-agnostic" is a fancy way of saying "it works with any kind of recommendation engine," whether it's a simple list or a complex AI. They didn't just guess which behaviors were good; they ran a massive offline screening process to find the specific actions that actually predicted long-term loyalty.

The Three Golden Clues

Through their analysis, they identified three main types of "rewards" (clues) that lead to happy, returning users:

The Deep Dive (Deeper Session Engagement):
It's not just about clicking; it's about how deep you go. The paper found that if a user clicks on a Pin and then immediately dives into a "P2P rabbit hole" (a chain of related Pins), saves it, or downloads it, that's a huge win.
- The Metaphor: It's the difference between peeking into a room and actually moving in. The system learned to reward users who take the time to explore the "rabbit hole" rather than just skimming the surface.
- The Result: They found that these deep actions are strongly linked to users returning. In fact, sessions with these deep actions led to users revisiting the app significantly sooner.
The "Fake It" Penalty (Negative Rewards):
Not all clicks are good. The team discovered that some "closeups" (zooming in on an image) are actually signs of boredom. If a user zooms in for less than a second and then immediately leaves, it's a "shallow closeup."
- The Metaphor: Imagine someone walking into a store, grabbing an item, looking at it for a split second, and dropping it immediately. They aren't interested; they're just killing time.
- The Fix: The system now treats these split-second glances as a negative reward. It's like the tour guide saying, "Hey, don't show this person this item again; they clearly didn't like it." They even found that different types of users need different rules: "core" users need a 1-second threshold to be considered shallow, while "non-core" users need only 0.5 seconds.
The New Adventure (Use Case Adoption):
Sometimes, users get stuck in a loop of their old interests. The paper introduced a reward for Use Case Adoption. This happens when a user engages with something totally new that doesn't fit their usual pattern.
- The Metaphor: If a user usually looks at "cat videos," and the system shows them a "how-to-fix-a-bike" video, and they actually watch and save it, that's a win. It means the system successfully expanded their horizons.
- The Goal: This encourages the AI to show users things they didn't know they wanted, helping them build new hobbies and interests, which keeps them on the platform longer.

The Engine Room: How They Made It Work

You might wonder, "How do you calculate all this without slowing down the app?" The authors built a new infrastructure called DRv2.

The Old Way: They used to pre-calculate all these rewards in huge batches, which took weeks to update. If they wanted to test a new idea, they had to wait weeks.
The New Way: They built a system that calculates these rewards on the fly, right when the data is being read. This cut the time to test new ideas from 3 weeks down to 2 days. It's like switching from baking a cake from scratch every time you want a slice to having a magic machine that instantly assembles a fresh slice whenever you ask.

The Results: Does It Actually Work?

The team tested these ideas in the real world using A/B testing (showing one group the old system and another group the new system). The results were consistent and positive:

More Successful Sessions: The number of sessions where users did something meaningful (like saving or downloading) went up by +0.36% across the board.
More Time Spent: Users spent more time in the app, with total time increasing by +0.10%.
Better Retention: Most importantly, the number of Weekly Active Users (WAU) increased by +0.1% for core users. This proves that the system didn't just trick people into clicking; it actually made them want to come back.
Cross-Surface Success: They didn't just fix the "Homefeed" (the main page). They applied the same logic to Search, Related Pins, and Notifications. In Search, the "fulfillment rate" (how often a search led to a real action) went up by +0.25%.

The Takeaway

This paper suggests that the secret to keeping users happy isn't just showing them what they already like, but guiding them into deeper, more meaningful interactions. By rewarding "deep dives," penalizing "fake clicks," and encouraging "new adventures," the system can turn a quick glance into a long-term relationship.

The authors are careful to note that this isn't a magic bullet that solves everything forever, but it is a powerful, flexible tool that works across different parts of the app. By focusing on the quality of the journey rather than just the speed of the click, they managed to build a recommendation system that feels less like a vending machine and more like a helpful friend who knows how to keep the conversation going.

Technical Summary: Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning

1. Problem Definition

The paper addresses the challenge of optimizing long-term user engagement and retention in large-scale recommendation systems, specifically within the context of Pinterest. While modern systems effectively optimize for short-term behavioral signals (e.g., immediate clicks), optimizing for long-term retention is difficult due to three primary factors:

Sparsity and Noise: Retention labels (e.g., whether a user returns in a week) are sparse, noisy, and only weakly observable.
Delay and Attribution: Retention signals are delayed, making it challenging to attribute future outcomes to specific earlier recommendations.
Distribution Shift: User interests and data distributions evolve over time, causing fixed surrogate objectives to become stale and misaligned with behaviors that drive long-term value.

Existing approaches using Reinforcement Learning (RL) or sequential modeling often require task-specific reward engineering, substantial computational overhead, and are difficult to generalize across different product surfaces (e.g., Homefeed, Search, Related Pins).

2. Methodology

The authors propose a unified, model-agnostic downstream reward (DR) framework. Instead of directly optimizing for delayed retention labels, the system learns to optimize for "downstream rewards"—proxy signals derived from observed user action patterns that are predictive of future retention.

2.1 Offline Screening Framework

Before defining rewards, the authors developed an offline screening framework to identify session-level behaviors that serve as valid proxies. Using a pivot-day analysis on low-engagement users, they screened hundreds of candidate features against three criteria:

Correlation: Must correlate with future sustained engagement (not just immediate activity).
Observability: Must be observable within or shortly after the session.
Incremental Predictivity: Must remain predictive after controlling for correlated signals.

Key Findings from Screening:

Depth over Breadth: Deep engagement (e.g., saves, long closeups, P2P rabbit hole exploration) is positively correlated with retention, whereas shallow, high-volume browsing is often misleading once exposure volume is controlled for.
P2P Exploration: Density of exploration within the "Pin-to-Pin" (P2P) rabbit hole is a strong predictor of faster return.
Session Length: Longer sessions are empirically linked to faster revisits.

2.2 Downstream Reward Designs

Based on the screening insights, the authors define three complementary reward families:

Rewards for Deeper Session Engagement:
- Calculates a discounted cumulative sum of high-intent actions (saves, downloads, screenshots) occurring downstream of a recommendation within a session.
- Formula: $R_{eng}(S_{u,t}) = \sum \gamma^{i-1} r_{eng}$ , where $\gamma$ is a discount factor favoring events closer to the initial recommendation.
Negative Rewards (Shallow Closeups):
- Explicitly penalizes "shallow" interactions that indicate poor user experience.
- Defines a "shallow closeup" as a closeup with a dwell time below a user-state-specific threshold (e.g., <1s for core users, <0.5s for non-core) that is not followed by any high-intent downstream action.
- This signal is treated as a negative reward to discourage the model from surfacing content that leads to immediate abandonment.
Rewards for Use Case Adoption:
- Encourages the model to surface content that expands a user's interest space beyond their established clusters.
- A reward is triggered if a user engages with an item whose embedding is dissimilar to all their precomputed interest clusters (similarity < $\eta$ ).
- This acts as a proxy for expanding user intent while maintaining alignment with engagement goals.

2.3 Infrastructure and Engineering

To address the combinatorial complexity of reward definitions and the need for rapid iteration, the authors built a DRv2 infrastructure:

DRv1 (Legacy): Pre-aggregated tables requiring full recomputation via Spark self-joins for any reward definition change (weeks to iterate).
DRv2 (Proposed): Stores user engagement as reusable daily sequences. Reward labels are computed on-demand within a Ray-based dataloader using configurable User-Defined Functions (UDFs).
Impact: This reduced the time-to-experiment from ~3 weeks to ~2 days and kept incremental training costs within 5% of the baseline.

2.4 Integration with Ranking Models

The framework is model-agnostic and can be applied to any supervised or RL-based ranking model. In production, downstream rewards are integrated as auxiliary heads in multi-task learning models. The final score is a linear combination of immediate engagement predictions and downstream reward predictions, with weights tuned via HyperOPT to balance immediate experience with long-term retention.

3. Experimental Results

The authors conducted extensive online A/B testing on Pinterest's Homefeed, Search, Related Pins, and Notifications.

3.1 Homefeed Results

Deeper Session Rewards: Led to a +0.36% increase in Successful Sessions (SS) and +0.10% increase in Total Time Spent. Weekly Active Users (WAU) increased by +0.1%.
Negative Rewards: After optimizing thresholds by user state (core vs. non-core), the model achieved a +0.16% increase in SS, a -0.40% decrease in unsuccessful sessions, and a +0.35% increase in total time spent. Crucially, this configuration maintained positive retention trends for both core and non-core users, unlike a uniform threshold approach which harmed non-core retention.
Use Case Adoption: Improved Homefeed saves by +0.42% and increased the proportion of users engaging with multiple use cases by +0.18%.

3.2 Cross-Surface Scaling

The framework was successfully extended to Search, Related Pins, and Notifications:

Search: +0.25% increase in Search Fulfillment Rate.
Related Pins: +0.15% increase in Successful Sessions.
Notifications: +0.14% increase in Successful Sessions and +0.11% WAU gain.
P2P Specifics: Deploying a specific shallow-closeup penalty on the P2P ranker (with a 2s threshold) resulted in +0.14% SS and +0.70% increase in sessions longer than 5 minutes.

4. Key Contributions

Insights for User Retention Modeling: An offline evaluation framework that identifies session-level signals (specifically deep P2P exploration and saves) as valid proxies for long-term retention, filtering out misleading shallow signals.
Model-agnostic Downstream Rewards: A unified framework that generates complementary reward signals (positive, negative, and use-case adoption) applicable across different deep learning models and product surfaces.
Reward Label Generation Infrastructure: A scalable, Ray-based infrastructure (DRv2) that enables rapid iteration on reward definitions without heavy engineering overhead.
Empirical Evaluations: Validation through long-term online A/B testing demonstrating consistent improvements in engagement and retention metrics across multiple Pinterest surfaces.

5. Significance and Claims

The paper claims that this framework provides a practical, scalable solution for optimizing long-term user value in industrial recommendation systems without the complexity of full Reinforcement Learning deployment. By shifting focus from immediate clicks to "downstream rewards" that capture the quality and depth of user sessions, the system successfully mitigates the sparsity and delay of retention labels.

The authors emphasize that the framework is generalizable and model-agnostic, allowing it to be deployed across diverse surfaces (Homefeed, Search, P2P) with minimal re-engineering. The results suggest that optimizing for the quality of the user journey (depth, diversity, and avoidance of shallow interactions) is more effective for long-term retention than optimizing for raw activity volume. The work has been fully deployed in production at Pinterest, demonstrating that balancing immediate engagement with downstream value is achievable and beneficial for both user experience and business metrics.