🤖 AI

Progressive Alignment of Recommender Foundation Model through Multi-Phase Post-Training

This paper proposes a three-phase progressive post-training framework (Linear Probing, Full Fine-Tuning, and Reinforcement Fine-Tuning) that effectively aligns foundation models for recommendation with business metrics by decoupling task adaptation from reward-based optimization, demonstrating superior performance in both offline experiments and large-scale online A/B tests.

Original authors: Oseong Choi, Hoeinn Kim, Jihoon Lee, Byungsoo Kang, Taeyeong Jang

Published 2026-08-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Oseong Choi, Hoeinn Kim, Jihoon Lee, Byungsoo Kang, Taeyeong Jang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive, magical library where the books talk back. This isn't just any library; it's a "Foundation Model" library. Think of this model as a super-smart, well-traveled librarian who has read billions of stories and knows exactly how people behave when they are browsing. This librarian has already learned the general rules of storytelling and human curiosity. But here's the tricky part: just because the librarian knows how to find a book doesn't mean they know what the library owner actually wants. The owner might care about how long people stay in the library, or how many of them buy a membership, not just how many books they grab off the shelf.

In the world of computer science, specifically in Recommender Systems, this is the daily struggle. We have these giant, pre-trained AI models (the librarians) that are great at predicting what you might click on next. But the business wants more than just clicks; they want deep engagement, like finishing a whole comic series or paying for premium content. The problem is that the signals for "clicks" are everywhere and easy to see, while the signals for "deep satisfaction" are rare and hard to find. If you try to teach the librarian to chase the rare signals directly, they often get confused, forget everything they already knew, and start recommending weird things. This paper explores a clever, step-by-step way to train these digital librarians so they don't just guess what you'll click, but actually help you find stories you'll love for the long haul.

The Three-Step Dance: How to Train a Recommendation Librarian

The researchers from NAVER WEBTOON, a giant platform for digital comics, realized that trying to teach their AI everything at once was a recipe for disaster. They proposed a "Progressive Alignment" framework, which is basically a three-phase training camp designed to turn a general-purpose AI into a business-savvy recommendation expert without breaking its brain.

Phase 1: The Warm-Up (Linear Probing)
Imagine you have a master chef (the pre-trained Foundation Model) who knows how to cook anything. You want them to make a specific new dish for a restaurant, but you don't want them to forget how to cook everything else. If you hand them a brand-new, random set of knives and pans (the "downstream modules") and tell them to start cooking immediately, they might panic and ruin their existing skills.

So, the first step is Linear Probing (LP). In this phase, the master chef's brain is frozen—they can't change their fundamental cooking style. You only let them practice with the new knives and pans. This helps the new tools get comfortable in the chef's hands without messing up the core recipes. It's like letting a new employee shadow the boss for a week before they are allowed to touch the stove. This stabilizes the connection between the AI's vast knowledge and the specific task of recommending comics.

Phase 2: The Full Practice (Full Fine-Tuning)
Once the new tools are stable, it's time to let the chef cook with full freedom. This is Full Fine-Tuning (FFT). Now, the whole model is unlocked. The chef can tweak their entire cooking style to specialize in this specific restaurant's menu. Because the new tools were already warmed up in Phase 1, the chef doesn't get overwhelmed. They can learn the specific tastes of the customers (the "downstream task") while keeping the solid foundation of their general knowledge intact. The researchers found that doing this in two steps (LP then FFT) was much better than trying to do it all in one giant leap, which often leads to "catastrophic forgetting"—where the AI forgets how to be smart because it's too busy trying to be specific.

Phase 3: The Reward Game (Reinforcement Fine-Tuning)
Here is where the magic happens. The chef is now good at cooking the menu, but are they cooking what the owner actually wants? The owner cares about deep satisfaction (like a customer buying a whole season of comics), not just a quick snack (a single click). But deep satisfaction is rare; most people just browse.

If you try to teach the chef to aim only for the rare "deep satisfaction" wins, they might start guessing wildly. Instead, the researchers used a Reward Model. Think of this as a taste-tester who has a special scorecard. The taste-tester looks at the rare, deep interactions and assigns a "reward score" to them.

In the third phase, Reinforcement Fine-Tuning (RFT), the chef doesn't just cook to please the crowd; they cook to please the taste-tester's score. The AI generates a bunch of recommendations, the taste-tester (the Reward Model) gives them scores based on how likely they are to lead to deep engagement, and the AI learns to adjust its strategy to get higher scores. Crucially, the AI still uses the "dense" signals (like clicks) it learned in Phase 2 to know what to recommend, but the "reward" signals guide how to rank them to maximize long-term happiness.

What They Found (and What They Said "No" To)

The team tested this three-step method on a massive dataset of real user interactions from their Webtoon platform. They compared their new method against the old ways of doing things.

The "Don't Do This" List:
The paper explicitly argues against two common shortcuts:

Don't just train on the rare business goals directly. If you try to teach the AI to optimize for "paid content completion" right from the start, it fails to generalize. It's like trying to teach a student to solve advanced calculus problems before they've learned multiplication. The signals are too sparse (too rare), and the model gets lost.
Don't just use the Reward Model as the final recommender. The researchers tried using the "taste-tester" (the Reward Model) directly as the person handing out recommendations. While this model was good at spotting deep engagement, it was terrible at ranking the millions of available comics for a click. It lacked the "discrimination" needed to pick the best item from a huge list. The paper suggests that the Reward Model is best used as a coach, not the player.

The Winning Strategy:
The "Progressive Alignment" (LP → FFT → RFT) worked best.

Offline Tests: When they ran simulations on historical data, the three-phase model beat the single-phase models. Specifically, the version that used GRPO (a specific type of reinforcement learning algorithm) with the learned reward model achieved the highest scores. It managed to keep the high click-rates (Rank NDCG) while also pushing users deeper into the content funnel (Funnel NDCG).
Real-World Proof: They didn't just stop at simulations. They ran a massive A/B test on their live platform. They compared their new AI against their standard, non-foundation model. The results showed that the new framework consistently improved user engagement. The "GRPO with Reward Model" version was particularly good at getting users to read further and reach paid episodes, while the "DPO" (another algorithm) version was slightly better at getting the initial click.

The Takeaway

The paper suggests that the secret to making powerful AI recommenders isn't just throwing more data at them or trying to optimize for the hardest goal immediately. It's about stages. First, stabilize the new tools. Second, specialize the model. Third, use a learned "reward" signal to gently steer the model toward the business's long-term goals without losing its ability to make good, immediate choices.

By separating the "learning the task" from the "aligning with business goals," the researchers found a way to make their AI both smart and profitable. It's a reminder that sometimes, to get the best result, you have to take it slow, step-by-step, rather than trying to sprint to the finish line.

Technical Summary: Progressive Alignment of Recommender Foundation Model through Multi-Phase Post-Training

Problem Statement

While Foundation Models (FMs) for recommendation have demonstrated strong capabilities in modeling long-horizon sequential user behavior, their practical deployment faces two critical challenges:

Catastrophic Forgetting: Directly fine-tuning massive pretrained backbones alongside randomly initialized task-specific modules in a single step often destabilizes the pretrained representations, leading to the loss of general behavioral knowledge.
Misalignment with Business Metrics: Standard Supervised Fine-Tuning (SFT) optimizes for dense implicit feedback (e.g., clicks, likes). However, these task-level objectives do not always align with sparse, delayed business metrics (e.g., long-term retention, paid conversion) that ultimately determine recommendation quality. Furthermore, directly optimizing serving policies on sparse business labels often results in poor generalization and weak ranking discrimination compared to policies trained on dense signals.

Existing approaches struggle to balance the stability of adaptation with the necessity of aligning the serving policy with long-term business objectives without sacrificing the ranking capacity built on dense feedback.

Methodology

The authors propose a three-phase progressive post-training framework that explicitly decouples downstream knowledge adaptation from business-metric alignment. The framework consists of the following stages:

1. Model Architecture

The system employs a Foundation–Expert paradigm. A pretrained sequential encoder (FM backbone) generates user embeddings, which are fused with a lightweight, task-specific downstream module encoding surface-specific features.

Downstream Encoder: Uses cross-attention pooling to compress the FM sequence into a fixed representation, concatenated with downstream features.
Reward Model: To handle sparse business objectives, a reward model is trained to predict the depth of user engagement (modeled as a 6-level ordinal funnel from impression to paid-content completion). This model incorporates an impression propensity head to correct for exposure bias using Self-Normalized Inverse Propensity Scoring (SNIPS).

2. Phase 1: Linear Probing (LP)

Goal: Stabilize the integration of task-specific components without corrupting the pretrained backbone.
Process: The FM backbone is frozen. Only the downstream module and prediction heads are optimized.
Objective: Minimize a combined loss of policy (click prediction), reward (funnel depth), and propensity (exposure) to align the semantic spaces of the downstream components with the frozen FM representation.

3. Phase 2: Full Fine-Tuning (FFT)

Goal: Jointly specialize the entire model for the target task while preserving the stability gained in Phase 1.
Process: The FM backbone is unfrozen. All parameters are updated.
Strategy: A discriminative learning-rate strategy is employed, where the FM backbone is updated with a learning rate one order of magnitude smaller than the downstream modules. This allows rapid adaptation of task-specific components while the backbone co-adapts conservatively to prevent catastrophic forgetting.

4. Phase 3: Reinforcement Fine-Tuning (RFT)

Goal: Align the stabilized serving policy with long-term business objectives using the learned reward model as a signal, rather than optimizing directly on sparse labels.
Process: The model snapshot from the end of FFT serves as a frozen reference policy ( $\pi_{ref}$ ). The policy head and FM backbone are optimized using Group Relative Policy Optimization (GRPO) or Direct Preference Optimization (DPO).
Mechanism:
- GRPO: For a given user state, the policy samples a group of candidate actions. The reward model provides dense scalar feedback. Advantages are computed by normalizing rewards within the group, and the policy is updated to maximize the clipped surrogate objective with KL regularization against $\pi_{ref}$ .
- DPO: Preference pairs are constructed based on the reward model's scores (or observed funnel outcomes), and the policy is optimized to prefer higher-reward items without explicit reward modeling during the update step.
Key Distinction: The reward model is not deployed as the serving policy. Instead, it acts as an alignment signal to steer the dense-feedback-trained policy toward business goals.

Key Contributions

Progressive Adaptation Framework: The paper introduces a structured LP-FFT-RFT pipeline that mitigates catastrophic forgetting by separating the stabilization of downstream heads from full-model specialization.
Decoupled Alignment Strategy: It proposes a novel separation of roles where dense implicit feedback trains the serving policy's ranking capacity, while sparse business metrics train a reward model used solely for alignment signals. This avoids the generalization issues of direct sparse-label optimization.
Reward-Based Alignment vs. Direct Serving: The work demonstrates that using a learned reward model as an alignment signal (via RFT) yields superior performance compared to deploying the reward model directly as the ranking policy.
Empirical Validation: The framework is validated through extensive offline ablation studies and large-scale online A/B tests on a production Webtoon platform.

Results

Offline Evaluation

LP-FFT Superiority: The two-phase SFT (LP $\to$ FFT) with discriminative learning rates outperforms single-phase baselines (either LP-only or FFT-only) on both Rank NDCG (full-catalog click ranking) and Funnel NDCG (ranking impressed titles by engagement depth).
RFT Effectiveness: Applying RFT to the stabilized SFT policy further improves both metrics.
Alignment vs. Direct Serving:
- A policy trained directly on the reward model achieved high Funnel NDCG but poor Rank NDCG, indicating it lacks discrimination for full-catalog ranking.
- GRPO with Reward Modeling achieved the best overall performance, matching the Funnel NDCG of the direct reward model while maintaining significantly higher Rank NDCG.
- DPO showed mixed results; while DPO with observed funnel preferences performed well, DPO with reward-model-induced preferences underperformed, suggesting that converting scalar rewards to binary pairs may discard useful relative information.

Online A/B Testing

Baseline Improvement: The 2-phase SFT foundation model baseline consistently outperformed the conventional non-foundation production model (CTRL) across all in-session metrics (Click, Engaged Reading, Paid Conversion).
Alignment Gains: The RFT-aligned variants provided additional gains over the SFT baseline.
- GRPO w/ RM: Showed the strongest improvements on mid-to-deep engagement metrics (e.g., reaching the last free episode, last paid episode), demonstrating effective steering toward long-term business goals.
- DPO w/o RM: Showed the largest gains on immediate interaction metrics (Click/Imp), suggesting it is highly effective for top-of-funnel ranking when using direct observed preferences.
Stability: All foundation-model variants exhibited stable lift after an initial transient period, with GRPO w/ RM showing the most persistent gains on deep engagement outcomes.

Significance and Claims

The paper claims that progressive post-training is a practical and effective approach for deploying foundation models in industrial recommender systems. Its primary significance lies in:

Stability: Explicitly separating the adaptation of task-specific knowledge (via LP and FFT) from reward-driven alignment prevents the instability and catastrophic forgetting associated with direct full-parameter optimization.
Bridging the Gap: The framework successfully bridges the gap between dense implicit feedback (which provides strong ranking discrimination) and sparse business metrics (which define true value) by using the latter only as a reward signal for alignment, rather than as a direct training objective for the serving policy.
Production Viability: The large-scale online results confirm that this multi-phase approach improves production recommendation quality over conventional baselines, validating that separating supervised adaptation from reward-based alignment enhances both optimization stability and downstream serving performance.