🤖 machine learning

PageLLM: A Multi-Grained Reward Framework for Whole-Page Optimization with Large Language Models

This paper introduces PageLLM, a multi-grained reward framework that leverages implicit user feedback to simultaneously optimize coarse page-level coherence and fine item-level placement for whole-page search and recommendation, achieving significant improvements in ranking metrics and production business KPIs without requiring costly human annotations.

Original authors: Xinyuan Wang, Liang Wu, Dongjie Wang, Yanjie Fu

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinyuan Wang, Liang Wu, Dongjie Wang, Yanjie Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, bustling digital department store. Every time a customer walks in, you have a huge warehouse of products to show them. Your job isn't just to pick a few good items; you have to arrange an entire shopping page for them.

This is the challenge of Whole-Page Optimization (WPO). It's not enough to just find the right shoes; you have to decide:

Where to put the shoes on the page?
Should you show them next to socks or next to a tent?
Are you showing too many red shoes and not enough blue ones?
Is the page too cluttered, or is it boringly repetitive?

For a long time, computers struggled with this because they were too "stupid" to understand the whole picture, or they needed expensive human managers to manually grade every single page layout.

Enter PageLLM, a new system that uses a "Super-Intelligent Assistant" (a Large Language Model) to design these pages automatically. But here's the catch: The authors found that simply telling the AI "make a good page" doesn't work well. It's like telling a chef, "Make a delicious meal," without specifying if they should focus on the taste of the steak (the individual item) or the balance of the whole menu (the page layout).

Here is how PageLLM solves this, using simple analogies:

1. The Problem: The "Blind" Chef

Previous attempts to use AI for this had two big issues:

The "Human Grading" Bottleneck: To teach an AI, you usually need humans to look at two different pages and say, "I like Page A better than Page B." Doing this for millions of users is too expensive and slow.
The "Single-Grain" Mistake: Most AI systems looked at the problem in only one way.
- Some looked only at individual items (e.g., "Did the user click this specific shoe?"). This is like a chef who only cares if the steak is cooked, ignoring that the soup is cold and the salad is wilted.
- Others looked only at the whole page (e.g., "Does this page look diverse?"). This is like a chef who cares about the menu balance but doesn't notice that the steak is burnt.

2. The Solution: The "Two-Eyed" Reward System

The authors realized that to teach the AI, you need two different kinds of feedback working together, like a chef with two eyes: one eye for the details, one for the big picture.

They built a system called PageLLM that uses "implicit feedback" (what users actually do, like clicks and purchases) instead of asking humans to grade pages. They turned this messy data into four types of "training scenarios":

Relevance: Showing items the user definitely doesn't want.
Ranking: Swapping the order of items to see if the user cares who is first.
Diversity: Showing a page full of only one type of item (e.g., only red shoes) to teach the AI that variety is good.
Redundancy: Showing too many similar items clustered together to teach the AI to spread things out.

3. The Magic Trick: The "Coarse" and "Fine" Rewards

This is the core innovation. PageLLM trains two separate reward heads (think of them as two different judges) on the same data:

Judge A (The Page-Level Coach): This judge looks at the entire shopping page at once. They ask, "Is this a balanced, coherent list? Does it cover different categories?" They are great at spotting if the whole menu is boring or repetitive.
Judge B (The Item-Level Coach): This judge looks at individual items and their positions. They ask, "Did moving this item from position 5 to position 2 make the user click more?" They are great at spotting tiny, crucial details that the big picture misses.

Why both?
The paper found that if you only use Judge A, the AI makes a nice-looking page but misses the specific items users want to buy. If you only use Judge B, the AI picks great items but arranges them in a messy, confusing way.

The Result: When you combine both judges, the AI gets 46.8% better at ranking items correctly than using just one judge. It's like having a conductor (Judge A) ensuring the orchestra plays together, while a soloist coach (Judge B) ensures every violinist hits the right note.

4. Real-World Proof

The team didn't just test this in a lab; they tried it on a real e-commerce site with 10 million users.

The Outcome: The system increased the store's total sales (GMV) by 0.44% and the number of clicks by 0.14%.
Why it matters: In a business with millions of customers, a tiny percentage like that translates to tens of thousands of extra sales.

5. The "Lazy" Deployment Bonus

One clever side effect of this system is that it's easy to roll out. Because the "Page-Level Coach" (Judge A) is so good at looking at the whole list, the company can use just that part on their existing, slower computer servers (CPUs) while they slowly upgrade their powerful servers (GPUs) to run the full AI. It's like being able to use a sketch of the menu before you have the full, high-definition photo ready.

Summary

PageLLM is a new way to organize online shopping pages. Instead of asking humans to grade every page, it teaches an AI by showing it "bad examples" of page layouts (too repetitive, wrong order, missing variety). It uses two different feedback loops—one for the big picture and one for the tiny details—to ensure the AI creates pages that are both logically balanced and perfectly timed for what the user wants to click. The result is a smarter, more profitable shopping experience without needing a team of human graders.

Technical Summary: PageLLM

Problem Statement

Whole-Page Optimization (WPO) involves generating a complete, ranked page of search or recommendation results that balances relevance, diversity, positional exposure, and redundancy. While Large Language Models (LLMs) offer a unified approach to WPO by treating page generation as conditional sequence generation, their adaptation to web-scale environments faces three critical bottlenecks:

Scarcity and Cost of Supervision: Standard Reinforcement Learning from Human Feedback (RLHF) requires expensive human annotations to compare entire pages. At industrial scale, the linear growth of catalog and audience sizes makes this prohibitive.
Single-Grained Reward Limitations: Existing methods typically collapse rewards into a single scalar (either page-level or item-level). A page-level reward captures global coherence but ignores fine-grained positional shifts critical for click-through rates. Conversely, an item-level reward optimizes individual placement but fails to ensure the page as a whole is diverse and non-redundant.
Complementary Failure Modes: The paper argues that single-grained formulations fail in "mirror-image" ways. Removing page-level signals degrades global list coherence, while removing item-level signals degrades positional sharpness. Neither signal alone is sufficient to maximize user satisfaction.

Methodology: The PageLLM Framework

PageLLM proposes a three-stage, reward-based fine-tuning framework that decouples supervision into two complementary granularities using only implicit user feedback.

1. Data Construction from Implicit Feedback

Instead of human labels, the framework constructs training data from user-item interaction logs (e.g., Amazon Review corpus):

Golden Lists: For each user, a "golden" ranked list is constructed by filtering positive interactions ( $r_{ui} > 3$ ), re-ranking by fine-grained popularity, and balancing category diversity.
Preference-Pair Families: Four families of negative lists are generated to localize specific failure modes, creating contrastive pairs against the golden list:
- Non-preferred ( $\sigma_{np}$ ): Items with low ratings (relevance failure).
- Ranking ( $\sigma_{rk}$ ): Adjacent swaps of items in the golden list (positional failure).
- Diversity ( $\sigma_{d}$ ): Replacement of unique items with homogeneous ones (diversity failure).
- Redundancy ( $\sigma_{rd}$ ): Over-concentration of items from one category (redundancy failure).

2. Multi-Grained Reward Modeling

The core innovation is a decoupled reward model with two heads trained on the same preference data but with different objectives:

Coarse-Grained (Page-Level) Reward ( $R_c$ ): A Bradley-Terry model that scores the entire generated list as a single unit. It learns global cues like category mix and average relevance, providing stability against noisy feedback.
Fine-Grained (Item-Level) Reward ( $R_f$ ): A model that scores individual items and averages them. It is trained to detect engagement-sensitive position swaps (e.g., how swapping two adjacent items affects the overall score).
Combination: The final reward is a simple sum ( $R = R_c + R_f$ ). The authors argue this additive approach is superior to a product, as it preserves independent gradients and allows for super-additive performance gains.

3. RLHF Fine-Tuning with PPO

A pre-trained LLM (e.g., GPT-2) undergoes a three-step training process:

Meta-Information Pre-training: The model is adapted to recommendation semantics via rating prediction and next-token prediction on meta-prompts.
Supervised Fine-Tuning (SFT): The model is fine-tuned on the golden lists to learn the output schema and baseline distribution.
PPO Optimization: The model is further optimized using Proximal Policy Optimization (PPO) with the combined multi-grained reward. A KL-divergence penalty against the SFT reference policy prevents the model from drifting into degenerate output spaces.

Key Contributions

Feedback-to-Reward Pipeline: The formalization of four families of preference pairs that transform noisy implicit feedback into usable RLHF supervision for page generation without human labels.
Multi-Grained Reward Architecture: The introduction of a decoupled reward system pairing a page-level Bradley-Terry model with an item-level position-swap model, empirically demonstrating their complementarity.
Scalable Evidence: Comprehensive evaluation across seven Amazon categories against eleven baselines, a 10-million-user online A/B test, and studies on cold-start robustness and backbone efficiency.
Deployment Modularity: A demonstration that the additive nature of the reward allows the coarse page-level component to be deployed as a CPU-side ranking feature, facilitating incremental rollout on existing infrastructure.

Experimental Results

Offline Performance: On seven Amazon categories, PageLLM outperforms eleven baselines (including matrix factorization, sequential transformers, and other LLM-based recommenders). It achieves up to a 46.8% improvement in NDCG@100 compared to a no-reward variant. The gains are most pronounced in categories with nuanced preferences (Luxury, Sports, Instruments, Beauty).
Ablation Studies: Removing either the page-level or item-level reward head significantly degrades performance (reducing NDCG@100 by 17.8% and 15.2%, respectively), confirming that the signals are complementary, not redundant.
Online A/B Test: Deployed on a commercial e-commerce search engine with over 10 million unique users, PageLLM increased GMV by 0.44% and Click-Through Rate (CTR) by 0.14% over a one-week period.
Robustness: In cold-start simulations (50% data reduction), PageLLM degraded less than sequential (SASRec) and autoencoder (Multi-VAE) baselines.
Backbone Efficiency: Experiments comparing GPT-2 and Llama3.2-1B suggest that GPT-2 offers a Pareto-optimal balance of performance and computational cost for this task, indicating that the reward structure is the limiting factor rather than model capacity.

Significance and Claims

The paper claims that the primary bottleneck for LLM-based WPO is not the lack of supervision data, but the granularity of the supervision signal. By decoupling the reward into page-level and item-level components, PageLLM successfully aligns LLMs with web-scale WPO objectives using only implicit feedback.

The authors emphasize that this approach:

Eliminates the need for costly human annotations.
Solves the "single-grained" failure mode where existing methods cannot simultaneously optimize for global page coherence and local item placement.
Provides a practical path to production deployment, as evidenced by the successful 10M-user A/B test and the ability to partially deploy components on CPU infrastructure.

The work concludes that multi-grained rewards derived from implicit feedback are sufficient to scale LLMs to production-level whole-page optimization, offering a new route for recommender systems that treats the page as a cohesive, generative entity rather than a bag of scored items.