💻 computer science

Multi-Stage Alignment of Large Language Models for Popularity Bias Mitigation in Generative Movie Recommendation

This paper proposes a multi-stage alignment pipeline combining preference extraction, supervised fine-tuning, and Direct Preference Optimization to effectively mitigate popularity bias in LLM-based movie recommenders, achieving improved novelty and catalog coverage while maintaining competitive accuracy.

Original authors: Subham Raj, Krishnakant Chourey, Sriparna Saha, Brijraj Singh, Niranjan Pedanekar

Published 2026-07-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Subham Raj, Krishnakant Chourey, Sriparna Saha, Brijraj Singh, Niranjan Pedanekar

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive, magical library that knows everything about you. You tell it, "I love sci-fi movies with time travel," and it instantly whispers a list of ten movies you might like. This is the promise of modern recommendation systems, powered by giant computer brains called Large Language Models (LLMs). These models are like super-readers who have devoured almost every book, review, and movie plot on the internet. They are incredibly smart at understanding language and context. However, there is a sneaky problem with these super-readers: they are obsessed with the "famous." Because they have read so much, they know that The Avengers or Titanic are mentioned everywhere, so they keep suggesting those same popular hits to everyone, even if you actually prefer a weird, obscure indie film from 1974. It's like a tour guide who only knows the top five tourist traps and refuses to show you the hidden gems, no matter how much you ask. This is called "popularity bias," and it makes the library feel boring and repetitive, hiding the unique treasures that might actually be perfect for you.

The researchers in this paper decided to fix this obsession with fame. They didn't just tell the computer brain to "try harder"; they built a special, three-step training camp to rewire how the model thinks about recommendations. First, they taught the model how to be a good librarian by showing it examples of what people actually liked (a step called Supervised Fine-Tuning). Then, they played a game of "this or that" where the model had to choose between a famous movie and a less famous but equally interesting one, rewarding it for picking the hidden gem (a step called Direct Preference Optimization). By separating the "learning to be helpful" part from the "learning to be diverse" part, they created a system that suggests movies you will actually enjoy, not just movies everyone else has seen. Their experiments showed that this new method successfully reduced the number of popular movies suggested by half in some cases, while still keeping the recommendations accurate and fun.

The Story of the Three-Step Fix

Think of a Large Language Model (LLM) as a very talented but slightly snobbish movie critic. This critic has read millions of reviews and watched thousands of films, but because they've seen so much, they have a bad habit: they only recommend the blockbusters. If you ask for a movie recommendation, they will almost always suggest the top 10 most popular movies of all time, ignoring the fact that you might hate action movies and love quiet documentaries. This is the "popularity bias" the paper tackles.

The authors realized that simply asking the critic to "be more diverse" doesn't work well. Instead, they designed a Multi-Stage Alignment Pipeline, which is like a three-act play to train the model to be a better, fairer guide.

Act 1: The Preference Profile (The Setup)
Before the training begins, the team takes a user's history—the movies they rated highly and the ones they hated—and turns it into a simple story. Instead of feeding the computer a messy list of numbers, they write a prompt like: "This user loves The Godfather and Inception, but they hated Gigli and Cats." This gives the model a clear, human-like picture of what the user actually likes.

Act 2: Supervised Fine-Tuning (SFT) – Learning the Rules
In this stage, the model is taught how to be a reliable recommender. The researchers show it thousands of examples where a user's story is paired with a list of movies that are actually good matches but not the most popular ones. It's like a teacher showing a student: "Here is a student who likes horror; here is a great horror movie that isn't The Exorcist. Memorize this pattern." The model learns to generate coherent, structured lists that match the user's taste, but at this point, it's just learning to follow instructions, not necessarily fighting its bias yet.

Act 3: Direct Preference Optimization (DPO) – The Bias Breaker
This is the magic step. Now that the model knows how to make a list, they teach it to prefer the underdogs. They create a game where the model sees two lists for the same user:

List A (The Rejected): Five very popular, mainstream movies.
List B (The Preferred): Five less popular movies that still fit the user's genre perfectly.

The model is trained to realize that List B is the "better" answer. It's like telling the snobbish critic, "You are wrong to pick the famous movie; the hidden gem is the right choice." By doing this, the model learns to actively suppress the urge to recommend the most famous items and instead boosts the visibility of niche, relevant content.

What They Found

The researchers tested this three-step method on three different movie datasets: MovieLens 1M (a classic dataset), Netflix Prize (a huge, messy dataset with extreme popularity bias), and Flickscore (a dataset focusing on Indian cinema and regional diversity). They compared their new method against the "old way" of just asking the model for recommendations without training (Zero-Shot) and against a simple trick where you just shuffle the list after the model makes it (Re-Ranking).

The results were quite clear:

The "Snobbish" Model: When left alone (Zero-Shot), the models recommended mostly famous movies. For example, on the Netflix dataset, the average popularity rank of recommended movies was around 1,600 (meaning they were very famous).
The New Method: After the three-step training, the models started suggesting much more diverse movies. On the Netflix dataset, the average popularity rank dropped to around 590. This means they were recommending movies that were significantly less famous, yet still relevant to the user.
Accuracy Stayed High: The best part was that the model didn't stop being accurate. It didn't start recommending random garbage just to be different. The "Precision" (how often the recommendation was actually good) stayed almost the same, while the "Novelty" (how new and unique the suggestions were) jumped up by huge margins—sometimes increasing by over 170% on the Netflix dataset.

They also found that this new method worked better than the "Re-Ranking" trick. Re-ranking is like telling the model to make a list, and then a human (or a simple computer script) goes in and swaps out the famous movies for obscure ones at the very end. The paper showed that training the model itself (Intrinsic Alignment) to want the obscure movies from the start produced better, more stable results than just fixing the list at the end.

Why This Matters

This paper suggests that we don't have to choose between a recommendation system that is accurate and one that is fair. By breaking the training process into steps—first teaching the model to understand the user, and then teaching it to value diversity—we can build AI that helps you discover the next great movie you've never heard of, rather than just telling you to watch the latest superhero blockbuster again.

The authors note that while their method works well, it does require a lot of computer power to train these models. They also point out that they didn't test this on users who have no history at all (the "cold start" problem), which is a challenge for the future. But for users who have already shared their tastes, this multi-stage approach offers a principled and scalable way to make movie recommendations feel less like a sales pitch for the most popular items and more like a genuine discovery of something special.

Technical Summary: Multi-Stage Alignment of Large Language Models for Popularity Bias Mitigation in Generative Movie Recommendation

1. Problem Statement

Large Language Models (LLMs) integrated into recommendation systems exhibit a strong inherent popularity bias. Due to their pre-training on massive corpora containing global interaction patterns, LLMs disproportionately recommend mainstream, high-frequency items while under-representing niche, user-aligned content. This bias creates a "rich-get-richer" feedback loop, limiting catalog diversity and user discovery.

While classical recommender systems have addressed this via post-hoc strategies (e.g., inverse propensity weighting, popularity-aware re-ranking), it remains unclear whether intrinsic alignment of LLMs can systematically outperform these classical methods in generative recommendation settings. Existing LLM-based approaches often rely on zero-shot prompting or single-stage supervised fine-tuning (SFT), which either fail to mitigate bias or reproduce the popularity distribution of the training data. Furthermore, there is a lack of systematic evaluation regarding how different alignment stages interact and whether bias correction should be decoupled from behavioral grounding.

2. Methodology

The authors propose a Multi-Stage Alignment Pipeline that explicitly separates behavioral grounding from bias-corrective alignment. This pipeline consists of three distinct stages:

Stage 1: Preference Extraction and Prompt Engineering

User interaction histories are transformed into structured natural-language preference profiles. A deterministic function converts a user's top 10 liked and bottom 5 disliked movies into a consistent textual prompt. This unifies input across different datasets (MovieLens 1M, Netflix Prize, Flickscore) and rating scales (5-star vs. binary).

Stage 2: Supervised Fine-Tuning (SFT)

The LLM is fine-tuned on (prompt, target) pairs to establish behavioral grounding.

Objective: Teach the model to generate coherent, structured recommendation lists that map user profiles to relevant items.
Data Construction: Targets are generated by selecting unrated movies from the bottom 50% of popularity that share high genre similarity (Jaccard index) with the user's liked movies.
Role: SFT stabilizes the model's output format and relevance but does not inherently solve the bias problem, as the model may still retain prior knowledge favoring popular items.

Stage 3: Direct Preference Optimization (DPO)

Building on the SFT model, DPO is applied to perform bias-corrective alignment.

Preference Pair Construction: For each user prompt, two lists are programmatically constructed:
- Preferred ( $y_w$ ): 5 movies from the bottom 50% of popularity with high genre overlap.
- Rejected ( $y_l$ ): 5 movies from the top 50% of popularity without genre matching.
Optimization: The model is trained to maximize the log-likelihood of the preferred list over the rejected list using the DPO loss function. This explicitly penalizes high-popularity items while rewarding topical relevance.
Key Distinction: The authors emphasize that DPO is not novel in itself, but its application after SFT is critical. Applying DPO without prior behavioral grounding leads to unstable outputs, while SFT alone fails to correct bias.

3. Experimental Setup

Datasets:
- MovieLens 1M: Balanced benchmark with 1M ratings.
- Netflix Prize: Large-scale, sparse dataset with extreme popularity bias.
- Flickscore: Regional dataset (Indian cinema) with binary ratings and multilingual context.
Models: Three open-source LLMs representing different generations and architectures:
- LLaMA 3.1 8B: State-of-the-art instruction-tuned model.
- LLaMA 2 7B: Previous generation baseline.
- Mistral 7B: Efficiency-focused architecture with Grouped-Query Attention (GQA) and Sliding Window Attention.
Baselines:
- Zero-Shot Prompting: Unmodified pre-trained models.
- Popularity-Aware Re-Ranking: A classical post-hoc baseline that penalizes high-popularity items in the generated list without modifying model parameters.

4. Key Results

The study evaluates performance using Recommendation Metrics (Precision@5, nDCG@5, MAP@5) and Bias Mitigation Metrics (Novelty, Catalog Coverage, Average Popularity).

Bias Mitigation vs. Accuracy: The multi-stage alignment (SFT + DPO) significantly reduces popularity bias while maintaining competitive accuracy.
- On Netflix Prize, DPO increased Novelty by ~170% and Catalog Coverage by ~280% compared to zero-shot baselines, while Average Popularity dropped by 63%.
- On MovieLens 1M, DPO reduced Average Popularity from ~1217 to ~575 (for LLaMA 3.1) with only a marginal drop in Precision@5.
Intrinsic vs. Extrinsic Debiasing: The proposed intrinsic alignment (SFT+DPO) consistently outperformed the classical Popularity-Aware Re-Ranking baseline.
- Intrinsic alignment achieved higher Novelty and Coverage with lower Average Popularity across all datasets and models.
- Re-ranking methods often introduced larger drops in ranking quality (Precision) compared to the more stable trade-offs observed with SFT+DPO.
Generalizability: The strategy proved robust across different model architectures (LLaMA 2, LLaMA 3.1, Mistral) and diverse cultural contexts (Western vs. Indian cinema), confirming that the decoupling of grounding and bias correction is not architecture-specific.

5. Key Contributions

Problem Decomposition: Formulated popularity bias mitigation in LLMs as a two-stage problem (behavioral grounding + bias correction) rather than a single fine-tuning objective.
Empirical Analysis of Limitations: Demonstrated that neither SFT nor DPO alone is sufficient; SFT alone reproduces bias, and DPO without grounding causes instability.
Multi-Stage Strategy: Validated a workflow where SFT grounds the model and DPO corrects bias, enabling stable, reproducible, and debiased generation.
Programmatic Preference Construction: Introduced a fully automated, dataset-driven method for constructing DPO preference pairs using item popularity statistics, eliminating the need for human annotation.
Cross-Model/Dataset Validation: Provided extensive evidence of robustness across three LLM families and three distinct datasets.
Intrinsic vs. Extrinsic Comparison: Systematically compared intrinsic alignment against classical post-hoc re-ranking, showing that embedding fairness objectives into model parameters yields superior exposure balance and stability.

6. Significance and Claims

The paper positions multi-stage LLM alignment as a principled and scalable strategy for mitigating popularity bias in generative recommender systems. The authors claim that:

Decoupling is Essential: Separating behavioral grounding (SFT) from bias correction (DPO) is necessary to achieve both stable recommendation behavior and effective debiasing.
Intrinsic Superiority: Embedding exposure objectives directly into the model via multi-stage alignment offers systematic advantages over classical post-hoc methods (like re-ranking) in candidate-free generative settings, providing better trade-offs between relevance and fairness.
Scalability: The approach is robust across different model sizes and cultural contexts, suggesting it can be applied broadly to diverse recommendation scenarios.

The authors acknowledge limitations, noting that the framework does not explicitly address extreme cold-start scenarios (where no user history exists) and that the computational cost of multi-stage fine-tuning is higher than traditional collaborative filtering. They also note that while popularity bias is mitigated, other latent stereotypes (e.g., demographic biases) inherited from pre-training data may persist.