📊 statistics

Causal Representation Learning for Generalisable Recommendation

This paper proposes a practical causal representation learning method that uses an information-theoretic disentanglement criterion to improve out-of-distribution generalization in recommender systems by leveraging only existing confounded logs, a claim validated by substantial online engagement gains in a large-scale Spotify A/B test alongside public and synthetic benchmarks.

Original authors: Yorgos Felekis, Michael O'Riordan, Oriol Corcoll, Ciarán M. Gilligan-Lee

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yorgos Felekis, Michael O'Riordan, Oriol Corcoll, Ciarán M. Gilligan-Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a music recommendation AI to pick songs for you. You feed it millions of records of what people clicked on in the past. The problem is, those past clicks aren't a random sample of all the music in the world. They are a filtered list of what the old AI decided to show people, mixed with what people happened to like at that moment.

It's like trying to learn what the best pizza in town is, but you only ever get to taste the slices that the delivery driver happened to drop off at your house. If the driver always brings you pepperoni because he thinks you like it, you might start thinking pepperoni is the only good pizza, even if you'd actually love a veggie one if you ever saw it.

This paper, titled "Causal Representation Learning for Generalisable Recommendation," tackles this exact problem. The authors, working with Spotify, propose a new way to train recommendation systems so they don't get tricked by these "delivery driver" biases.

Here is the breakdown using simple analogies:

The Problem: The "Spurious Signal"

In the world of data, there are two types of clues:

Real Causes: The actual reason someone likes a song (e.g., they love jazz).
Spurious Signals: Coincidences that look like causes but aren't (e.g., the song was played because the user was in a specific city where the old algorithm happened to push jazz).

Standard AI models are like students who cheat on a test. They memorize the "spurious signals" (the cheat sheet) because it helps them get a high score on the practice exam (the training data). But when they take the real test (the live app with new users), they fail because the cheat sheet doesn't apply anymore.

The Solution: The "Causal Detangler"

The authors propose a method called Causal Representation Learning (CRL). Think of their method as a causal detangler.

They want the AI to learn a "clean" representation of a song. This clean version should only contain the information that actually causes a user to click (the "Real Causes"). It must actively throw away the "Spurious Signals" (the biases).

To do this, they invented a new rule for the AI to follow, which they call the Disentanglement Criterion. You can think of this rule as a strict teacher giving the AI two instructions:

Be Useful: "You must still be able to predict if a user will click." (Keep the good stuff).
Be Independent: "But you must not be able to guess the user's background or the old algorithm's bias just by looking at the song." (Throw away the bad stuff).

If the AI tries to use a "cheat sheet" (a spurious signal) to make a prediction, this rule punishes it. The AI learns that the only way to get a high score is to understand the real reasons people like music, not the accidental patterns.

How They Proved It Works

The paper doesn't just guess; they tested this in three different "arenas":

The Simulation (The Lab): They built a fake world with a computer where they knew exactly which clues were real and which were fake. The "Causal Detangler" AI successfully ignored the fake clues and only used the real ones, while a normal AI got confused by the fakes.
The Public Dataset (The Classroom): They used a public dataset called KuaiRand. This dataset has two parts: one that is heavily biased (like the old delivery driver) and one that is random (like a fair taste test). The new method performed just as well as the old method on the biased data, but when tested on the fair, random data, it was much better. It proved the AI had learned the real rules, not just the bias.
The Real World (The Stadium): This is the big one. They tested this on Spotify, with millions of real users. They ran a live experiment (A/B test) where half the users got the old AI and half got the new "Causal Detangler" AI.
- The Result: The new AI didn't just look good on paper; it actually made people listen to more music and skip fewer songs. It delivered a 0.75% increase in streams and a 0.50% increase in listening time.

The Key Takeaway

The most important finding is that you cannot tell if a recommendation system is "broken" just by looking at its performance on the data it was trained on.

It's like a student who memorizes the answers to a specific practice test. They get 100% on the practice test, but that doesn't mean they understand the subject. The paper shows that standard metrics often lie. The only way to know if a system is truly smart is to see how it handles new situations where the old biases don't exist.

By using this "Causal Detangler" method, the AI learns to ignore the noise and focus on the signal, making it more robust and effective when it meets the real world.

What It Does Not Do

The paper is very specific about what this method is not:

It does not try to guess what a user would have clicked if they had seen a different song (no "what-if" scenarios).
It does not require the company to run expensive, random experiments to get the data. It works with the messy, biased logs they already have.
It does not require the AI to understand the deep, hidden "physics" of why a user likes jazz. It just needs to separate the real reasons from the fake ones to make better predictions.

In short, the paper teaches AI to stop cheating by memorizing patterns of bias and start learning the actual reasons people make choices.

Technical Summary: Causal Representation Learning for Generalisable Recommendation

Problem Statement
Predictive models in recommender systems face a fundamental generalisation gap: they are trained on observational interaction logs that are confounded by the deployed policy, past user behavior, and platform filtering. This phenomenon, known as exposure bias, creates a distribution shift between the training data (where items are selected by a specific policy) and the serving-time candidate set (which is broader and differently distributed). Consequently, offline metrics often fail to predict online performance. While Causal Representation Learning (CRL) offers a principled solution by targeting causal mechanisms, classical CRL requires strong assumptions (e.g., interventional data, multiple environments, or known anchor variables) that are rarely met in observational recommendation logs. Existing CRL approaches for recommendations often rely on complex machinery like labelled contrastive pairs or specific bias mechanisms, limiting their practicality.

Methodology
The authors propose a method motivated by CRL that targets generalisation under distribution shift without requiring full identification of latent causal factors or interventional data. The approach is grounded in a Structural Causal Model (SCM) where the observed treatment $T$ (an item) is an entangled mixture of latent causal components $T_C$ and non-causal components $T_{nC}$ . The non-causal components create a spurious backdoor path $T \leftarrow T_{nC} \leftarrow Z \rightarrow Y$ (where $Z$ is an observed confounder, e.g., user features, and $Y$ is the outcome).

The core of the method is a new Disentanglement Criterion, an information-theoretic objective designed to learn a representation $g(T)$ that:

Maximizes Utility: Retains information predictive of the outcome $Y$ given the confounder $Z$ ( $I(g(T); Y | Z)$ ).
Minimizes Confounder Dependence: Penalizes information shared with the confounder $Z$ that is not required for predicting $Y$ ( $I(g(T); Z)$ ).

The objective function is defined as:
$J(g) = I(g(T); Y | Z) - \lambda I(g(T); Z)$
where $\lambda \geq 0$ controls the trade-off.

Theoretical Guarantees
The paper proves that under specific structural assumptions (that the outcome depends on treatment only through causal latents given the confounder, and that non-causal latents retain unique spurious leakage about the confounder):

The optimal representation under this criterion discards non-causal information while preserving causal content.
There exists a "lossless regime" where, for sufficiently small $\lambda$ , the optimal representation preserves the full causal component $T_C$ without compressing it, provided the penalty reduction from compression does not outweigh the utility loss.

Tractable Implementation
Since mutual information terms are intractable from finite data, the authors derive a variational lower bound for optimization:

Utility Term: Approximated using a lower bound via the InfoNCE estimator (equivalent to standard in-batch cross-entropy ranking loss).
Penalty Term: Approximated using an upper bound via the NCE-CLUB estimator, implemented via an adversarial critic network trained to distinguish joint samples $(g(T), Z)$ from independent samples.
Optimization: The final objective is a straightforward modification of standard supervised models, adding a mutual information penalty term. It requires no randomised data, no propensity scores, and adds no inference-time cost.

Key Contributions

Disentanglement Criterion: A new information-theoretic objective proven to favor representations that retain causal content and discard confounder-dependent non-causal information at the population level.
Tractable Variational Bound: A derivation making the criterion optimisable from finite observational data alone using standard deep learning components (InfoNCE and NCE-CLUB).
Deployment-Ready Recipe: A method that modifies any standard supervised model to improve robustness under distribution shift, requiring only existing confounded logs.
Empirical Validation: Validation across three scales: a synthetic SCM, the KuaiRand benchmark, and a live A/B test on Spotify.

Results

Synthetic SCM: The method successfully isolates causal components, showing intervention sensitivity near zero (invariance to non-causal changes) while maintaining predictive accuracy (MAE) comparable to baselines.
KuaiRand Benchmark: On a real-world dataset with known distribution shifts (algorithmic vs. random exposure), the CRL variant matched the baseline on the biased training distribution (in-distribution AUC) but significantly outperformed it on the held-out random-exposure split (unconfounded distribution). This demonstrates "effective robustness."
Spotify A/B Test: In a production environment with millions of users for personalised playlist generation, a capacity-matched CRL variant performed on par with the production baseline on offline metrics but delivered substantial online gains: +0.75% track streams, +0.50% minutes played, and -0.61% track skips (all significant at 95% confidence).

Significance and Claims
The paper claims that standard offline evaluation on training distributions is often uninformative about a model's ability to generalise to serving-time distributions. The proposed method addresses the specific failure mode of exposure bias by suppressing spurious correlations directly. The significance lies in the fact that the method achieves robustness and online gains without the heavy requirements of traditional CRL (e.g., interventional data or complex causal graph identification). The results suggest that adding a causal disentanglement objective to standard supervised models can meaningfully improve out-of-distribution generalisation in production recommender systems, bridging the gap between offline metrics and online performance. The authors note limitations, including the inability to empirically verify structural assumptions on real data and the current evaluation on a single product surface.