💻 computer science

Why This, Not That? Mining User Profiles for Pair-wise Counterfactuals

This paper proposes a new task for pairwise item ranking explanations grounded in recommendation algorithm logic, utilizing counterfactual learning to identify specific user profile items that determine why one item is ranked higher than another.

Original authors: Meysam Varasteh, Veronika Bogina, Noam Koenigstein, Robin Burke

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Meysam Varasteh, Veronika Bogina, Noam Koenigstein, Robin Burke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

We live in an age where machines constantly suggest what we should watch, read, or buy. For decades, researchers have focused on answering a simple question: "Why did the system recommend this specific movie to me?" They have developed ways to explain that single choice, often by pointing to a feature the user likes, such as a favorite actor or a preferred genre. However, human decision-making rarely happens in a vacuum. When we choose one option over another, we are often comparing them side by side. We ask, "Why did I pick this film instead of that one?" This comparative question is fundamental to how we understand our own preferences, yet it has received far less attention from the designers of recommendation systems. Understanding the difference between two choices is often more revealing than understanding a single choice in isolation, because it highlights the specific factors that tipped the scale.

A team of researchers set out to solve this problem by developing a new way to explain why a recommendation system ranks one item higher than another. They approached the task by asking a "what if" question: what would have to change in a person's history for the system to swap the order of these two items? This method, known as counterfactual reasoning, looks for the minimal set of changes needed to reverse a decision. In the context of a movie recommendation, the researchers wanted to identify the specific films a user had previously watched that caused the system to prefer a science fiction classic over a historical epic. If those specific past views were removed from the user's history, the system should theoretically recommend the historical epic first. The goal was to find the smallest, most precise group of past items that, if erased, would flip the ranking.

To test this idea, the researchers built a new type of computer model designed to act as an explainer. They trained this model on three large collections of user data: one containing over half a million movie ratings, another with music preferences, and a third with pinning activity. They tested their approach against several existing methods, including techniques that explain single items or those that try to compare items by looking at their general features. The researchers found that their new method, which learned to look at the target item and the competing item simultaneously, was far more effective. It could identify the specific past items responsible for the ranking difference with greater accuracy and using fewer items than the other methods. In many cases, the new model could explain the preference for the top item over the second item for more than 80 percent of the users in their test groups, often requiring the removal of only a small number of past interactions to achieve the swap.

The study also revealed that the quality of these explanations depends heavily on the type of recommendation system being used. When the underlying system was based on a method that learns hidden patterns in user behavior, the new explainer performed exceptionally well. However, the researchers noted a complication involving popular items. Sometimes, the most effective way to change a ranking was to remove extremely famous movies or songs that almost everyone has seen. While this is mathematically correct, the researchers suspected it might not be a helpful explanation for a real person. Telling a user, "You got this movie because you watched Star Wars and Titanic," might feel obvious or unhelpful, as those are items almost everyone has seen. To test this, they ran a version of their model that ignored the most popular items. This version could still find explanations, but it worked for fewer people, suggesting that popular items do play a significant role in how these systems make decisions, even if they are not the most interesting part of the story.

The researchers concluded that their approach successfully generates comparative explanations that are both accurate and concise. They showed that by training a model to look at two items at once, rather than one at a time, they could uncover the specific reasons for a ranking difference. The results suggest that this method is a viable way to make recommendation systems more transparent, allowing users to see not just why they got a suggestion, but why they got one suggestion instead of another. While the current work focuses on the technical ability to generate these explanations, the researchers noted that the next step is to see if people actually find these explanations useful and easy to understand. They plan to conduct studies with human users to determine if these "what if" scenarios help people make sense of the choices machines make for them.

Technical Summary: Mining User Profiles for Pair-wise Counterfactuals

Problem Definition

The paper addresses a gap in recommender system (RS) explanation research. While existing methods effectively explain why a single item is recommended (e.g., "Why was Item A recommended?"), they often fail to address the comparative questions users naturally ask, such as "Why is Item A ranked higher than Item B?"

Drawing on psychological findings that humans prefer contrastive explanations for decision-making, the authors propose a new task: Pairwise Interpretation of Item Rankings. The goal is to generate a counterfactual explanation that identifies the minimal set of items in a user's profile whose removal would reverse the ranking order of a target item ( $T$ ) and a comparative item ( $C$ ). The explanation takes the form: "Item $T$ is ranked above $C$ because the user profile includes items $x_1, x_2, x_3$ . If these were absent, $C$ would be ranked higher."

The authors define two quality metrics for these explanations based on Grice's maxims:

Veracity: Removing the identified items must actually cause the recommender to rank $C$ above $T$ .
Succinctness: The set of removed items (the perturbation) should be as small as possible.

Methodology

The proposed approach extends the LXR (Learning-based eXplanation for Recommendations) framework, which was originally designed for single-item counterfactual explanations. The authors introduce a class of techniques called CLXR (Comparative LXR) to handle the pairwise task.

1. Baseline Adaptation: CLXR-score

This approach reuses a pre-trained LXR explainer ( $e_l$ ) trained for single items. It generates two separate masks: one for the target item ( $m^l_T$ ) and one for the comparative item ( $m^l_C$ ). The comparative mask ( $m_C$ ) is computed via element-wise multiplication: $m_C = m^l_T \odot (1 - m^l_C)$ . This logic assumes that items contributing to $T$ 's rank but not $C$ 's rank are the cause of the preference.

2. Direct Optimization Approaches

The authors propose two methods that directly optimize for the comparative objective using a loss function composed of four terms:

$\mathcal{L}_{cmp}$ : Maximizes the score gap between $T$ and $C$ after applying the mask.
$\mathcal{L}_{pred}$ : Ensures $T$ remains highly ranked when supported by the mask.
$\mathcal{L}_{inv}$ : Ensures $T$ 's rank drops when the mask is inverted (items removed).
$\mathcal{L}_{reg}$ : Enforces sparsity (minimizing the size of the perturbation).

Two specific implementations are detailed:

CLXR-joint: Uses a Siamese network architecture. A shared intermediate network ( $e_s$ ) processes both $T$ and $C$ to generate intermediate masks, which are then combined to form the final comparative mask. The network is trained end-to-end using the combined loss function (Eq. 2).
CLXR-tdlr: Adapts the Targeted Deviation - Logit Ranking (TDLR) loss, commonly used in adversarial attacks. This method directly learns the comparative mask without an intermediate network, relying solely on the comparative loss and sparsity regularization.

3. Evaluation Metrics

Since no standard metrics exist for pairwise explanations, the authors define:

MPRR (Minimal Perturbation for Ranking Reversal): The smallest fraction (or count) of top-ranked profile items (based on the explainer's scores) that must be removed to reverse the rank of $T$ and $C$ . Lower is better.
Coverage: The percentage of user-item pairs for which a valid perturbation (causing a rank reversal) can be found within the search space.

Experimental Setup

The study evaluates the methods across three datasets (MovieLens-1M, Yahoo Music, Pinterest) and two recommendation algorithms:

Matrix Factorization (MF): A standard latent factor model.
Variational Autoencoder (VAE): A probabilistic generative model.

The CLXR models are compared against several baselines:

Popularity (POP): A simple popularity-based ranker.
SHAP & LIME: Model-agnostic perturbation methods (adapted for RS).
ACCENT: A counterfactual method based on influence functions.
LXR: The original single-item explainer.

Key Results

The experiments demonstrate that explicitly optimizing for the comparative task yields superior results compared to adapting single-item methods or using model-agnostic baselines.

Performance: CLXR-joint consistently outperforms all baselines in both Coverage and MPRR across most dataset/recommender combinations. For example, on the MovieLens dataset with a VAE recommender, CLXR-joint achieved 92% Coverage with an MPRR of 21 items, significantly outperforming the next best method (LXR at 82% Coverage, 30 items).
CLXR-tdlr is also highly competitive, particularly with Matrix Factorization models.
Baselines: While LXR (single-item) performs reasonably well, it generally falls short of the jointly optimized CLXR models. SHAP and LIME often suffer from lower coverage or require larger perturbations. ACCENT performs well in specific VAE/Pinterest scenarios but lacks the consistency of CLXR-joint.
Popularity Bias: The authors investigated the impact of popular items on explanations. When popular items (top 15%) were filtered out during training (CLXR-filter), Coverage dropped significantly (e.g., from 81% to 56.3% on ML-1M with MF). This suggests that while popular items often drive the ranking reversals, relying on them may produce explanations that are technically correct but potentially unsatisfying or unintuitive for users (e.g., "If you hadn't watched Star Wars...").

Significance and Claims

The paper positions itself as an initial foray into comparative ranking explanations using counterfactual learning. Its primary contributions are:

Task Formulation: It formally defines the "Why is $T$ ranked above $C$ ?" problem and argues that effective solutions must be grounded in the logic of the recommendation algorithm itself, rather than being decoupled.
Methodological Innovation: It demonstrates that counterfactual learning can be effectively adapted to approximate profile perturbations for comparative explanations. The CLXR-joint approach, utilizing a Siamese network and a combined contrastive objective, is shown to be the most effective method for generating succinct and verifiable explanations.
Empirical Validation: Through extensive experiments, the authors show that explicitly optimizing for the contrastive objective (CLXR-joint) leads to more faithful and succinct explanations than adapting single-item methods (CLXR-score) or using model-agnostic techniques.
Limitations and Future Work: The authors modestly acknowledge that their current method provides efficient approximations rather than guaranteed minimal explanations. They note that some rank reversals may be impossible due to strong popularity biases (e.g., a niche film vs. a blockbuster). They conclude that while their offline metrics validate technical performance, future work must include human-centered evaluation to determine if these counterfactual perturbations are actually perceived as useful and actionable by real users. They also suggest that future work should explore summarization mechanisms (e.g., clustering) to handle cases where large perturbations are required.