📈 economics

Two-Sided Prioritized Ranking: A Coherency-Preserving Design for Marketplace Experiments

This paper proposes Two-Sided Prioritized Ranking (TSPR), a novel experimental design that leverages position bias in ranked lists to estimate the total average treatment effect of price changes in online marketplaces while preserving platform coherency and mitigating interference bias.

Original authors: Mahyar Habibi, Zahra Khanalizadeh, Negar Ziaeian

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mahyar Habibi, Zahra Khanalizadeh, Negar Ziaeian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, busy hotel booking website. You want to test a new idea: What happens if we lower the prices of 25% of our hotels?

In a perfect world, you could just show the cheap prices to half your visitors and normal prices to the other half, then compare the results. But in the real world of online marketplaces, this is a nightmare for two reasons:

The "Fairness" Problem (Coherency): If you show a user a hotel for \ $100 and their friend sees the *same* hotel for \$ 80, the first user will feel cheated. They might quit the site and never come back. You can't show different prices to different people for the same item.
The "Crowded Room" Problem (Interference): Your search results are a list of 20 hotels. If you lower the price of Hotel A, people will click on it instead of Hotel B. Even if Hotel B's price didn't change, its sales will drop because it lost customers to Hotel A. This is called interference. If you try to measure the effect of the price drop on Hotel A, you accidentally mess up the data for Hotel B, too.

The Old Ways (And Why They Fail)

The "Split the Room" Method (User-Level A/B Test): You show the cheap prices to Group A and normal prices to Group B.
- Why it fails: It violates the "Fairness" rule. Group A sees a \ $100 price tag; Group B sees \$ 80. People get angry.
The "Random Shuffle" Method (Item-Level A/B Test): You keep the same list for everyone, but you randomly mark some hotels as "treated" (price drop) and others as "untreated."
- Why it fails: It violates the "Crowded Room" rule. Because the treated hotels are cheaper, they steal all the attention from the untreated ones. Your data gets messy because the items are fighting each other.
The "Grouping" Method (Cluster Randomization): You group hotels by neighborhood (e.g., "Downtown" vs. "Beach") and give the whole Downtown group a price drop.
- Why it fails: It's too slow and imprecise. You only have a few neighborhoods, so your data is "noisy." It's like trying to guess the weather by looking at only two clouds instead of the whole sky.

The New Solution: Two-Sided Prioritized Ranking (TSPR)

The authors of this paper propose a clever trick called TSPR. Instead of changing what people see (the prices), they change where things appear in the list.

Think of your search results like a concert lineup.

The Headliners (Top of the list): Everyone sees them. They get 90% of the attention.
The Opening Acts (Bottom of the list): Only the most dedicated fans see them. They get almost no attention.

How TSPR works:

The Setup: You have a list of hotels. You secretly mark some as "Cheap" (Treated) and some as "Normal" (Untreated). You also add a few "Dummy" hotels (Placebo) to act as a buffer so the lists look balanced.
The Split: You divide your visitors into two groups: Group A and Group B.
The Magic Trick:
- Group A sees a list where the Normal hotels are at the very top (Headliners), and the Cheap hotels are pushed to the bottom (Opening Acts).
- Group B sees a list where the Cheap hotels are at the very top (Headliners), and the Normal hotels are pushed to the bottom.
The Result:
- Everyone sees the same prices. No one is cheated. Fairness is preserved!
- The "Cheap" hotels get a massive boost in attention for Group B because they are at the top.
- The "Cheap" hotels get almost no attention for Group A because they are hidden at the bottom.

Why This Solves the Problem

Because Group B sees the cheap hotels first, they buy them. Because Group A sees the normal hotels first, they buy those.

By comparing the total sales of Group B vs. Group A, you can mathematically figure out exactly how much the price drop helped, without the "Crowded Room" problem messing up your numbers. You aren't fighting the interference; you are using the interference (the fact that people only look at the top of the list) to your advantage.

The Analogy: The "Golden Ticket" Parade

Imagine a parade with 100 floats.

The Problem: You want to know if giving a float a "Golden Ticket" (a discount) makes it more popular. But if you give the ticket to a float in the middle, people ignore it because they are watching the float at the front.
The Old Way: You give tickets to random floats. The ones at the front get all the love, the ones at the back get none. You can't tell if the ticket worked or if it was just the position.
The TSPR Way:
- Parade A: You put all the "Golden Ticket" floats at the very back. The crowd barely sees them.
- Parade B: You put all the "Golden Ticket" floats at the very front. The crowd goes wild for them.
- The Math: Since the only difference between the two parades is where the tickets are placed, the difference in crowd reaction tells you exactly how powerful the "Golden Ticket" is.

The Bottom Line

The paper proves that by simply reordering the list (putting treated items at the top for one group and the bottom for another), companies can run fair, accurate experiments without breaking trust with their users or getting confused by items competing with each other.

It's a "win-win":

Users see consistent, fair prices.
Companies get clear, accurate data on what works, even in a messy, competitive marketplace.

1. Problem Statement

The paper addresses a critical challenge in online marketplace experimentation: estimating the Global Treatment Effect (GTE) of interventions (e.g., price changes) in environments characterized by interference and coherency constraints.

Interference (SUTVA Violation): In ranked lists (e.g., search results), items compete for limited user attention and demand. Changing the price or treatment status of one item affects the outcomes of other items in the same list (substitution or complementarity). Standard item-level A/B tests fail here because they assume the Stable Unit Treatment Value Assumption (SUTVA), leading to biased estimates.
Coherency Constraints: Platforms often face operational and legal requirements that prevent standard experimental designs:
- Price Parity: All users must see the same price for a given item (ruling out user-level randomization where different users see different prices).
- Full Catalog Access: Users must see the same set of available items (ruling out designs that hide items from certain users).
The Gap: Existing solutions like cluster randomization suffer from low statistical power (high variance), while standard item-level randomization suffers from high bias due to interference. There is a need for a design that preserves coherency while accurately estimating global treatment effects under interference.

2. Methodology: Two-Sided Prioritized Ranking (TSPR)

The authors propose TSPR, an experimental design that leverages position bias (the tendency of users to focus on top-ranked items) to create variation in treatment exposure without violating coherency.

Core Mechanism

Item Partitioning: Items are randomly partitioned into three subsets: Treated, Untreated, and Placebo.
- The Placebo subset is crucial to ensure the Treated and Untreated groups are of comparable size, preventing quality imbalances between groups.
User/Query Randomization: Incoming queries are split into two groups, $Q_A$ and $Q_B$ .
Re-Ranking Strategy:
- Group $Q_B$ (Treatment-Prioritized): Items are reordered so that Treated items appear at the top, followed by Placebo, then Untreated.
- Group $Q_A$ (Control-Prioritized): Items are reordered so that Untreated items appear at the top, followed by Placebo, then Treated.
Coherency Preservation: All users see the exact same set of items at the same prices. The only difference is the order in which they appear, which systematically alters exposure to the treatment due to position bias.

Theoretical Framework

Estimand: The Global Lift ( $\Phi$ ), defined as the proportional change in total outcomes when moving from a world where all items are untreated to a world where all items are treated.
Identification Logic:
- The design assumes that user attention decays with rank (Attention Function $F(l)$ ).
- By prioritizing treated items for one group and untreated items for the other, the experiment creates a "partial outcome" contrast at different depths ( $l$ ) of the list.
- The authors derive that the partial lift at depth $l$ , $\phi(l)$ , converges to the global lift $\Phi$ as $l \to \infty$ .
- Assumptions:
  1. Separability: Treatment affects the level of outcomes but not the shape of the attention function.
  2. Symmetric Distortion: The re-ranking perturbation affects attention similarly in both arms.
  3. Partial Treatment Effect: The effect of treating a block of items scales with the block size (captured by a function $\tau(l)$ ).
  4. Negligible Contamination: Treated items appearing deep in the control arm have minimal impact due to low attention.

Estimation

Two estimators are proposed to recover $\Phi$ from the partial lift data:

Parametric (Weighted Least Squares): Assumes a specific functional form for the scaling factor $\tau(l)$ (e.g., $1 + 1/l$ ) and fits a line to the observed partial lifts.
Non-Parametric (Isotonic Regression): Uses shape restrictions (monotonic convergence of partial lift to global lift) to estimate $\Phi$ without assuming a specific functional form, offering robustness.

3. Key Contributions

Novel Experimental Design: TSPR is the first design to simultaneously satisfy price parity, full catalog access, and interference robustness in ranked-list marketplaces.
Theoretical Identification: The paper provides a formal derivation showing how TSPR identifies the global treatment effect under interference, utilizing position bias as the source of exogenous variation.
Robust Estimation: It introduces parametric and non-parametric estimators that correct for the bias introduced by partial treatment exposure and re-ranking distortions.
Placebo Buffering: The inclusion of a "Placebo" item group is a novel methodological contribution to prevent quality imbalances between treatment and control arms during re-ranking.

4. Results

The authors evaluated TSPR using semi-synthetic simulations calibrated on Expedia hotel search data (approx. 2 million records). They compared TSPR against two baselines: Bernoulli-randomized Item-level A/B tests and Cluster-randomized experiments.

Bias Reduction:
- Bernoulli Randomization: Suffered from severe downward bias (Mean $\hat{\Phi} = -0.1028$ vs. True $\Phi = -0.0691$ ) because it ignored within-list interference.
- TSPR: Significantly reduced bias. The non-parametric estimator achieved a bias of only $0.0069$.
Variance and Efficiency:
- Cluster Randomization: Achieved low bias but suffered from extremely high variance (Standard Deviation $\approx 0.1345$ ), making estimates unreliable.
- TSPR: Achieved a superior bias-variance tradeoff. The parametric TSPR estimator had the lowest Root Mean Squared Error (RMSE = 0.0210) and the lowest standard deviation (0.0162).
Performance Metrics:
- TSPR reduced the standard deviation of the estimate by approximately 8.3x compared to cluster randomization.
- TSPR reduced bias by roughly 60% compared to standard item-level A/B tests.

5. Significance and Implications

Operational Feasibility: TSPR can be implemented by adjusting ranking priorities within existing recommender systems, avoiding the legal and reputational risks of user-level price discrimination or catalog partitioning.
Methodological Shift: The paper demonstrates that platform structure (specifically, ranked attention) can be leveraged as a source of identifying variation even in the presence of complex interference, challenging the reliance on network modeling or cluster randomization.
Practical Application: The design is particularly valuable for high-stakes pricing experiments in e-commerce, travel, and ridesharing where substitution effects are strong and user trust (coherency) is paramount.
Limitations & Future Work: The design relies on strong position bias and slack supply (no binding capacity constraints). Future work could extend TSPR to supply-constrained environments or continuous treatments.

In conclusion, TSPR offers a pragmatic and statistically rigorous solution for a pervasive problem in marketplace experimentation, enabling platforms to make data-driven pricing decisions without compromising user experience or regulatory compliance.