🤖 machine learning

Personalizing Marketplace Policies with Competing Objectives and Constrained Experiments: Evidence from a Job Marketplace

This paper presents an integrated framework for personalizing job marketplace policies that successfully balances competing employer and job seeker objectives under constrained experimental conditions by combining ensemble-based hybrid ranking models with treatment effect extrapolation, resulting in significant metric improvements while maintaining engagement guardrails.

Original authors: Yufei Wu, Zhen Yan

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yufei Wu, Zhen Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling digital marketplace, like a giant job board, where two very different groups meet: Employers (who want to hire) and Job Seekers (who want to work).

The platform's goal is to help both sides, but they often have conflicting interests. If the platform is too generous with free services, employers might stop paying for premium features, hurting the company's ability to run the site. If the platform is too strict, job seekers might get frustrated because they can't see good jobs, and they might leave.

The authors of this paper faced a tricky problem: How do you set the rules (policies) for this marketplace so that everyone is happy, without breaking the bank?

Here is a simple breakdown of their solution, using everyday analogies.

1. The Problem: One Size Doesn't Fit All

In the past, platforms used a "one-size-fits-all" rule. For example, "Every job posting gets 3 free views, then you must pay."

The Issue: Some jobs are like gold mines (high demand, easy to fill), while others are like selling ice in winter (low demand). Treating them the same is inefficient. You might be charging for a job that no one wants, or giving away free views for a job that would have paid anyway.

2. The Two Big Hurdles

The team wanted to personalize these rules (give different rules to different jobs), but they hit two major roadblocks:

Hurdle A: The "See-Saw" Effect (Cross-Side Externalities)
If you push too hard to get employers to pay (the "Target"), you might accidentally annoy the job seekers (the "Guardrail"). It's like a see-saw: if you push one side down too hard, the other flies up. The challenge is that different jobs react differently. Some jobs can be monetized without annoying seekers; others cannot. You need a strategy that balances both sides simultaneously.
Hurdle B: The "Blindfolded Chef" (Constrained Experiments)
To test new rules, you usually need to run experiments. But in a marketplace, you can't just test a rule on one person; you have to test it on a whole group (a "cluster") to avoid chaos. This means they could only test two specific settings: "Strict" and "Lenient."
- The Problem: They wanted to find the perfect middle ground (a continuous range of options), but they only had data for two extreme points. It's like a chef who only knows how to cook food at "Burnt" or "Raw" and needs to figure out the perfect "Medium-Rare" temperature without being able to test the middle.

3. The Solution: A Three-Part Toolkit

The authors built a smart system to solve these problems using three main tricks:

Trick 1: The "Double-Check" Scorecard (Hybrid Ranking)

Instead of just looking at one score (e.g., "How much money will this make?"), they built a scorecard that looks at two things separately:

Money Potential: How much revenue will this job generate?
Risk Factor: How likely is this to annoy job seekers?

They created a Hybrid Ranking system:

Top Tier: Jobs that make money and don't annoy seekers. These get strict rules (pay up!).
Bottom Tier: Jobs that don't make much money but are very risky for seekers. These get relaxed rules (keep them free to keep seekers happy).
The Result: This approach reduced the risk of annoying job seekers by over 10% compared to just chasing money, without losing any revenue.

Trick 2: The "Straight-Line" Guess (Extrapolation)

Since they could only test "Strict" and "Lenient," they needed to guess what would happen in the middle.

The Analogy: Imagine you know a car goes 0 mph at a stop sign and 60 mph at a green light. You assume that halfway between the signs, the car is going 30 mph.
The Science: They assumed the relationship between the policy and the result was a straight line. They tested this assumption and found that even if the line wasn't perfectly straight, their guesses were still accurate enough to make good decisions. They didn't need to know the exact speed; they just needed to know which cars were faster than others.

Trick 3: The "Group Leader" Strategy (Ensemble Modeling)

Some employers post one job; others post hundreds. If you change the rule for one job, it might affect how the employer behaves with their other jobs.

The Fix: They built a model that looked at the job and the employer together. It's like a teacher who knows that if they give a hard test to one student, it might stress out the whole study group. They combined these perspectives to get a more accurate prediction.

4. The Result: Real-World Success

They put this system into action on LinkedIn, serving millions of job postings.

Did it work? Yes. The system successfully increased revenue (the target) significantly.
Did it break the rules? No. It stayed strictly within the safety limits for job seeker engagement (the guardrail).
Was the guess right? Yes. When they tested the "middle ground" policies they had only guessed at, the real-world results matched their predictions almost perfectly.

Summary

The paper shows that even when you are blindfolded (limited experiments) and walking a tightrope (balancing conflicting goals), you can still find the perfect path. By using a smart ranking system that respects both sides of the marketplace and making educated, validated guesses about the middle ground, they created a personalized policy that makes money while keeping users happy.

Technical Summary: Personalizing Marketplace Policies with Competing Objectives and Constrained Experiments

1. Problem Statement

The paper addresses the challenge of personalizing policies in two-sided marketplaces, specifically a job platform connecting employers and job seekers. The core tension lies in cross-side externalities: optimizing metrics for one side (e.g., employer monetization via paid promotions) often degrades the experience for the other (e.g., job seeker engagement).

The authors focus on free-value thresholds, a policy parameter determining the scope of complimentary services for job listings before paid promotion is required. Setting these thresholds uniformly is suboptimal due to heterogeneity in job segments. However, standard uplift modeling faces two critical barriers in this setting:

Multi-Objective Optimization: Policies affect multiple stakeholder groups with heterogeneous responses. Maximizing employer conversions risks harming seeker engagement, requiring a balance that single-objective uplift models cannot capture.
Constrained Experiments (Positivity Violation): To avoid interference (bias) caused by marketplace competition, experiments must use cluster-level randomization (e.g., randomizing by employer or job segment) rather than individual-level randomization. This limits the number of available treatment levels to a few discrete values. Consequently, standard continuous dose-response methods (which assume dense support across the treatment space) cannot be applied, creating a "positivity violation."

2. Methodology

The authors propose an integrated framework comprising three components to solve these simultaneous challenges:

A. Ensemble-Based Multi-Objective Hybrid Ranking
Instead of a single objective function or complex Pareto frontier navigation, the authors employ a hybrid ranking strategy:

Separate Modeling: They estimate Conditional Average Treatment Effects (CATE) separately for the target metric (employer conversions) and the guardrail metric (seeker engagement) using an X-learner meta-learner.
Handling Spillovers: To address the aggregation challenge where outcomes are measured at the employer level but targeting occurs at the job-segment level, they ensemble two CATE estimates: one at the job level (for single-job employers) and one at the employer level (average outcome across all postings). The final estimate is a weighted sum based on segment composition.
Ranking Logic: Jobs are ranked in two halves. The top 50% are ranked by target uplift (descending). The bottom 50% are ranked by guardrail impact (ascending, prioritizing low risk).
Policy Assignment: Policies are assigned based on these tiers (e.g., restrictive policies for high-uplift/low-risk segments; relaxed policies for high-risk segments).
Constraint Satisfaction: The system iteratively relaxes policies for lower-ranked segments until the aggregate guardrail constraint ( $\sum \tau_V \geq -\delta$ ) is satisfied. This avoids the need for contested weight specifications between teams.

B. Treatment Effect Extrapolation
Since experiments only provide two discrete treatment levels (baseline $c_0$ and treatment $c_1$ ), the framework must estimate effects at untested policy levels.

Linear Extrapolation: The authors assume a linear dose-response relationship: $\hat{\tau}(X, c) = \hat{\tau}(X, c_1) \cdot \frac{c - c_0}{c_1 - c_0}$ .
Justification: This is treated as an identification constraint imposed by the experimental design (two levels) rather than a modeling preference. The authors argue that for the specific goal of ordinal ranking (which drives policy assignment) rather than precise cardinal magnitude estimation, linearity is sufficient within a bounded range.
Validation: The validity of this approach is tested via offline sensitivity analysis and post-deployment validation.

C. Production Deployment Architecture
The system was deployed using segment-based policy lookup tables rather than real-time model inference. This ensures consistent treatment for similar jobs, reduces latency to sub-seconds, and simplifies auditing.

3. Key Contributions

Hybrid Ranking Framework: A method to optimize for a target metric while explicitly constraining a guardrail metric without requiring contested weights. It achieves over 10% lower guardrail risk for equivalent target gains compared to single-objective optimization.
Extrapolation under Positivity Violation: A principled approach to extend CATE estimates from limited discrete experimental levels to continuous policy values, validated empirically to be robust for ordinal ranking decisions within bounded ranges.
Production Validation: Evidence from a live system serving millions of job postings, confirming that extrapolated predictions align with observed effects and that guardrail constraints are met.

4. Results

Offline Validation: The CATE models showed strong calibration (correlation of 0.95 for target, 0.86 for guardrail at the decile level). The hybrid ranking improved the guardrail Qini coefficient by 91% (from 0.11 to 0.21) while maintaining target Qini, demonstrating that single-objective optimization fails to capture engagement risks.
Sensitivity Analysis: Under alternative curvature assumptions (20% concave or convex), the estimated policy impact varied by only 8–9%, confirming that the policy recommendations are robust to deviations from linearity.
Post-Launch Performance:
- Target Lift: Observed lift was statistically significant and within 10% of offline predictions.
- Guardrail Compliance: All engagement metrics remained within the defined safety thresholds.
- Extrapolation Accuracy: During a gradual ramp-up, observed treatment effects at intermediate policy levels correlated highly ( $r=0.85$ ) with linear predictions, validating the extrapolation assumption.

5. Significance and Claims

The paper claims to demonstrate that meaningful personalization is achievable in marketplace settings characterized by severe experimental constraints (cluster randomization) and competing objectives, conditions common in real-world platforms but often lacking established solutions.

Key takeaways emphasized by the authors include:

Ordinal vs. Cardinal: Personalization does not require precise cardinal effect estimates if policy decisions rely on ordinal rankings; this allows linear extrapolation to succeed even with sparse treatment variation.
Constrained Optimization: Multi-objective tradeoffs can be managed through explicit guardrail constraints rather than negotiating complex weight parameters, aligning better with organizational structures where different teams own different metrics.
Ensemble Necessity: In marketplaces where targeting and outcome aggregation occur at different levels (e.g., job segment vs. employer), ensemble methods are necessary to correctly capture spillover effects.

The authors remain modest regarding the generalizability of the linear assumption, noting it is valid within the deployment range (~50% of observed variation) but may break down at policy extremes. They identify future work in richer experimental designs (multi-arm holdouts) and online learning (contextual bandits) to relax these constraints.