Technical Summary: When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation?

Problem Statement

Organizations routinely face budget-constrained allocation problems, such as selecting the top $k$ fraction of customers for a retention offer or patients for a treatment, based on estimated treatment effects. Before deploying a new deterministic "top-$k$" policy ($\pi_e$), practitioners rely on Off-Policy Evaluation (OPE) using logged data from a previous policy ($\pi_b$) to estimate the policy's value.

The core difficulty arises from the deterministic nature of the target policy. Unlike stochastic policies, a deterministic top-$k$ rule assigns probability 1 to a single action and 0 to others. This eliminates averaging over actions, causing importance weights ($w = \pi_e(a|x)/\pi_b(a|x)$) to become extreme (0 or $1/\pi_b$) wherever the logger rarely took the action the target demands. While standard OPE literature addresses weak overlap in stochastic settings, this paper investigates whether offline evaluation remains trustworthy when the target is a deterministic top-$k$ policy, specifically under bounded weights (propensities floored at 0.02).

Methodology

The author constructs a controlled, reproducible benchmark to evaluate six estimators (DM, IPS, SNIPS, DR, Switch-DR, mIPS) across five datasets and two known-effect hardening sweeps.

• Datasets: The study utilizes five datasets spanning three regimes:
  1. Synthetic: Known potential outcomes ($\mu_0, \mu_1$) with tunable effects, providing exact ground truth.
  2. Semi-synthetic: IHDP (continuous outcome, known effects) and Jobs (binary outcome, randomized subset).
  3. Real RCTs: Hillstrom and Lenta marketing experiments with known constant treatment probabilities.
  4. Hardening Layer: Six ACIC-2017-style Data Generating Processes (DGPs) over real IHDP covariates to test robustness across response surfaces.
• Logging Design: Logged feedback is synthesized via rejection sampling from a randomized trial or a known propensity. The logging policy is a temperature-scaled softmax over a candidate score. Crucially, the paper varies the alignment between the logger and the target:
  • Self-aligned: Logger built from the target's own score (but not its specific top-$k$ boundary).
  • Misaligned: Logger built from a different candidate's score.
  • Independent: Logger built from a candidate-independent score.
• Evaluation Protocol: Every estimate is paired with an explicit reference value. For synthetic/IHDP data, this is the exact ground truth. For randomized trials, it is a noisy Horvitz–Thompson estimate. The study explicitly separates these reference types to avoid conflating estimator error with reference noise.
• Key Variables: The study sweeps over logging temperature ($\tau$), budget ($k$), and candidate policies (T-learner, S-learner, random baseline). It also tests the impact of estimated propensities ($\hat{\pi}_b$) versus exact propensities ($\pi_b$) and outcome model quality ($\hat{\mu}$).

Key Contributions

1. Overlap is Governed by Action Alignment, Not Logging Sharpness

The paper corrects a common misconception that "sharpening" a logging policy (lowering temperature $\tau$) inherently improves overlap for deterministic targets.

• Finding: For a deterministic top-$k$ target, overlap is determined by the logger's probability of the specific actions the target selects, not the sharpness of the score distribution.
• Mechanism: If a logger is "score-aligned" (centered on the mean score) but the target cuts off at a budget threshold, sharpening the logger merely concentrates mass away from the target's actions, collapsing the Effective Sample Size (ESS). Conversely, an "action-aligned" logger (centered on the cutoff) sharpens toward full support.
• Diagnostic: ESS and support deficiency can rank risk across different logging environments (e.g., choosing between logs) but are weak at ranking candidates within a single fixed log.

2. The Optimizer's Curse and Splitting Strategies

The paper addresses the "optimizer's curse" where a policy learned on the same data used for evaluation is optimistically biased.

• Finding: Cross-fitting only the outcome nuisance model (while keeping the policy frozen) does not remove reuse bias; in fact, it often makes the optimism worse compared to plain Doubly Robust (DR) estimation.
• Reasoning: The optimizer selects units with inflated scores due to estimation noise. An honest (out-of-fold) nuisance model fails to shrink these residuals, re-introducing the optimism.
• Solution: "Honest" policy-level splitting (re-learning the policy on out-of-fold data and evaluating on held-out folds) is required to eliminate reuse bias. This targets the value of the learning procedure rather than the specific full-sample policy, changing the estimand but providing a valid evaluation of the deployment process.

3. Propensity Estimation Error is the Dominant Failure Mode

Replacing exact propensities with estimated propensities ($\hat{\pi}_b$) causes the largest degradation in performance.

• Finding: For Inverse Propensity Scoring (IPS), estimation error increases failure rates (relative bias > 10%) from ~6% to 37–63%.
• Inversion of Diagnostics: A poor propensity model can invert the ESS diagnostic. If the estimated propensity is nearly constant, it carries no dispersion information, causing the estimated ESS to collapse to coverage. This leads to high ESS scores in regions where true propensity is extreme (high risk), making the diagnostic actively misleading.
• Robustness: Doubly Robust (DR) estimators remain stable under propensity estimation error because the outcome model term absorbs the error.

Results

• Estimator Accuracy (RQ1): Model-based estimators (DM, DR) generally outperform pure weighting estimators (IPS, SNIPS) on exact-value datasets, with DR showing the most stability. On datasets with noisy references (RCTs), the performance gap compresses, and rankings become unreliable due to reference noise.
• Overlap Diagnostics (RQ2): The risk of weak overlap is rankable across logging environments using ESS, provided a credible propensity model exists. However, within a single log, ESS cannot reliably rank which candidate policy is safe to deploy.
• Policy Selection (RQ4): When selecting among candidates, using a "common log" (one historical log for all candidates) is essential. Using candidate-specific logs (where each candidate is evaluated on a log aligned with itself) confounds the selection by comparing policy-logger pairs rather than policies. IPS tends to over-select candidates that are easiest to evaluate (lowest variance), not necessarily the best.
• Non-Simulated Check: A check on the "Twins" dataset (real paired outcomes) confirmed that the mechanisms (alignment effects, cross-fitting signs) replicate, though calibrated cut points do not transfer.

Significance and Claims

The paper claims to provide an operational guide for practitioners, emphasizing that offline evaluation of deterministic top-$k$ policies is highly sensitive to the alignment between the logger and the target.

• Practical Guidance:
  1. Check Alignment: Before trusting an importance-weighted estimate, verify if the logger's policy broadly agrees with the target's actions.
  2. Use DR under Uncertainty: Doubly Robust estimation is the safest default when outcome model adequacy is uncertain or when propensity estimation is required.
  3. Honest Splitting: To avoid the optimizer's curse, use policy-level splitting rather than just nuisance cross-fitting.
  4. Shared Logs for Selection: When comparing multiple candidate policies, evaluate them all on a single, shared historical log to avoid selection bias.
• Limitations: The study is limited to equal-cost, binary-action top-$k$ allocation. The findings on cut points are specific to the benchmark's data distributions and do not transfer directly to other domains without recalibration. The paper explicitly states it does not solve the problem of estimating propensities from observational data but highlights the severe cost of doing so poorly.

The paper concludes that while OPE is a necessary tool for budget-constrained allocation, its reliability is conditional on specific design choices (logger alignment, propensity modeling, and splitting strategies) that are often overlooked in standard practice.