📊 statistics

Logging Policy Design for Off-Policy Evaluation

This paper proposes a unifying framework for designing logging policies that minimize off-policy evaluation error by characterizing a fundamental reward-coverage tradeoff and deriving optimal strategies across various informational regimes to guide firms in selecting treatment policies for high-stakes experimentation.

Original authors: Connor Douglas, Joel Persson, Foster Provost

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Connor Douglas, Joel Persson, Foster Provost

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to decide if a new recipe (the Target Policy) is better than your current one. You can't just cook it for everyone tomorrow because if it's bad, people will be unhappy. So, you want to test it "offline" using a notebook of past meals (the Log Data) to predict how good it would be.

The problem is: How did you write down those past meals?

If your old notebook only recorded what people ate when you were flipping a coin to decide the menu (a Uniform Logging Policy), your data is messy. You have thousands of records of people eating things they hated, and very few records of the good stuff. Trying to guess how the new recipe would do with this messy data is like trying to predict the weather by looking at a single, blurry photo.

This paper is about designing a better way to write that notebook so you can accurately judge the new recipe without actually serving it to everyone yet.

The Core Problem: The "Coverage vs. Reward" Tug-of-War

The authors explain that designing this notebook (the Logging Policy) involves a tricky balancing act between two goals:

Coverage (The Safety Net): You need to record enough variety. If your new recipe suggests a dish you never wrote down in the past, you can't evaluate it. You need to make sure you have data on the things the new recipe might suggest.
Reward (The Good Stuff): You want to record the meals that people actually enjoyed. If you only write down the boring, safe meals, your data is boring. If you only write down the rare, amazing meals, you might miss the context of what people usually eat.

The Analogy: Imagine you are a scout trying to find the best hidden treasure spots (high rewards) for a future explorer (the target policy).

If you only look in the spots you think have treasure, you might miss a spot the explorer decides to visit.
If you look everywhere randomly, you waste time digging in empty holes, and your map of the "good" spots is very shaky because you didn't dig deep enough in the right places.

The paper's main discovery is that the best notebook isn't just the explorer's plan, and it isn't random chaos. It's a specific, calculated mix that leans toward the good spots but still keeps an eye on the spots the explorer might choose.

The Three Scenarios of Knowledge

The paper breaks down how to design this perfect notebook based on what you know before you start writing:

1. The "Blind" Scenario (We know nothing)
If you have no idea what the new recipe is or what people like, the safest bet is to write down everything equally (Random Sampling). It's not efficient, but it's the only way to guarantee you don't miss anything completely.

2. The "Crystal Ball" Scenario (We know everything)
If you know exactly what the new recipe is and exactly what people like, you don't just follow the new recipe. You actually do something smarter:

You focus heavily on the items the new recipe likes.
But, you boost the probability of recording items that are very likely to be enjoyed, even if the new recipe doesn't pick them as often.
The Magic Result: The paper proves that if you use this "super-smart" notebook, you can actually get a more accurate prediction of the new recipe's success than if you had just served the new recipe live to people. Plus, while writing the notebook, people are actually happier because you're serving them better food than the new recipe would have!

3. The "Fuzzy Crystal Ball" Scenario (We have guesses)
In the real world, you usually have a guess (a machine learning model) about what people like, but it's noisy.

The Trap: If you use your noisy guess directly to decide what to write down, you might get it wrong and waste your time.
The Fix: The authors suggest a technique called "Posterior Shrinkage." Think of this as a "safety filter." If your guess says a dish is amazing, but you aren't 100% sure, you pull that guess back toward the average. It's like saying, "This looks great, but let's not bet the farm on it yet." This simple adjustment makes your notebook much more reliable.

Practical Advice: The "Soft-Greedy" Approach

The paper admits that in the real world, companies can't always build the perfect, complex mathematical notebook. They have constraints.

So, they suggest a simpler, practical approach called "Soft-Greedy."
Instead of a perfect calculation, imagine a dial you can turn:

Turn it all the way to "Random": You write down everything equally. (Safe, but inaccurate).
Turn it all the way to "Greedy": You only write down the absolute best items you know of. (Efficient, but risky if you miss something).
Turn it to the "Sweet Spot": You mostly write down the good stuff, but you leave a little room for other things.

The paper shows that by tuning this dial correctly (based on how much data you have), you can get results almost as good as the perfect mathematical solution, without needing a supercomputer to figure it out.

Summary

This paper teaches us that how we collect data matters just as much as the data itself. By carefully designing what we choose to record (the logging policy), balancing the need to see the "good stuff" with the need to see "what the new plan might do," we can make much better decisions about new strategies without the risk of trying them out live. It turns the messy process of experimentation into a precise science.

Technical Summary: Logging Policy Design for Off-Policy Evaluation

Problem Statement

Off-policy evaluation (OPE) is a critical mechanism for estimating the value of a target treatment policy (e.g., a new recommender system) using data collected by a different "logging" policy. While OPE enables high-stakes experimentation without live deployment, its accuracy is heavily dependent on the logging policy used to generate the data. Existing literature often treats the logging policy as a fixed, potentially unknown behavior policy, focusing instead on improving estimators (e.g., Inverse Propensity Weighting, Doubly Robust). However, in practice, the choice of logging policy dramatically affects the bias and variance of OPE estimates, particularly in finite-sample regimes with large action spaces and sparse rewards.

The core problem addressed is how to design the logging policy (specifically, the action propensities $\pi_l(a|x)$ ) to minimize the Mean Squared Error (MSE) of an Inverse Propensity Weighting (IPW) estimate for a given target policy $\pi_t$ . This design must navigate a fundamental reward–coverage tradeoff: concentrating probability mass on high-reward actions reduces variance from reward realization, while ensuring coverage across actions the target policy might select is necessary to control bias and variance arising from action sampling.

Methodology

The authors frame the problem within a contextual bandit setting (Markov Decision Process without transition dynamics) where contexts $X$ arrive from a distribution $p_X$ , and actions $A$ are selected by a logging policy $\pi_l$ . The reward $R$ is binary, drawn from $Bernoulli(\mu(a, x))$ .

1. Objective Function

The primary objective is to minimize the MSE of the IPW estimator $\hat{V}_{IPW}(\pi_t | \pi_l)$ :
$MSE = E[(\hat{V}_{IPW} - V(\pi_t))^2] = Bias^2 + Variance$
The authors derive the bias-variance decomposition explicitly as a function of the logging policy $\pi_l$ . They establish that while overlap (ensuring $\pi_l(a|x) > 0$ for all $a$ where $\pi_t(a|x) > 0$ ) eliminates bias, it does not necessarily minimize MSE, especially when sample sizes are small.

2. Informational Regimes

The paper analyzes optimal logging policies across a spectrum of informational constraints, categorized by knowledge of the target policy ( $\pi_t$ ) and the reward distribution ( $\mu$ ):

Unknown $\pi_t$ and Unknown $\mu$ : In the absence of information, the minimax optimal strategy is uniform randomization. This aligns with the intuition that an uninformative prior is optimal when the estimand is unknown.
Known $\mu$ , Unknown $\pi_t$ : The optimal policy allocates mass proportional to $\frac{\mu(a,x)}{c + \mu(a,x)^2}$ , prioritizing high-reward actions while maintaining a safety margin.
Unknown $\mu$ , Known $\pi_t$ : The optimal strategy is to log data directly under the target policy $\pi_t$ itself.
Known $\pi_t$ and Known $\mu$ : This is the ideal case. The optimal logging policy follows a Neyman allocation directed toward the target policy:
$\pi_l^*(a|x) \propto \pi_t(a|x)\sqrt{\mu(a,x)}$
This solution weights actions by the product of the target policy's mass and the square root of the reward probability.

3. Intermediate Settings and Practical Constraints

Recognizing that perfect information is rarely available, the paper extends the framework to intermediate settings:

Distribution over Target Policies: When the exact target is unknown but a distribution $Pr(\pi_t)$ is known, the optimal logging policy targets a "second-moment pseudo-target" $\tilde{\pi}_t(a|x) = \sqrt{E[\pi_t(a|x)^2]}$ .
Noisy Reward Estimates: When $\mu$ is estimated via a model $\hat{\mu}$ with noise, directly plugging in $\hat{\mu}$ inflates variance. The authors propose posterior shrinkage, shrinking noisy estimates toward a per-context mean. Under a Gaussian hierarchical prior, the optimal shrinkage weight is derived, leading to an "empirical Bayes" interpretation where the shrinkage parameter is estimated from held-out data.
Soft-Greedy Classes: For settings where theoretical optima are infeasible due to engineering or organizational constraints, the paper proposes three policy classes indexed by a "greediness" parameter: Top-k, Softmax, and Power-normalized. These classes interpolate between uniform randomization and deterministic greedy selection, allowing firms to balance the reward-coverage tradeoff via a single hyperparameter.

Key Results

Optimality of Modified Logging: In the case where both $\pi_t$ and $\mu$ are known, the optimal logging policy $\pi_l^*$ (Neyman allocation) yields a lower MSE for the IPW estimate than running the target policy $\pi_t$ directly (on-policy evaluation). Furthermore, $\pi_l^*$ accrues a higher expected reward during the logging period than $\pi_t$ itself. This demonstrates that optimized OPE can simultaneously improve statistical precision and user experience compared to live A/B testing.
Reward-Coverage Tradeoff: The paper formalizes that optimal logging is rarely identical to the target policy. Even if the target policy is known, the logging policy must balance sampling high-reward items (to reduce reward variance) against sampling items the target policy selects (to reduce sampling variance).
Impact of Noise: Using noisy reward predictions without shrinkage leads to suboptimal logging policies that can perform worse than uniform randomization. Posterior shrinkage significantly mitigates this risk, often allowing optimized logging to outperform on-policy evaluation even with imperfect reward models.
Performance of Heuristics: Simulations show that simple "soft-greedy" policies (specifically Top-k and Power-normalized) can approach the MSE of the theoretical optimum. In small-sample regimes, intentionally truncating the action support (inducing slight bias) can reduce overall MSE compared to policies enforcing strict overlap.

Significance and Contributions

The paper claims to provide a unifying framework for logging policy design, moving beyond ad-hoc heuristics or safety-only constraints. Its significance lies in:

Theoretical Grounding: It connects logging policy design to classical problems in optimal experimental design (Neyman allocation) and importance sampling, providing tractable optimal solutions under various informational constraints.
Actionable Guidance: It offers concrete strategies for firms choosing among candidate systems, particularly when the target policy is unknown or reward estimates are noisy. The proposed shrinkage methods and soft-greedy classes provide practical tools for implementation.
Dual Benefit: It challenges the notion that data collection for evaluation must sacrifice user experience. The results show that theoretically optimal logging policies can yield both more precise offline evaluations and higher immediate rewards than the policies being evaluated.
Clarification of Tradeoffs: By explicitly characterizing the reward-coverage tradeoff and the bias-variance decomposition in the context of logging design, the paper clarifies when and why different logging strategies (e.g., uniform vs. greedy vs. Neyman) are appropriate.

The authors conclude that while their theoretical results rely on assumptions (such as the use of IPW and weak overlap), they provide a principled basis for firms to improve offline experimentation efforts, replacing conservative fallbacks like uniform randomization with data-driven, theoretically optimized logging strategies.