🤖 machine learning

Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits

This paper proposes a novel Cross-Domain Off-Policy Evaluation and Learning framework that leverages historical datasets from both target and source domains to overcome critical challenges like few-shot data, deterministic logging policies, and new actions, thereby enabling effective policy evaluation and optimization in scenarios where existing methods fail due to high variance or limited exploration.

Original authors: Yuta Natsubori, Masataka Ushiku, Yuta Saito

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuta Natsubori, Masataka Ushiku, Yuta Saito

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a ship trying to navigate a stormy sea, but you've never actually sailed this specific route before. You have a map of your current ship's past voyages (the "logged data"), but that map is incomplete. Maybe your old captain was too cautious and only sailed in calm waters, leaving the dangerous reefs uncharted. Or perhaps, the map is missing entirely because the ship was too small to explore the whole ocean. In the world of computer science, specifically in a field called Machine Learning, this is a common problem known as Contextual Bandits. It's how computers learn to make decisions—like recommending a movie, suggesting a medication, or showing an ad—based on the situation (the "context") without actually trying out every single option in the real world, which could be expensive or risky.

To solve this, scientists use a technique called Off-Policy Evaluation (OPE). Think of OPE as a "flight simulator" for decision-making. Instead of testing a new strategy on real people (which is risky), you run it through a computer using old data to see how well it would have done. The problem is, most simulators break if the old data is too boring (the captain only did one thing) or if the new strategy wants to try something the old captain never did. If the old data has no record of a specific action, the simulator can't guess what would happen, leading to wild guesses or total failure. This paper tackles the tricky scenario where the old data is either too limited, too rigid, or missing crucial new options entirely.

Enter the researchers from Hakuhodo DY Holdings and Cornell University, who propose a clever new way to fix this broken simulator. They call their idea Cross-Domain Off-Policy Evaluation and Learning. Imagine you are trying to learn how to surf in a new, unknown ocean (the "target domain"), but your local beach (the "source domain") has a different wave pattern. If you only look at your local beach, you might get confused when you see a giant wave that never happened there. But, what if you could also look at a video of surfing in a neighboring ocean that has similar waves? Even if the water is slightly different, the physics of the wave might be the same.

The authors suggest that instead of being stuck with just the boring or incomplete data from your current situation, you can borrow insights from other, similar situations (other "domains"). They realized that while every hospital, country, or user group is unique, they often share underlying patterns. For example, a treatment might work similarly in two different hospitals, even if the patient demographics vary slightly. The paper introduces a new method called COPE (Cross-domain Off-Policy Evaluation). It works by splitting the "reward" (the good outcome) into two parts: a shared part that is common across similar groups (like the general physics of the wave) and a unique part that is specific to just your group (like the local wind).

By using data from other domains to figure out the "shared part," COPE can fill in the blanks for actions that were never tried in your specific data. It's like using a map from a neighboring city to figure out the layout of a street in your town that your own map forgot to draw. The researchers tested this on real-world data from a video-sharing app (KuaiRec) and found that when the target data was scarce, or when the old data was too rigid (deterministic), or when there were brand-new actions to try, their method was far more accurate than the old ways. They showed that by combining data from multiple sources but being careful to account for their differences, they could evaluate and learn new policies much more effectively, even in situations where previous methods simply gave up or made huge errors.

In their experiments, they simulated scenarios where the new actions made up to 80% of the possible choices, or where the old data was so limited that there was only one data point per action. In these tough cases, their new method reduced errors significantly compared to the standard approaches. They also showed that this works for "learning" (finding the best strategy), not just "evaluating" (checking a strategy). The paper suggests that by treating different data sources as a connected family rather than isolated islands, we can build smarter, safer, and more adaptable decision-making systems, especially when we don't have enough data to learn from scratch.

Technical Summary: Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits

1. Problem Formulation

The paper addresses the limitations of existing Off-Policy Evaluation (OPE) and Off-Policy Learning (OPL) methods in contextual bandits when facing three specific, challenging scenarios prevalent in real-world applications (e.g., personalized medicine, advertising, education):

Deterministic Logging Policies: The historical logging policy selects a specific action with probability one, leaving no stochastic exploration.
New Actions: Actions available to the new policy were never observed in the historical logged data.
Few-Shot Data: The target domain has extremely limited logged data.

Standard estimators like Inverse Propensity Scoring (IPS) and Doubly Robust (DR) rely on the Common Support assumption (Condition 2.1), which requires that any action taken by the new policy must have a non-zero probability of being taken by the logging policy. When this assumption is violated (due to determinism or new actions), these estimators suffer from severe bias because they lack reward information for unexplored actions. Furthermore, naive attempts to mitigate this by pooling data from multiple domains (e.g., IPS-ALL) often introduce substantial bias by ignoring differences in Data Generating Processes (DGPs) across domains.

To solve this, the authors propose a new problem formulation: Cross-Domain OPE/L. In this setup, the goal is to evaluate and optimize a policy in a target domain ( $T$ ) by leveraging not only the limited logged data from $T$ but also historical logged datasets from multiple source domains ( $S$ ). These source domains may have different context distributions, reward distributions, and logging policies, but they share the same action space.

2. Methodology

2.1 Reward Function Decomposition

The core of the proposed method is a decomposition of the expected reward function $q_k(x, a)$ for any domain $k$ into two components:
$q_k(x, a) = \underbrace{g(x, a, \phi(k))}_{\text{Domain-Cluster Effect}} + \underbrace{h(x, a, k)}_{\text{Domain-Specific Effect}}$

Domain-Cluster Effect ( $g$ ): A component common to domains within the same cluster $\phi(k)$ . This captures shared causal effects (e.g., the general efficacy of a treatment across similar hospitals).
Domain-Specific Effect ( $h$ ): A component unique to each specific domain $k$ , representing deviations from the cluster average.

2.2 The COPE Estimator

The authors propose Cross-domain Off-Policy Evaluation (COPE), an estimator that leverages this decomposition to handle support deficiency:

Estimating the Cluster Effect: For the domain-cluster effect $g$ , COPE applies Multiple Importance Weighting (MIW) using data from all domains within the target cluster $\phi(T)$ . This allows the estimator to utilize exploration data from source domains to evaluate actions that are unexplored in the target domain, provided they are explored within the cluster.
Estimating the Specific Effect: For the domain-specific effect $h$ , COPE uses Reward Regression ( $\hat{q}_T$ ) trained exclusively on the target domain's logged data. This corrects for the unique characteristics of the target domain.

The final COPE estimator combines these two parts:
$\hat{V}_{COPE}(\pi; D_{\phi(T)}) = \underbrace{\frac{1}{n_{\phi(T)}} \sum_{k \in \phi(T)} \sum_{i=1}^{n_k} \frac{\pi(a_i^k|x_i^k)}{p_{\phi(T)}(a_i^k|x_i^k)}(r_i^k - \hat{q}_T(x_i^k, a_i^k))}_{\text{MIW on Cluster Data}} + \underbrace{\frac{1}{n_T} \sum_{i=1}^{n_T} \sum_{a \in A} \pi(a|x_i^T)\hat{q}_T(x_i^T, a)}_{\text{Regression on Target Data}}$
Here, $p_{\phi(T)}(a|x)$ is the joint distribution of context and action averaged over the target cluster, serving as the denominator for the importance weights.

2.3 Extension to Policy Learning (COPE-PG)

The framework is extended to Off-Policy Learning (OPL) by deriving a policy gradient estimator (COPE-PG). This estimator uses the same decomposition to update policy parameters $\theta$ , enabling the learning of policies that select new actions or handle deterministic logging scenarios by leveraging source domain gradients.

2.4 Theoretical Properties

Relaxed Support Condition: The method relies on Common Cluster Support (Condition 3.1), which is satisfied if the logging policy has common support within the cluster of domains, not necessarily the target domain alone. This allows evaluation even when the target domain's logging policy is deterministic or new actions exist, as long as the cluster contains exploration data.
Bias Analysis: The bias of COPE is characterized by the difference between the weighted average of regression errors in the cluster and the error in the target domain. Under the Conditional Pairwise Correctness (CPC) condition (where the regression model accurately preserves relative reward differences between domains in the cluster and the target), COPE is proven to be unbiased.
Generalization: COPE is shown to be a strict generalization of existing estimators. If the cluster size is 1 (only the target domain), it reduces to standard DR. If the cluster size includes all domains, it approaches DR-ALL (though with bias adjustments).

3. Key Contributions

Problem Formulation: Introduction of Cross-Domain OPE/L, a novel setup that utilizes source domain data to solve OPE/L challenges (deterministic logging, new actions, few-shot data) in the target domain.
New Estimator (COPE): A novel estimator based on reward function decomposition (cluster vs. specific effects) and multiple importance weighting. It theoretically achieves unbiasedness under milder conditions than standard IPS/DR.
Policy Gradient Extension: Development of COPE-PG, enabling policy learning in scenarios where traditional gradient estimators fail due to lack of exploration.
Empirical Validation: Comprehensive experiments on a real-world recommendation dataset (KuaiRec) and synthetic data demonstrating superior performance in Mean Squared Error (MSE), bias reduction, and policy value optimization compared to baselines (IPS, DR, DM, and their "ALL" domain variants).

4. Experimental Results

The authors evaluated COPE and COPE-PG against baselines (IPS(T), DR(T), DM(T), IPS-ALL, DR-ALL, DM-ALL) under varying conditions:

New Actions: As the ratio of new actions in the target domain increased, COPE maintained low bias and MSE, while target-only methods (IPS(T), DR(T)) suffered from high bias, and naive pooling methods (IPS-ALL) failed due to DGP mismatches.
Deterministic Logging: When a large portion of users in the target domain had deterministic logging policies, COPE remained robust with low bias, whereas standard methods exhibited rapidly increasing bias.
Few-Shot Data: In low-data regimes (e.g., 1 data point per action), COPE significantly outperformed the best baseline (DM), achieving a 6.79x reduction in MSE in one scenario.
OPL Performance: COPE-PG successfully learned policies that selected high-reward new actions, a capability absent in target-only methods. It maintained high policy values even as the degree of deterministic logging increased.
Hyperparameter Sensitivity: The method showed robustness to the size of the target cluster $|\phi(T)|$ , performing well across a range of cluster sizes, though moderate sizes (e.g., 6-8) yielded optimal results.

5. Significance and Claims

The paper claims that existing OPE/L methods are fundamentally limited by the common support assumption, rendering them ineffective in realistic scenarios involving deterministic logging, new actions, or sparse data. By formulating the problem as Cross-Domain OPE/L, the authors provide a practical solution that leverages the abundance of historical data available in related domains (e.g., other hospitals, countries, or user segments).

The significance of this work lies in its ability to:

Enable secure evaluation and learning of new policies in high-stakes environments where online A/B testing is risky or impossible.
Overcome the "variance vs. bias" trade-off in few-shot settings by transferring exploration information from source domains without introducing the bias associated with naive data pooling.
Offer a theoretically grounded approach (via reward decomposition and MIW) that analytically guarantees unbiasedness under conditions where standard methods fail.

The authors conclude that while their current domain clustering relies on heuristics (empirical average rewards), the framework opens a new direction for research into principled cross-domain transfer in bandit problems, specifically targeting the unsolved issues of deterministic logging and new actions.