📊 statistics

CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

The paper introduces CANDOR, a family of doubly robust off-policy evaluation estimators that strategically incorporates imperfect expert-annotated counterfactuals solely into the direct method component to achieve superior robustness and performance in healthcare decision-making scenarios.

Original authors: Aishwarya Mandyam, Shengpu Tang, Jiayu Yao, Jenna Wiens, Barbara E. Engelhardt

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aishwarya Mandyam, Shengpu Tang, Jiayu Yao, Jenna Wiens, Barbara E. Engelhardt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Testing New Rules Without Breaking Things

Imagine you are a doctor trying to decide if a new treatment plan (like a new way to give potassium to patients) is better than the old one. You can't just try it on real patients immediately; if the new plan is bad, it could hurt them.

So, you want to run a "simulation" using old patient records to see how the new plan would have performed. This is called Off-Policy Evaluation (OPE).

The Catch: Old records only show what happened when doctors followed the old rules. If the new plan suggests a treatment that was never given in the past (e.g., a specific high dose of potassium), the old records have no data on it. It's like trying to predict how a car handles a new type of road when you've only ever driven on highways. You are stuck because you lack data for those specific scenarios.

The Proposed Fix: Asking Experts for "What Ifs"

To fix the missing data, researchers tried asking human experts (or AI) to look at old patient records and say, "If we had given this patient a different treatment, here is what we think would have happened." These are called Counterfactual Annotations.

Think of this as filling in the blanks in a story. If the original story says, "The patient took Drug A," an expert might add a note: "If they had taken Drug B, they probably would have felt better."

The Problem: Experts aren't perfect. They might guess wrong, or they might be biased. If you blindly trust these guesses, your simulation could end up being less accurate than if you had just ignored the guesses entirely.

The Solution: The "CANDOR" Strategy

The authors propose a new family of methods to use these imperfect expert guesses safely. They built their method on a concept called Doubly Robust (DR) estimation.

To understand this, imagine you are trying to guess the final score of a sports game. You have two ways to guess:

The Statistician (Direct Method): You look at the teams' past stats and build a model to predict the score.
The Bookie (Importance Sampling): You look at the actual game results and adjust them based on how likely the teams were to play that way.

A Doubly Robust method combines both. It says: "I will use the Statistician's prediction, but if that prediction is wrong, I will correct it using the Bookie's data." The magic is that the method works well if either the Statistician is right or the Bookie's data is right. You don't need both to be perfect.

The Three Ways to Mix in the "Expert Guesses"

The paper tests three different ways to add those imperfect expert guesses into this Doubly Robust formula:

Method A (DM-IS+): Use the expert guesses to help the "Bookie" (the data adjustment part) but keep the "Statistician" (the prediction model) trained only on real, observed data.
Method B (DM+-IS+): Use the expert guesses to help both the Bookie and the Statistician.
Method C (DM+-IS): Use the expert guesses to help the Statistician (the prediction model) but keep the "Bookie" part strictly based on real, observed data.

The Big Discovery

The authors ran experiments using simulated medical data and real hospital records (MIMIC-IV). They found that Method C (DM+-IS) was the clear winner.

Here is why, using an analogy:
Imagine you are trying to navigate a city using a map (the Statistician) and a GPS that corrects your path based on traffic reports (the Bookie).

The expert guesses are like "crowd-sourced traffic reports" that might be wrong sometimes.
Method A and B let these potentially wrong crowd-sourced reports mess up the GPS correction. If the crowd is wrong, the GPS steers you off a cliff.
Method C uses the crowd-sourced reports only to improve the map. Even if the crowd is wrong about a specific street, the map just gets a little fuzzier, but the GPS (which relies on real traffic data) still knows how to steer you safely.

The Result:

When the expert guesses were perfect, all methods worked well.
When the expert guesses were imperfect (biased or noisy), the methods that mixed them into the "GPS" part (Method A and B) failed badly. Their estimates became worse than if they had ignored the experts entirely.
Method C (DM+-IS) remained robust. It could use the experts to fill in missing gaps in the map without letting the experts' mistakes ruin the final navigation.

Summary

The paper introduces CANDOR, a method that allows us to safely use imperfect expert guesses to evaluate new medical policies. It proves that the safest way to do this is to use the guesses only to train the prediction model (the "map"), while keeping the data correction mechanism (the "GPS") strictly based on real, observed facts. This ensures that even if the experts make mistakes, the final evaluation of the new policy remains reliable and safe.

Technical Summary: CANDOR – Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

Problem Statement

Off-policy evaluation (OPE) is essential for safely deploying contextual bandit algorithms in high-stakes domains like healthcare, where new treatment policies must be assessed using retrospective data before real-world implementation. However, standard OPE techniques are fundamentally limited by the coverage of the available behavior dataset. If the historical data lacks observations for certain state-action pairs (e.g., a new drug never prescribed), OPE methods cannot reliably estimate the value of a policy recommending those actions.

Recent approaches, such as the IS+ estimator, attempt to mitigate this by augmenting behavior datasets with expert-annotated counterfactual samples (predicted rewards for unobserved actions). However, these methods typically assume annotations are perfect. In practice, expert annotations (or those generated by Large Language Models) are often imperfect, containing bias or noise. Naively incorporating such imperfect annotations into existing estimators can degrade performance, sometimes yielding worse results than ignoring the annotations entirely. The core challenge addressed by this work is determining how to integrate imperfect counterfactual annotations into OPE estimators without compounding errors, particularly in the presence of misspecified reward models.

Methodology

The authors propose a family of OPE estimators grounded in the Doubly Robust (DR) framework, which combines Importance Sampling (IS) and a Direct Method (DM) reward model. The DR framework is chosen for its theoretical guarantee of unbiasedness if either the propensity scores (IS) or the reward model (DM) is correctly specified.

The paper investigates three specific strategies for incorporating counterfactual annotations ( $g_i$ ) into the DR estimator, defined by how the annotations are utilized in the DM and IS components:

DM+-IS: The reward model ( $\hat{R}^+$ ) is trained on the augmented dataset (factual + counterfactual), but the IS component uses only the original factual data and standard propensity scores.
$\hat{V}_{DM^+-IS} = \frac{1}{N} \sum_{i=1}^N \left( \hat{R}^+(s_i, \pi_e) + \rho_{s_i}(a_i)(r_i - \hat{R}^+(s_i, a_i)) \right)$
DM-IS+: The reward model ( $\hat{R}$ ) is trained only on factual data, while the IS component is augmented using the counterfactual annotations with re-weighting (similar to the IS+ approach).
$\hat{V}_{DM-IS^+} = \frac{1}{N} \sum_{i=1}^N \left( \hat{R}(s_i, \pi_e) + \sum_{a \in A} w_i^a \rho^+_{s_i}(a)(c_i^a - \hat{R}(s_i, a)) \right)$
DM+-IS+: Both the reward model and the IS component utilize the augmented dataset.
$\hat{V}_{DM^+-IS^+} = \frac{1}{N} \sum_{i=1}^N \left( \hat{R}^+(s_i, \pi_e) + \sum_{a \in A} w_i^a \rho^+_{s_i}(a)(c_i^a - \hat{R}^+(s_i, a)) \right)$

Theoretical Analysis

The authors provide a rigorous bias-variance analysis under three annotation regimes:

Perfect Annotations: All three estimators are unbiased.
Biased Annotations: The authors prove that DM+-IS remains an unbiased estimator of the target policy value, even when annotations are systematically biased. In contrast, DM-IS+ and DM+-IS+ introduce a bias term proportional to the annotation error ( $\epsilon_G$ ), making them susceptible to performance degradation.
Noisy Annotations: While all estimators experience increased variance due to annotation noise, the theoretical derivation shows that DM+-IS isolates the variance impact to the reward model estimation, whereas the other methods compound noise through the re-weighted IS term.

Experimental Results

The proposed methods were evaluated across three domains:

Heartsteps: A semi-synthetic mobile health simulator for physical activity interventions.
Sepsis: A semi-synthetic simulator for sepsis treatment (adapted from an MDP setting to a contextual bandit).
MIMIC-IV: A real-world electronic health record (EHR) dataset focusing on intravenous potassium repletion. Counterfactual annotations for this dataset were generated using the OpenAI "o1" model.

Key Findings:

Robustness to Bias: In scenarios with biased annotations, DM+-IS consistently achieved the lowest Root Mean Squared Error (RMSE). The RMSE of DM+-IS remained stable as annotation bias increased, whereas DM-IS+ and DM+-IS+ showed significant performance degradation.
Robustness to Misspecification: When combined with misspecified reward models (a common real-world scenario), DM+-IS remained the most robust estimator. It effectively leveraged the counterfactual data to improve coverage without inheriting the bias from the annotations.
Real-World Performance: On the MIMIC-IV dataset, DM+-IS outperformed all baselines (including standard IS, DM, DR, and annotation-augmented variants) across varying numbers of counterfactual annotations. Notably, as the number of annotations increased, the performance of DM+-IS improved and then plateaued, while other estimators (like IS+ and DM-IS+) saw their error rates increase or remain high.
Annotation Quality: The experiments demonstrated that annotation bias has a far greater negative impact on estimator performance than annotation variance (noise).

Significance and Contributions

The paper makes the following contributions:

Novel Estimator Family: It introduces and theoretically analyzes a family of DR-inspired estimators designed specifically to handle imperfect counterfactual annotations.
Theoretical Insight: It identifies that incorporating counterfactual annotations only into the Direct Method (DM) component (the DM+-IS strategy) is the optimal approach. This strategy preserves the unbiasedness of the estimator even when annotations are biased, a property not shared by methods that augment the IS component.
Empirical Validation: Through extensive experiments on synthetic, semi-synthetic, and real-world healthcare data, the authors demonstrate that DM+-IS is the most robust choice for policy evaluation when both reward model accuracy and annotation quality are unknown.

Conclusion

This work addresses the critical gap between the theoretical promise of counterfactual data augmentation and the practical reality of imperfect annotations. By proposing the DM+-IS estimator, the authors provide a method that broadens the applicability of OPE in high-stakes settings like healthcare. The approach allows practitioners to safely utilize imperfect expert or AI-generated annotations to evaluate policies for actions not well-covered in historical data, without risking the introduction of systematic bias that could lead to unsafe policy deployment. The paper concludes that under realistic conditions where annotation quality is unknown, DM+-IS offers the most reliable path forward for off-policy evaluation.