🤖 machine learning

Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication

This paper proposes a framework for evaluating and training clinical prediction models that explicitly addresses the challenge of treatment-induced label indeterminacy by leveraging expert-annotated counterfactual outcomes for uncertain cases, revealing a critical trade-off between accuracy on observable outcomes and reliability for the very patients who need prognostic support most.

Original authors: Xiaobin Shen, Chloe Y. H. Huang, Jonathan Elmer, George H. Chen

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaobin Shen, Chloe Y. H. Huang, Jonathan Elmer, George H. Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to predict the weather. You show it thousands of photos of clouds and tell it, "This one means rain, this one means sun." The robot gets really good at matching the photos to the weather you saw. But what happens if, in some of the photos, a giant, invisible hand suddenly snatches the sky away before you can see if it rains or shines? The robot doesn't know what happened in those missing moments. In the real world, this isn't just about clouds; it happens in hospitals. Sometimes, doctors have to make a hard choice to stop life-saving treatments because a patient looks very sick. If they stop the treatment, the patient might pass away. But here's the tricky part: we never get to see if the patient would have woken up and recovered if the doctors had kept fighting. The outcome is "indeterminate"—it's a mystery. This creates a huge problem for computer models trying to learn from medical data. They can only learn from the patients whose outcomes they actually saw, but the patients who need the most help are often the ones whose outcomes are hidden by those very treatment decisions.

This paper tackles that exact mystery in the world of heart attack recovery. The researchers looked at nearly 2,500 patients who survived a cardiac arrest but were still in a coma. They split these patients into two groups: "Certain" cases, where the patient either clearly woke up or clearly didn't, and "Uncertain" cases, where treatment was stopped or failed for other reasons, leaving their true recovery potential unknown. For the "Uncertain" group, the researchers didn't just guess; they asked a team of expert doctors to look at the charts and say, "If we had kept treating this person, what do you think would have happened?" The experts gave their best guesses, which the researchers treated as a "shadow" label—a hint at the truth, but not the truth itself.

The big discovery here is that standard computer models are playing a dangerous game of "hide and seek." When the researchers tested different models, they found that a model could look perfect on the "Certain" patients (getting the right ranking of who would recover) but be completely wrong about the "Uncertain" patients. It's like a student who aces a multiple-choice test on history but fails to understand the actual story behind the dates. The paper shows that there is a strict trade-off: if you tune the model to listen more closely to the experts' guesses about the "Uncertain" patients, the model gets better at predicting the hidden outcomes, but it starts making more mistakes on the "Certain" patients whose outcomes we already know. Conversely, if you make the model perfect for the "Certain" group, it tends to ignore the "Uncertain" group and give them overly optimistic or pessimistic predictions that don't match the experts' intuition.

The authors suggest that we can't just pick the "best" model based on a single score. Instead, we have to choose a balance. They built a special tool that lets doctors slide a knob to decide how much they want the model to trust the experts' guesses versus sticking to the hard data of what actually happened. The study concludes that ignoring the "Uncertain" patients because their data is messy is a mistake. It hides a failure mode where the model might give bad advice to the very people who need it most. By acknowledging that some outcomes are guesses rather than facts, and by accepting that we have to trade off accuracy in one group for accuracy in another, we can build better, more honest tools for saving lives.

Technical Summary: Learning Under Treatment-Induced Label Indeterminacy

Problem Formulation
The paper addresses a fundamental limitation in clinical prediction modeling known as treatment-induced label indeterminacy. In standard supervised learning, it is assumed that the target outcome is cleanly observed for every patient. However, in high-stakes settings like post-cardiac-arrest neurological prognostication, treatment decisions (specifically the withdrawal or limitation of life-sustaining therapies, WLST) or competing terminal events (e.g., multisystem organ failure) can permanently obscure the clinically relevant outcome (neurological recovery).

This creates a dataset split into two distinct subsets:

Certain Cases: Patients whose outcomes are directly observed (e.g., regaining consciousness or death without prior treatment limitation). These provide valid supervised labels.
Uncertain Cases: Patients whose outcomes were rendered indeterminate by treatment decisions. For these patients, the "true" counterfactual outcome (what would have happened had treatment continued) is unobservable.

The core challenge is that standard evaluation metrics (like AUROC) computed only on certain cases fail to capture model behavior on uncertain cases—the very population for whom prognostic support is most critical. Furthermore, strategies that assign uncertain cases the same label as observed poor outcomes (e.g., death) cause models to learn patterns of physician decision-making (when WLST occurs) rather than true recovery potential.

Methodology
The authors propose a framework that explicitly distinguishes between certain and uncertain cases during both training and evaluation, utilizing expert assessments to guide the learning process for uncertain cases.

Data and Expert Assessments: The study uses a retrospective cohort of 2,497 post-cardiac-arrest patients (1,068 certain, 1,429 uncertain). For uncertain cases, the authors leverage structured counterfactual assessments from independent clinical experts (3–5 per patient). These experts estimate the probability of recovery had life-sustaining therapies continued. These assessments are treated as imperfect pseudo-labels (expert-informed references) rather than ground truth.
Model Architecture: The primary model is a single-head neural network designed to predict the probability of favorable neurological recovery.
Training Objective: The authors introduce a two-parameter objective function to control the tradeoff between fitting observed data and aligning with expert beliefs:
- $L_{certain}$ : Weighted binary cross-entropy on certain cases. A weight $\omega_{bad} \ge 1$ is applied to observed poor-outcome cases to control the influence of the low-probability region.
- $L_{align}$ : A mean squared error term on uncertain cases, measuring the difference between the model's prediction and the aggregated expert-derived probability ( $\tilde{p}_j$ ). A hyperparameter $\lambda_{align} \ge 0$ controls the strength of this alignment.
- Total Loss: $L(\theta) = L_{certain}(\theta) + \lambda_{align} L_{align}(\theta)$ .
Evaluation Framework: The paper proposes a split evaluation strategy:
- Certain Cases: Evaluated using standard metrics (AUROC and Brier score) against observed binary outcomes.
- Uncertain Cases: Evaluated using Mean Absolute Error (MAE) against the aggregated expert reference probabilities.
- Tradeoff Analysis: Models are not selected based on a single scalar metric but are analyzed along the frontier of the tradeoff between certain-case probability accuracy (Brier score) and uncertain-case alignment (MAE).

Key Contributions

Formalization of Label Indeterminacy: The paper formalizes predictive modeling where target labels differ qualitatively between subsets of the population (observed outcomes vs. expert counterfactual guesses).
Split Evaluation Framework: It introduces a framework that evaluates models on two axes: standard discrimination/accuracy on certain cases and alignment with expert belief on uncertain cases. This reveals failure modes hidden by aggregate metrics.
Empirical Characterization of Tradeoffs: Through experiments with tabular baselines (XGBoost, Random Forest) and neural networks, the authors demonstrate that models with similar AUROC on certain cases can exhibit vastly different behaviors on uncertain cases.
Explicit Tradeoff Control: The proposed two-parameter neural objective ( $\omega_{bad}$ and $\lambda_{align}$ ) allows practitioners to explicitly navigate the tradeoff between fitting observed data and aligning with expert counterfactual beliefs.

Results

The Hidden Gap: Models with nearly identical certain-case AUROC (e.g., 0.882 vs. 0.884) showed dramatic differences in certain-case Brier scores (0.116 vs. 0.407) and uncertain-case MAE (0.410 vs. 0.053). This indicates that ranking performance (AUROC) masks significant miscalibration in probability estimates.
The Tradeoff Frontier: Improving alignment with expert guesses on uncertain cases (lowering uncertain-case MAE) generally comes at the cost of worse probability accuracy on certain cases (higher Brier score).
Probability Placement: Standard "observed-only" models assigned relatively high recovery probabilities to uncertain cases, overlapping significantly with "certain good" cases. Incorporating expert alignment shifted uncertain-case predictions downward, better separating them from favorable outcomes but compressing the probability scale.
Baseline Comparison: Tabular baselines (XGBoost) exhibited the same tradeoff pattern as neural networks, suggesting the phenomenon is driven by the label structure rather than the specific model architecture.

Significance and Claims
The paper positions itself primarily as a problem formulation and evaluation framework rather than a novel algorithmic contribution. Its primary significance lies in demonstrating that:

Conventional evaluation metrics focusing solely on certain cases are insufficient in settings with treatment-induced label indeterminacy.
The "failure mode" in these settings is often probability misplacement (miscalibration) on the most critical subpopulation (uncertain cases), which is invisible to AUROC.
There is no single "best" model; instead, there is an explicit design choice regarding how much weight to place on observed outcomes versus expert counterfactual beliefs.
Expert guesses of counterfactual outcomes, while imperfect, serve as necessary supervision to study and improve model behavior on uncertain cases.

The authors explicitly state that they do not claim to recover true ground-truth counterfactual probabilities or solve label indeterminacy generally. Instead, they provide a method to use carefully elicited expert assessments to make the tradeoff between observed performance and counterfactual alignment explicit and navigable for practitioners.

Technical Summary: Learning Under Treatment-Induced Label Indeterminacy

More like this