Counterfactual Survival Q-learning via Buckley-James Boosting, with Applications to ACTG 175 and CALGB 8923

Technical Summary: Counterfactual Survival Q-learning via Buckley-James Boosting

Problem Statement
The paper addresses the challenge of estimating optimal dynamic treatment regimes (DTRs) from longitudinal randomized clinical trials where outcomes are right-censored survival times. A primary difficulty in this domain is the presence of incomplete follow-up data due to patient dropout or administrative censoring, which complicates the estimation of potential outcomes. Furthermore, standard methods often rely on the proportional hazards (PH) assumption (e.g., Cox regression), which may not hold when covariate effects are nonlinear or when hazard ratios are difficult to interpret clinically. Existing approaches, such as linear Buckley–James (BJ) Q-learning, struggle with nonlinear relationships and may exhibit convergence issues when the true data-generating mechanism is complex. The authors aim to develop a flexible framework that avoids the PH assumption, handles nonlinear covariate effects, and provides robust estimation of stage-specific Q-functions under censoring.

Methodology
The proposed method, BJ Boost Q-learning, integrates counterfactual Q-learning with the Buckley–James (BJ) boosting algorithm within an accelerated failure time (AFT) framework.

1. Framework Setup: The study considers a sequentially randomized trial with $K$ decision stages. It utilizes potential outcome notation to define stage-specific durations ($T_{i,k}$) and cumulative survival times. The goal is to learn decision rules $d_k^*$ that maximize the expected cumulative survival time via backward recursion of Q-functions.
2. Buckley–James Boosting for Imputation: To handle right-censored outcomes, the method employs BJ boosting to impute censored survival times. Instead of modeling hazard functions, it models the conditional mean of the log-transformed survival time directly.
   • Imputation Mechanism: For a given stage and treatment arm, a regression function $f_{k,a}(\cdot)$ is fitted. For censored observations, the residual distribution is estimated using the Kaplan–Meier estimator. The censored outcome is imputed as the fitted value plus the conditional mean of the residual given that it exceeds the observed time.
   • Flexible Base Learners: The paper implements two boosting strategies to estimate the regression function $f_{k,a}$:
     • BJ-LS: Uses componentwise least squares with twin boosting refinement (Algorithm 1).
     • BJ-Tree: Uses regression trees (Algorithm 2).
   • Iterative Process: The imputation and model fitting are performed iteratively. Cross-validation is used within each stage and treatment arm to select the number of boosting steps, with imputed responses used for scoring to account for censoring.
3. Backward Q-Learning: Once censored outcomes are imputed, the method proceeds with standard backward recursion. At the final stage $K$, the Q-function is estimated directly from the imputed outcomes. For preceding stages $k < K$, pseudo-outcomes are constructed by adding the imputed stage outcome to the maximum estimated Q-value of the subsequent stage. These pseudo-outcomes are then regressed on the history and current treatment using the same boosting learners to estimate $Q_k$.
4. Identifiability: The approach relies on standard assumptions for sequentially randomized trials: consistency, sequential randomization, positivity, and no interference.

Key Contributions

• Novel Framework: The paper introduces a counterfactual Q-learning framework that combines the AFT model with iterative boosting, specifically tailored for right-censored survival data.
• Flexibility and Robustness: By utilizing flexible base learners (componentwise least squares and regression trees) within the BJ framework, the method captures nonlinear covariate-treatment interactions without relying on the proportional hazards assumption.
• Direct Modeling of Time: Unlike Cox-based Q-learning, which models hazards, this approach directly models the conditional survival time, yielding clinically interpretable time-scale contrasts.
• Algorithmic Implementation: The authors provide detailed algorithms (Algorithms 1 and 2) for implementing BJ twin boosting and BJ tree boosting with iterative Buckley–James imputation.

Results
The performance of BJ Boost Q-learning was evaluated through simulation studies and real-data applications.

• Simulation Studies:
  • Single-Stage Setting: In a setting with 50% censoring and nonlinear covariate effects, BJ-Tree achieved the highest treatment decision accuracy (median >91% for $n=1000$), significantly outperforming linear BJ-Q and Cox-based Q-learning (median <45%). BJ-LS also performed well but was slightly less accurate than BJ-Tree.
  • Two-Stage Setting: In a multistage setting, estimation errors can compound. BJ-Tree maintained the highest accuracy (median >84%), while Cox-based Q-learning performed poorly (median <20%). The results highlighted that flexible, nonparametric models are crucial for capturing complex structures in multistage regimes.
• Application to ACTG 175 (HIV Trial):
  • Analyzed 2,139 patients comparing ddI monotherapy vs. AZT+ddI combination therapy.
  • Both BJ-LS and BJ-Tree strongly recommended the combination therapy for the vast majority of participants.
  • The learned rules identified clinically interpretable covariates (e.g., Karnofsky score, symptom status, prior antiretroviral use) associated with treatment recommendations, consistent with prior literature.
• Application to CALGB 8923 (Leukemia Trial):
  • Analyzed a two-stage trial (induction and post-remission therapy) with 388 participants.
  • Unlike the ACTG 175 results, the Cox, BJ, and BJ-LS models produced statistically significant paired differences favoring one treatment, leading to decisive recommendations.
  • In contrast, BJ-Tree yielded balanced recommendations with no statistically significant average difference in counterfactual survival, aligning more closely with the trial's reported lack of average survival benefit. This suggests BJ-Tree may be less prone to overfitting or inflating contrasts in settings with weak treatment effects.

Significance and Claims
The paper claims that BJ Boost Q-learning offers a robust and flexible alternative to Cox-based Q-learning for learning dynamic treatment regimes from censored survival data. Its primary significance lies in:

1. Avoiding Model Misspecification: By avoiding the proportional hazards assumption and using flexible learners, it reduces bias in settings with nonlinear effects.
2. Improved Decision Accuracy: Simulation results demonstrate that the method, particularly the tree-based variant, significantly improves the accuracy of treatment decisions compared to linear or hazard-based approaches.
3. Stability in Multistage Settings: The method is shown to be particularly valuable in multistage trials where errors in earlier stages can compound, providing more stable counterfactual contrasts.
4. Clinical Interpretability: The direct modeling of survival time allows for the derivation of individualized decision rules that are interpretable in terms of time-to-event benefits.

The authors modestly note limitations, including the need for further assessment under high censoring rates, potential bias propagation in early boosting stages, and the current restriction to randomized trials (requiring extensions for observational data).