Latent Utility Q-Learning for Preference-Adaptive Dynamic Treatment Regimes

Technical Summary: Latent Utility Q-Learning for Preference-Adaptive Dynamic Treatment Regimes

Problem Statement
Dynamic Treatment Regimes (DTRs) are statistical methods designed to optimize sequential treatment decisions for chronic conditions. Standard DTR methods typically assume a single univariate outcome, which fails to address the reality that patients often weigh multiple, competing outcomes (e.g., pain reduction vs. functional improvement) differently. Existing preference-based methods are often limited to two-stage settings, rely on restrictive parametric models, or require patients to explicitly rank treatment trajectories—a task often infeasible in clinical trials where patients experience only one trajectory. Furthermore, deep reinforcement learning approaches generally require large sample sizes incompatible with moderate-sample clinical studies like Sequential Multiple Assignment Randomized Trials (SMARTs). The core challenge is to learn a policy that optimizes patient-specific, preference-weighted combinations of multivariate outcomes ($Y \in \mathbb{R}^d$) across $K$ decision points without requiring explicit outcome rankings or large-scale data.

Methodology: Latent Utility Q-Learning (LUQ-Learning)
The authors propose LUQ-Learning, a latent-variable extension of Q-learning that estimates DTRs optimizing a latent utility function $U^* = E^\top Y^*$, where $E$ is an unobserved $(d-1)$-dimensional preference vector on a probability simplex, and $Y^*$ represents potential outcomes.

• Latent Variable Framework: Patient preferences ($E$) are not directly observed but are inferred through indirect measurements ($W_k$), such as discrete choice questions, satisfaction ratings, or quality-of-life subscales, collected at various stages.
• Conditional Mean Factorization: The central methodological insight is a conditional mean factorization. Under a "Sequential Preference Independence" assumption (Assumption A5), the expected utility can be decoupled:
  $$E[U^* | H_k, A_k] = E[E | H_k]^\top E[Y | H_k, A_k]$$
  This factorization separates the problem into two modular components:
  1. Preference Estimation: Estimating the posterior distribution of latent preferences $P(E | H_k)$ based on observed preference instruments $W_k$ and covariates.
  2. Outcome Regression: Estimating the conditional mean of outcomes $E[Y | H_k, A_k]$.
• Algorithm: The algorithm proceeds via backward induction (similar to standard Q-learning). At each stage $k$, it estimates the posterior mean preference $\hat{E}[E | H_k]$ and the outcome regression, then computes the Q-value as the inner product of these estimates. This reduces the multi-outcome, preference-adaptive problem to a sequence of scalar Q-learning problems.
• Uncertainty-Aware Variants: Recognizing that preference estimation introduces uncertainty, the authors propose two policy variants:
  • Modal Policy ($\hat{\pi}^{mod}$): Selects the action that is optimal for the largest proportion of the posterior preference distribution.
  • Robust Policy ($\hat{\pi}^{rob}$): Selects the action that maximizes the minimum expected utility within an ellipsoidal uncertainty region defined by the posterior covariance.

Key Contributions

1. Theoretical Guarantees:
   • Consistency: The authors establish that the estimated value function converges in probability to the true optimal value under weak regularity conditions on the preference and outcome estimators. The value gap is shown to be driven by the additive accumulation of preference model error, outcome regression error, and propagated value-function error.
   • Unified $\epsilon$-Optimality: The paper derives bounds on policy value loss in terms of posterior preference uncertainty. These bounds provide interpretable, data-driven criteria (e.g., thresholds for uncertainty radius $\delta$ or posterior mass $\alpha$) for selecting between mean, modal, and robust policies.
2. Modular and Flexible Design: The framework accommodates heterogeneous preference data formats (e.g., binary choices, rankings, Likert scales) without requiring scale alignment across instruments. It does not require patients to explicitly rank outcomes or trajectories.
3. Handling Uncertainty: By utilizing a Bayesian framework, the method propagates uncertainty from the latent preference estimation directly into treatment recommendations and value estimation, allowing for risk-averse (robust) or coverage-oriented (modal) decision-making.

Results
The authors evaluated LUQ-Learning using simulations calibrated to the Biomarkers for Evaluating Spine Treatments (BEST) trial, a two-stage SMART for chronic low back pain.

• Performance: LUQ-Learning consistently outperformed:
  • Standard Q-Learning: Which aggregates multivariate outcomes via a simple average.
  • Last-Reported Satisfaction ($\hat{\pi}^{W_{last}}$): Which optimizes based solely on the final satisfaction rating without longitudinal preference data.
  • Existing Preference Methods: Specifically, the method showed lower preference estimation error and higher value gains compared to the approach by Butler et al. (2018) in a single-stage setting.
• Stability: While the "Oracle" policy (knowing true preferences) achieved the highest value, LUQ-Learning approached this performance closely. In contrast, methods ignoring longitudinal preference data ($\hat{\pi}^{W_{last}}$) exhibited higher variance and instability, particularly as the trajectory length ($K$) increased.
• Trajectory Length: In extended simulations with $K \in \{2, 4, 6, 8\}$, the performance gap between LUQ-Learning and the Oracle decreased as $K$ increased, suggesting that longer trajectories provide richer information for estimating latent preferences. Conversely, methods relying only on terminal proxies deteriorated as $K$ grew.

Significance and Claims
The paper claims that LUQ-Learning provides a statistically principled foundation for preference-adaptive decision-making in complex, multi-outcome clinical environments. Its significance lies in:

• Bridging the Gap: It offers a solution for incorporating heterogeneous patient preferences into DTRs without the data requirements of deep reinforcement learning or the trajectory-ranking constraints of preference-based RL.
• Interpretability: The derived $\epsilon$-optimality bounds offer concrete, data-driven criteria for policy selection, helping clinicians and regulators navigate trade-offs between average performance and robustness to preference uncertainty.
• Causal Inference Foundation: Unlike standard reinforcement learning from human feedback, LUQ-Learning treats preference instruments as identifiable proxies for latent utilities within a causal inference framework (using potential outcomes and proximal causal inference concepts), making it suitable for randomized trial settings where no single expert policy governs treatment assignments.

The authors conclude that while the framework is robust, its successful deployment requires careful study design, validated preference instruments, and alignment between development and target populations to ensure the captured preferences reflect stable, long-term trade-offs rather than transient satisfaction.