Technical Summary: Learning to Compress Time-to-Control

Problem Statement

Reinforcement learning (RL) in healthcare has historically struggled with specific failure modes: sparse reward signals (e.g., mortality), unreliable off-policy evaluation due to uncharacterized behavior policies, and a significant gap between simulation performance and real-world deployment. The authors argue that chronic disease management represents a structurally more tractable setting for RL than the acute-care problems (e.g., sepsis) that have dominated the field. However, this potential is only realized if the problem is formalized to exploit the unique properties of chronic care: high decision density, frequent outcome observability via biomarkers, and contractual alignment with value-based payment models like the CMS ACCESS Model.

The core challenge addressed is how to design an RL framework that avoids "reward gaming," handles the deployment-simulation gap (specifically regarding execution constraints), and effectively learns from heterogeneous clinician data without simply imitating suboptimal population averages.

Methodology

The paper proposes a formalization of the chronic care RL problem centered on four structural innovations:

1. Patient State Machines and Typed Actions

Instead of a monolithic action space, the framework partitions actions into two distinct layers with different causal structures:

• Clinical Layer ($A_{clin}$): Actions targeting the patient's biological state directly (e.g., medication titration, lab orders).
• Operational Layer ($A_{ops}$): Actions targeting the execution of clinical decisions (e.g., scheduling, outreach, follow-through tracking).
  This separation allows the model to distinguish between clinical decision quality and operational execution capability. The patient state is modeled as a continuous-time process observed at discrete measurement times, partitioned into controlled ($C_c$) and uncontrolled ($U_c$) regions based on clinical thresholds.

2. The TTG/TTO/TTC Reward Hierarchy

To address reward sparsity, the authors define a tiered reward structure based on three first-passage times measured from an index uncontrolled observation:

• TTG (Time-to-Goal): Time to a clinically meaningful step-change (e.g., $\ge$15 mmHg SBP reduction).
• TTO (Time-to-Optimization): Time to a substantial but incomplete improvement or structural milestone (e.g., full titration).
• TTC (Time-to-Control): Time to sustained control within the target region.
  This hierarchy is calibrated to the CMS ACCESS Model, providing external validity and ensuring the reward signal is dense (5–30 events per patient-year) compared to single-endpoint formulations.

3. Execution Intensity ($\epsilon$) as a Constrained MDP

The framework treats execution intensity ($\epsilon = \epsilon_{cap} \cdot \epsilon_{will}$) not as an exogenous nuisance but as a learned state variable. $\epsilon_{cap}$ represents operational capacity, and $\epsilon_{will}$ represents clinician willingness.

• Constrained MDP: The agent operates in a constrained Markov Decision Process where the available action set $A_{avail}(s)$ is a function of the estimated execution intensity $\hat{\epsilon}(s, a)$.
• Stall States: The framework defines specific "stall states" where a patient fails to progress within expected response windows. These states trigger autonomous operational intensification (e.g., increased outreach) and escalate clinical decisions.

4. Two-Loop Architecture Coupling Preference Learning and RL

The most distinct contribution is the coupling of the RL agent to a clinician preference-learning system (from a companion paper, Singh et al., 2026). This creates a two-loop architecture:

• Outer Loop (Preference Learning): Continuously updates posterior estimates of clinician capability ($\kappa$) and execution intensity ($\epsilon$) based on accumulated override and outcome data.
• Inner Loop (RL Training): Uses these estimates as structural inputs:
  • $\hat{\epsilon}$ constrains the action space (Equation 3).
  • $\kappa$ weights the transitions in the offline RL objective (Equation 4), scaling the gradient contribution of transitions generated by clinician $i$ by $w_i = \exp(\beta \kappa_i)$.
    This allows the agent to preferentially learn from high-capability clinicians rather than imitating the population mean.

5. The Engagement Harness

A supervisory layer is introduced to bound agent autonomy. It routes decisions based on uncertainty, atypicality, and risk thresholds into three categories: autonomous execution, clinician review with a default, or clinician decision required. This ensures safety while allowing the agent to expand its operating envelope as reliability is demonstrated.

Key Results (Simulation)

The authors evaluated the framework using synthetic state machines for Hypertension (HTN) and Type 2 Diabetes (T2D).

• Capability-Weighted vs. Uniform-Weighted RL:
  • In Study A, capability-weighted offline RL significantly outperformed uniform-weighted offline RL.
  • For T2D, the capability-weighted variant achieved a 15 percentage point absolute improvement in Time-to-Control (TTC) rates (42% vs. 27% for the behavior policy baseline).
  • Crucially, the uniform-weighted formulation underperformed the heterogeneous behavior policy on T2D (16% TTC vs. 27%), demonstrating that imitating the population mean of clinician behavior (which includes therapeutic inertia) is a strictly worse target than the average clinician.
• Execution Intensity Generalization:
  • In Study B, policies trained with explicit $\epsilon$ awareness ($\epsilon$-aware) generalized across different deployment regimes (varying action availability probabilities).
  • $\epsilon$-naive policies (trained without $\epsilon$ as a state variable) failed to adapt to higher deployment $\epsilon$, remaining flat in performance even when action availability improved.
• Reward Formulation: While the tiered reward provided denser signals, the simulation showed that capability-weighted RL with a sparse terminal-event reward produced the strongest policies in the tabular setup tested, suggesting that clean end-of-trajectory signals may be robust against over-crediting partial responses in simple representations.

Significance and Claims

The paper claims that chronic disease management is a favorable domain for RL only if the problem is formalized to exploit its structural properties rather than importing acute-care assumptions.

1. Structural Tractability: The combination of dense biomarker rewards, high decision density, and contractual alignment (CMS ACCESS) makes chronic care more amenable to RL than acute care, provided the reward and constraint structures are correctly specified.
2. The "Load-Bearing" Innovation: The tight coupling to clinician preference learning is the central technical claim. The simulation confirms that an agent utilizing $\kappa$ (capability) and $\epsilon$ (execution intensity) as structural elements outperforms standard offline RL. Specifically, the framework solves the "imitation of the mean" failure mode by learning from high-capability outliers rather than the suboptimal population average.
3. Deployment-Simulation Gap: By making execution intensity an explicit state variable in a constrained MDP, the framework addresses the deployment-simulation gap. Policies trained with this formulation generalize to different operational contexts, whereas standard policies do not.
4. Safety and Autonomy: The engagement harness provides a mechanism to safely scale autonomy, routing high-risk or uncertain decisions to clinicians while allowing autonomous execution of routine operational tasks.

The authors conclude that while the framework's structural arguments hold in simulation, quantitative validation in real-world deployments (retrospective and prospective) is the necessary next step to confirm these claims in production environments.