Technical Summary: Decision-Targeted Digital Twins (DT2)

1. Problem Statement

Digital Twins (DTs) are virtual models of real-world systems designed to simulate dynamics and assist in decision-making by evaluating candidate policies. However, standard machine learning-based DTs are typically trained to minimize one-step transition errors (e.g., Mean Squared Error or Negative Log Likelihood) across all state-action variables. The authors argue that this "context-agnostic" optimization is suboptimal for the primary downstream goal of DTs: ranking policies according to a reward function.

When model capacity is limited, minimizing simulation fidelity can lead to models that accurately reproduce irrelevant dynamics while failing to capture the specific variables or temporal slices crucial for distinguishing between policies. The paper posits that a DT need not be a perfect replica of observed reality to be useful; rather, it must preserve the correct preference ordering of policies. The authors prove theoretically that standard training can induce suboptimal policy rankings even when the hypothesis space is limited, and empirically demonstrate that this misalignment persists even with expressive model classes.

2. Methodology: DT2

To address the misalignment between simulation fidelity and decision quality, the authors introduce DT2, a decision-targeted training paradigm. The core objective is to train a DT to generate rollouts that preserve the pairwise policy rankings derived from offline data, rather than solely minimizing transition errors.

2.1. Theoretical Motivation

The authors establish that if a DT hypothesis class $\mathcal{F}$ cannot perfectly model the true transition distribution $\Phi$, optimizing standard simulation losses ($L_{sim}$) does not guarantee optimal decision-making.

• Theorem 3.1 & 3.3: Prove that when $\Phi \notin \mathcal{F}$, there exist reward functions and policy sets where the $L_{sim}$-optimal DT induces an incorrect preference ordering, while a different model in $\mathcal{F}$ (which may have slightly higher simulation error) induces the correct ordering.

2.2. Training Framework

DT2 combines standard simulation objectives with a ranking loss derived from Off-Policy Evaluation (OPE). The total loss function is:
$$L_{DT2}(\theta) = (1 - \lambda)L_{sim}(\theta) + \lambda L_{rank}(\theta)$$
where $\lambda \in [0, 1]$ balances simulation fidelity and decision-targeting.

A. Proxy Ground Truth via Fitted Q-Evaluation (FQE)

Since ground-truth policy values $J(\pi)$ are unavailable in offline settings, DT2 uses Fitted Q-Evaluation (FQE) to estimate scalar policy values from the offline dataset. These estimates serve as "proxy ground-truth" values to construct pairwise ranking targets ($\Delta_{ij} = J(\pi_i) - J(\pi_j)$). The authors note that while FQE estimates may be biased, pairwise comparisons are often more robust, as errors tend to cancel out in relative ordering.

B. Smoothed Ranking Loss

To optimize for rankings using gradient-based methods, the authors replace the non-differentiable sign function in Kendall's rank correlation with a hyperbolic tangent ($\tanh$). The ranking loss is defined as:
$$L_{rank}(\theta) = 1 - \frac{1}{|C|} \sum_{(i,j) \in C} \tanh\left(\frac{\Delta_{ij}}{\alpha}\right) \cdot \tanh\left(\frac{\hat{\Delta}_{ij}(\theta)}{\alpha}\right)$$
where $\alpha$ is a temperature parameter controlling the trade-off between faithfulness to the discrete correlation and gradient smoothness. This formulation implicitly weights pairs: if the proxy values are close ($\Delta_{ij} \approx 0$), the gradient contribution vanishes, allowing the model to focus on distinguishing high-regret pairs.

C. Bootstrapped Value Estimates

Computing the DT-estimated value difference $\hat{\Delta}_{ij}$ requires simulating long trajectories, which is computationally expensive and prone to gradient instability. DT2 employs a bootstrapping scheme:

1. Rollouts are truncated at a fixed horizon $H$.
2. The remaining return is approximated using a frozen Q-function ($Q^\pi_\psi$) trained via FQE.
   This allows for efficient backpropagation through the DT while maintaining the ability to generate full trajectories for human inspection.

3. Key Contributions

1. Theoretical Proof: Demonstrated that standard DT training is theoretically suboptimal for decision-making when the hypothesis space is limited, establishing the motivation for decision-targeted training.
2. DT2 Paradigm: Proposed an architecture-agnostic training method that aligns simulations with proxy policy rankings using a smooth approximation of Kendall's rank correlation and FQE-derived targets.
3. Empirical Validation: Showed that the theoretical misalignment gap persists even with expressive hypothesis spaces and that DT2 significantly improves policy ranking and reduces decision regret compared to conventional training.

4. Experimental Results

The authors evaluated DT2 across six continuous control environments (Pendulum, Lunarlander, Hopper, Walker, Cheetah, Ant) and a medical cancer treatment simulation.

• Limited Capacity Scenarios: In toy problems with restricted neural capacity or misspecified mechanistic models, standard DTs failed to identify the optimal policy (success rates as low as 20-65%), whereas DT2 achieved 100% ranking accuracy by focusing capacity on decision-critical dynamics.
• Expressive Hypothesis Spaces: Across six Mujoco/Gymnasium environments and five architectures (Transformer, GRU, MLP, ResNet, NODE), DT2 consistently outperformed standard DTs trained with Negative Log Likelihood ($L_{NLL}^{sim}$).
  • Decision Regret: DT2 reduced average decision regret by 54% (from 4.08 to 1.88).
  • Ranking Alignment: Spearman's rank correlation improved by 47% (from 0.45 to 0.66).
  • Fidelity Trade-off: This improvement came at a modest cost, with DT2 models showing a 17% increase in test set transition MSE compared to standard DTs.
• Generalization: DT2 demonstrated better performance on unseen policies (both PPO checkpoints and manually defined medical plans) that were not present during training, suggesting it learns dynamics relevant to general decision-making rather than overfitting to specific training policies.
• Baselines: DT2 outperformed various Model-Based Reinforcement Learning (MBRL) baselines (MOReL, MOPO, ROMI, VaGraM) and a recent DT baseline (HDTwin) in terms of both regret and ranking correlation, often placing in the top two methods.
• Hyperparameter Sensitivity: The trade-off parameter $\lambda$ allows practitioners to navigate the fidelity-accuracy curve. The authors suggest small $\lambda > 0$ (e.g., 0.1) yields the best balance, as $\lambda \to 1$ can lead to training instability.

5. Significance and Claims

The paper claims that the prevailing practice of maximizing simulation fidelity in Digital Twin construction is misaligned with their primary purpose: decision support. By shifting the optimization objective to prioritize the correct ranking of candidate policies, DT2 offers a more effective tool for high-stakes domains like medicine and public policy.

The authors emphasize that DT2 does not require the DT to be a perfect replica of reality. Instead, it "distills" the ranking ability of black-box OPE methods into the interpretable structure of a DT. This allows users to interrogate simulations and verify constraints while ensuring the underlying dynamics lead to correct policy selections. The work suggests that for human-in-the-loop decision support, decision-targeted training is a necessary evolution over context-agnostic simulation training.

Limitations: The authors acknowledge that DT2's performance depends on the quality of the proxy ranking targets (FQE estimates), which can be biased if the offline data poorly covers the state-visitation distributions of candidate policies. Additionally, DT2 introduces computational overhead due to the need for pre-trained Q-functions and bootstrapped rollouts during training.