Trajectory Design and Budgeted Querying for Digital Twin Calibration

Technical Summary: Trajectory Design and Budgeted Querying for Digital Twin Calibration

Problem Formulation
Digital Twin (DT) calibration requires interaction data to align simulator parameters with a real system, yet acquiring privileged parameter measurements (ground truth) is often expensive or safety-critical. The authors frame calibration as a Partially Observable Markov Decision Process (POMDP) where the system has hidden parameters $\vartheta$ constant within an episode. The core challenge involves two coupled acquisition decisions:

1. Trajectory Design: Which control policies should generate interaction data to maximize parameter observability?
2. Budgeted Querying: When should a limited budget of expensive "oracle" measurements (privileged access to true parameters) be spent to correct the DT's state?

The goal is to minimize the terminal error of the deployed parameter estimate $\tilde{\vartheta}(T)$ subject to a hard budget $B$ on the number of queries $Q$.

Methodology
The proposed framework decouples the calibration pipeline into three sequential components, trained in stages rather than jointly:

1. Excitation-Oriented Controller: Unlike standard controllers that optimize for task reward (which may suppress informative motion), this controller is shaped to excite the system dynamics relative to the hidden parameters.

   • In the Pendulum environment, the reward function includes a squared sensitivity proxy ($\sin^2(\phi_t)$) to encourage motion where gravity $g$ has the strongest dynamic effect, counteracting the stabilizing tendency of standard controllers.
   • In Waterworld, a single reward was insufficient. Instead, a mixture of five distinct controllers (including broad exploration, velocity conservation, and diversity-focused policies) was assembled to generate a diverse set of trajectories.

2. Recurrent Parameter Estimator: A Gated Recurrent Unit (GRU) processes the trajectory history (observations, actions, rewards) to output a predictive mean $\hat{\vartheta}(t)$ and standard deviation $\sigma(t)$.

   • The estimator is trained first with Mean Squared Error (MSE) and then with Gaussian Negative Log-Likelihood (NLL) to learn the uncertainty distribution.
   • The recurrent state serves as a deterministic history embedding, not a Bayesian posterior.

3. Budgeted Query Policy: A separate policy ($\pi_q$) decides whether to query the oracle at step $t$.

   • State: The policy observes episode progress, the estimator's predictive uncertainty, standardized output, and remaining budget.
   • Action: A binary decision ($q_t \in \{0, 1\}$).
   • Mechanism: A query provides the true parameter $\vartheta_{true}$ for a single step, correcting the deployed parameter $\tilde{\vartheta}(t)$ used by the twin. Crucially, the query does not update the estimator's internal state or the controller's trajectory generation; it acts as a one-step correction.
   • Training: The policy is trained via Proximal Policy Optimization (PPO) against a frozen estimator, optimizing a reward that balances query cost against terminal accuracy.

Key Results
The paper presents exploratory case studies in two environments:

• Pendulum (Fully Observed, 1 Hidden Parameter):

  • Recoverability: A Random Forest trained on task-oriented (stabilizing) trajectories failed to recover mass or length and only weakly recovered gravity ($R^2 \approx 0.02$).
  • Excitation Impact: A GRU trained on excitation-oriented trajectories achieved a Mean Absolute Error (MAE) of 0.0066 with zero queries.
  • Budgeted Querying: When continuous oracle access was withdrawn mid-episode (after 50 steps of ground truth), the combined pipeline (estimator + query policy) achieved a terminal error of 0.0092 with a budget of 3 queries. This significantly outperformed an uncalibrated twin (error 0.2031) and a randomly drawn parameter (error 0.3047).

• Waterworld (Partially Observed, 3 Hidden Parameters):

  • Controller Mixture: Five different controllers produced distinct error profiles across the three hidden parameters (sensor range, max acceleration, pursuer speed).
  • Performance: An estimator trained on a mixture of these controllers achieved online normalized errors of roughly 4–5%.
  • Offline vs. Online Gap: The paper notes a significant discrepancy where the checkpoint with the best offline validation performance degraded significantly in online evaluation, suggesting offline selection does not reliably predict online error in this setting.

Significance and Claims
The authors position this work as an exploratory investigation rather than a controlled ablation study. Their primary claims are:

1. Feasibility: It is possible to treat trajectory design and query allocation as explicit design variables for data-scarce calibration.
2. Sensitivity: Calibration outcomes are highly sensitive to the data acquisition process (trajectory distribution) and the estimator architecture.
3. Limitations: The study does not claim causal improvements over standard identification methods due to the lack of matched comparisons (e.g., same estimator trained on different trajectory distributions). The components were trained sequentially, and experiments were run only once per configuration.
4. Future Directions: The authors suggest that future work should isolate the marginal contribution of querying via query-free baselines under identical protocols, utilize information-theoretic objectives for excitation, and test scalability on higher-dimensional physical systems.

The paper concludes that while the framework demonstrates low terminal errors in simulation, the specific contributions of trajectory design versus querying remain entangled in the current experimental setup.