Technical Summary: Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment

1. Problem Statement

Deep reinforcement learning (RL) agents for cyber-physical systems (e.g., autonomous vehicles, robotics) are typically trained in simulators due to the high cost and safety risks of real-world exploration. However, deploying these agents in real environments often leads to performance degradation and safety violations caused by the Sim-to-Real (Sim2Real) gap—the inevitable mismatch between simulated and real-world dynamics.

Existing approaches face significant trade-offs:

• Domain Randomization: Improves robustness but lacks explicit safety guarantees and can lead to unstable training.
• Distributionally Robust/Risk-Sensitive RL: Formulates transfer as a worst-case optimization, enhancing reliability under severe distribution shifts but often resulting in overly conservative policies and degraded task performance.
• Meta-RL/Domain Adaptation: Attempts to infer latent contexts but often assumes fixed risk preferences, lacking mechanisms to dynamically regulate the safety-performance trade-off during deployment.

The core challenge addressed is developing a transferable RL agent that is trained entirely in simulation yet remains adaptive and safe during real-world deployment, specifically handling unmodeled system dynamics without sacrificing efficiency.

2. Methodology

The authors propose a unified framework that integrates Probabilistic Latent Context Embeddings with Distributional Reinforcement Learning (Distributional RL) within a Constrained Markov Decision Process (CMDP) formulation.

2.1. Latent Context Variable Adaptation

The framework models the environment as a family of CMDPs indexed by an environment parameter $\xi$. To bridge the gap, a latent context variable $z$ is introduced to encode salient environment-specific information.

• Encoder: An inference network $q_\phi(z|D_\xi)$ is trained to estimate $z$ from a context set $D_\xi$ (transition samples). The encoder is optimized using a variational lower bound that includes reward and cost critics, alongside an information bottleneck (KL-divergence) to prevent overfitting.
• Inference: At deployment, the agent collects a limited number of real-world interactions ($N_{real}$) to refine the estimate of the latent context $\hat{z}_{\xi^*}$, allowing the policy to condition its actions on the inferred environment state.

2.2. Distributional RL Formulation

The framework employs Implicit Quantile Networks (IQN) to model the full distribution of returns (rewards and costs) rather than just their expectations.

• Critic: The critic learns quantile functions $Z(s, a, z; \tau)$ for the reward and cost.
• Actor: A deterministic actor $\pi_\phi(s, z)$ is trained with a risk-neutral objective based on the expectation of the IQN critics.

2.3. Dynamic Risk-Sensitive Policy Adaptation

A key innovation is the ability to dynamically adjust the risk level of the deployed policy based on the accuracy of the latent context estimation.

• Risk-Neutral vs. Risk-Sensitive: While the training policy is risk-neutral, the deployment policy can be adjusted to be risk-sensitive by evaluating the cost critic using upper-tail quantiles $U([\eta, 1])$ instead of the full range $U([0, 1])$.
• Dynamic $\eta$: The quantile parameter $\eta$ is adjusted dynamically during deployment.
  • Early Stage: When $N_{real}$ is small and the estimation error $\epsilon(N_{real})$ is high, a higher $\eta$ (more conservative) is used to ensure safety margins.
  • Later Stage: As more context is collected and $\hat{z}$ becomes accurate, $\eta$ is gradually decreased, allowing the policy to become less conservative and more efficient, eventually approaching the risk-neutral policy.
• Action Refinement: A lightweight, online projected gradient refinement is applied to the nominal action to satisfy the risk-aware cost constraint without updating network parameters.

2.4. Theoretical Guarantees

The paper provides theoretical proofs for:

• Convergence: The latent-conditioned distributional Bellman operators preserve the contraction property, ensuring training stability.
• Safety Upper-Bound: A theorem establishes that the cost of the deployed policy is bounded by $d + L_c \epsilon(N_{real})$, where $d$ is the safety threshold and $\epsilon$ is the estimation error.
• Conditional Safety: It is proven that if the cost reduction achieved by the risk-sensitive policy ($\Delta c$) exceeds the error-induced cost increase ($L_c \epsilon$), the safety constraint is satisfied.
• Existence of Safe Quantile: The authors prove that there exists a feasible quantile $\eta$ that ensures safety once a sufficient number of context samples ($N_{real} > N_{min}$) are collected.

3. Key Contributions

1. Unified Safe and Adaptive Framework: A CMDP-based framework integrating probabilistic latent context adaptation with distributional RL, enabling safe and adaptive policy transfer under Sim2Real mismatch.
2. Inference-Time Risk Regulation: A mechanism to dynamically adjust the risk level of the deployed policy at inference time based on the accuracy of the latent context estimation, balancing safety and performance.
3. Theoretical Safety Guarantees: Proofs regarding training convergence, the benefits of latent context adaptation, and assured safety under the Sim2Real gap via a derived safety upper-bound.

4. Experimental Results

The method was evaluated on two tasks: POINTGOAL2 (OpenAI Safety Gym) and an Autonomous Driving task (vehicle platoon oscillation reduction).

• Performance Comparison:
  • Baselines: Compared against Nominal (no randomization), Domain Randomization (D.R.), SPiDR (provable pessimistic safety), and Robust RL.
  • Results: In deployment, Nominal, D.R., and Robust RL achieved high rewards but suffered severe cost violations (safety breaches). SPiDR and the proposed method maintained safety but SPiDR was overly conservative, leading to lower rewards.
  • Proposed Method: Achieved a superior reward-cost trade-off, maintaining costs below the safety threshold while achieving higher rewards than SPiDR. In the autonomous driving task, it also significantly reduced traffic oscillations (O.R.), jerk, and fuel consumption compared to baseline controllers.
• Dynamic Adaptation: Experiments showed that the dynamic risk-sensitive adaptation module successfully maintained costs below the threshold throughout deployment. The risk parameter $\eta$ transitioned from conservative to less conservative as more real-world context was collected.
• OOD Sensitivity: The method consistently maintained safety across Mild, Medium, and High Out-of-Distribution (OOD) levels, whereas other methods failed under higher OOD shifts.
• Ablation Study: Removing the risk adaptation module resulted in cost violations, confirming the necessity of the dynamic risk regulation. The latent encoder alone improved performance over the nominal baseline but did not guarantee safety without the risk adaptation.

5. Significance and Claims

The paper claims that this framework addresses the critical limitation of existing Sim2Real methods that either sacrifice efficiency for robustness or fail to ensure safety when encountering unmodeled dynamics. By explicitly modeling the uncertainty of the environment context and dynamically regulating risk, the approach enables safe start-up in the early deployment stage (when context is scarce) and efficient operation once the agent has adapted.

The authors position this work as a step toward deploying RL in safety-critical cyber-physical systems where direct training is infeasible. They note that while the current work focuses on fixed but unknown environments, future directions could include handling continuously changing environments (e.g., sudden weather shifts) via change detection mechanisms. The framework is also noted as potentially adaptable to offline RL contexts, though data limitations in real-world offline settings pose challenges for accurate context and distribution estimation.