Technical Summary: Statistical Guarantees for Offline Domain Randomization

Problem Statement

Reinforcement learning (RL) agents frequently fail when deployed from simulation to the real world due to the "sim-to-real gap," caused by discrepancies between simulated dynamics and reality. While Domain Randomization (DR) is a dominant strategy to mitigate this by training policies across a family of simulators with randomized physical parameters, standard DR typically relies on uniform randomization (UDR) over broad parameter ranges. This approach ignores existing offline data collected from the real system.

Offline Domain Randomization (ODR) addresses this by first fitting a distribution over simulator parameters to an offline dataset before training the policy. While empirical methods like DROPO and DROID have shown success, the theoretical foundations of ODR remain unexplored. Specifically, it is unknown whether the fitted distribution converges to the true dynamics as data increases, or how ODR theoretically compares to UDR in reducing the sim-to-real gap.

Methodology and Formulation

The authors cast ODR as a Maximum Likelihood Estimation (MLE) problem over a parametric family of simulators.

1. Setup: The environment is modeled as an episodic Markov Decision Process (MDP). The real-world environment is $M_{\xi^*}$, where $\xi^*$ is the unknown true parameter vector. A simulator class $\mathcal{U} = \{M_\xi : \xi \in \Xi\}$ is defined.
2. Offline Data: An offline dataset $\mathcal{D} = \{(s_i, a_i, s'_i)\}_{i=1}^N$ is collected from the real system under an unknown behavior policy.
3. Estimation: The goal is to estimate a distribution $p_\phi(\xi)$ (parameterized by $\phi = (\mu, \Sigma)$, assumed Gaussian) that maximizes the likelihood of the observed transitions. The objective is:
   $$\hat{\phi}_N = \arg\max_{\phi \in \Phi} \sum_{(s,a,s') \in \mathcal{D}} \log \left( \mathbb{E}_{\xi \sim p_\phi(\xi)} [P_\xi(s' | s, a)] \right)$$
   This formulation treats the transition probability as a mixture of simulator transitions weighted by the learned distribution.
4. Policy Training: Once $\hat{\phi}_N$ is learned, a policy is trained in the simulator using this distribution as the prior for domain randomization.

Key Contributions and Theoretical Results

The paper provides the first statistical consistency proofs for ODR, establishing conditions under which the learned distribution converges to the true dynamics.

1. Weak Consistency (Section 4)

Under mild regularity, positivity, and identifiability assumptions, the authors prove that the ODR estimator is weakly consistent.

• Assumptions:
  • Simulator Regularity: Transition densities are bounded and continuous with respect to parameters.
  • Compactness: The parameter space is compact.
  • Mixture Positivity: The induced mixture density is strictly positive over the dataset support.
  • Identifiability: The only mixture distribution that exactly recovers the true transition kernel is the degenerate distribution at the true parameters $\xi^*$.
• Result: As the dataset size $N \to \infty$, the empirical maximizer $\hat{\phi}_N$ converges in probability to the true parameter $\phi^* = (\xi^*, 0)$.
  $$\hat{\phi}_N \xrightarrow{P} \phi^*$$

2. Strong Consistency (Section 5)

By adding a Uniform Lipschitz Continuity assumption on the log-likelihood function, the authors upgrade the result to strong consistency.

• Assumption: The log-likelihood function is uniformly Lipschitz continuous with respect to the distributional parameters $\phi$.
• Result: The estimator converges almost surely to the true parameter.
  $$\hat{\phi}_N \xrightarrow{a.s.} \phi^*$$
  This implies that with probability one, the estimated distribution collapses exactly onto the true simulator parameters as data grows.

3. Practicality and Relaxations (Section 6)

The authors analyze the practicality of their assumptions and provide relaxations to broaden applicability:

• i.i.d. Assumption: The requirement for independent and identically distributed data can be relaxed to strict stationarity and ergodicity, which holds for data collected by fixed behavior policies.
• Mixture Positivity: The strict positivity assumption can be relaxed to a logarithmic tail condition, allowing for light-tailed families (e.g., Gaussian) where the density may approach zero.
• Lipschitz Continuity: A sufficient condition for the Lipschitz requirement is provided: the simulator's transition kernel must be twice continuously differentiable ($C^2$) with bounded gradients and Hessians with respect to physical parameters. This is satisfied by simulators governed by smooth physics.
• Partial Coverage: In cases where the dataset does not cover the entire state-action space, the estimator converges to an identified set $Q^*_\mu$ of parameters that are observationally indistinguishable given the data, rather than a single point.

Significance and Claims

The paper claims to place Offline Domain Randomization on a principled footing. By proving consistency, it validates the empirical observation that ODR methods (like DROPO) outperform hand-tuned uniform DR. The work bridges the gap between empirical success and theoretical understanding, demonstrating that offline logs are not merely passive datasets but powerful tools for principled domain randomization.

The authors explicitly state that their results clarify when offline data can soundly guide the choice of a randomization distribution for downstream offline RL. They do not claim to solve the sim-to-real gap universally but provide the statistical guarantees necessary to justify the use of ODR in settings where the underlying assumptions (or their relaxations) hold. The work unifies empirical successes with theoretical guarantees in a single framework, offering diagnostics and relaxations to justify ODR's applicability across a broader range of settings.