How Should a Simulation-to-Reality Transfer Budget Be Spent?

Technical Summary: How Should a Simulation-to-Reality Transfer Budget Be Spent?

Problem Statement
In robot learning, the "reality gap" arises when policies trained in simulation fail on physical hardware due to discrepancies between simulated and true dynamics. Two primary strategies address this: System Identification (ID), which uses real-robot data to estimate physical parameters and tune the simulator, and Domain Randomization (DR), which trains policies over a distribution of randomized simulators to learn robust behaviors. Both strategies compete for the same scarce resource: the limited budget of real-robot interaction time (rollouts). While existing literature often studies these methods in isolation, the tradeoff between allocating budget to precise measurement (ID) versus justifying a broader randomization distribution (DR) remains poorly understood. This paper frames the choice as a budget-allocation problem: given a fixed number of real-robot rollouts, how should they be divided between identification and randomization width?

Methodology
The authors conducted a controlled sim-to-sim study to isolate this tradeoff without the confounding variables of physical hardware.

• Setup: A "Reality" simulator ($R$) with hidden, fixed parameters (bob mass $m$ and rod length $\ell$) served as the target system. A "Training" simulator ($T$) allowed these parameters to be adjusted. The true parameters were set to $\theta^* = (2.0, 1.5)$, significantly offset from the nominal prior $\theta_0 = (1.0, 1.0)$.
• Budget Allocation: The study varied two independent variables:
  1. $n$ (Identification Budget): The number of rollouts on $R$ used to estimate parameters. These rollouts utilized a random-torque excitation policy. Parameters were estimated via grid search minimizing one-step angular-velocity prediction error, subject to Gaussian sensor noise ($\sigma=1.0$).
  2. $w$ (Randomization Width): The width of the uniform distribution used to randomize parameters during training, defined as a fraction of the point estimate $\hat{\theta}$. $w=0$ implies training at the point estimate; larger $w$ implies a wider band centered on $\hat{\theta}$.
• Training & Evaluation: For each $(n, w)$ configuration, a Soft Actor-Critic (SAC) policy was trained for 75,000 steps. Performance was evaluated via zero-shot return on the hidden target system $R$ over 20 deterministic episodes.
• Scope: The study swept $n \in \{0, 5, 10, 25, 50\}$ and $w \in \{0, 0.1, 0.2, 0.3, 0.5\}$ across ten seeds, totaling 250 runs. Statistical significance was assessed using paired-bootstrap confidence intervals.

Key Results
The experiments revealed a distinct hierarchy in the value of measurement versus randomization breadth:

1. Measurement Dominance: The allocation of real-robot rollouts to identification ($n$) was the primary driver of performance. The transfer gap closed sharply with the first few rollouts. Specifically, moving from $n=0$ to $n=10$ yielded a massive improvement in return (approx. +427 points), whereas increasing $n$ beyond 10 provided diminishing returns.
2. Point Estimates Outperform Randomization: Once any real data was available ($n > 0$), training at the estimated point parameters ($w=0$) significantly outperformed training over any widened randomization band. Even a narrow band ($w=0.2$) resulted in statistically lower returns compared to $w=0$.
3. Randomization Cannot Substitute for Measurement: Broad randomization that theoretically contained the true system (e.g., a prior-range baseline covering $[0.3, 3.0]$) failed to substitute for measurement. Policies trained with $n=0$ and wide randomization performed poorly (return $\approx -749$), comparable to the $n=0, w=0$ baseline, and significantly worse than the $n=50, w=0$ "pure fidelity" baseline (return $\approx -337$).
4. Robustness: These findings held across different reality gaps (small, main, large) and sensor noise levels. While the specific "knee" of the budget curve shifted with noise levels (e.g., requiring more rollouts at higher noise), the qualitative ordering remained: measurement is critical, and widening the band after measurement is detrimental.

Contributions and Significance
The paper does not propose a new algorithm but rather provides a measurable characterization of the identification-versus-randomization tradeoff under a fixed budget.

• Formalization: It formulates the choice between ID and DR as a single budget-allocation problem and maps the full return surface.
• Empirical Evidence: It demonstrates that in a regime where dynamics are identifiable and the simulator family contains the target system, measurement dominates breadth. A small number of identification rollouts closes the majority of the transfer gap.
• Strategic Recommendation: The authors conclude that sim-to-real pipelines should prioritize measuring parameters that can be identified within the simulator's model class. Domain randomization should be reserved only for the uncertainty that remains after measurement, rather than used as a substitute for it.

Limitations and Scope
The authors explicitly note that these results apply to a "benign regime" where:

• The simulator is well-specified (no structural model mismatch).
• Only two parameters are unknown.
• The true system is representable by the simulator family.

The paper acknowledges that in real-world scenarios involving structural mismatches (e.g., unmodeled friction, backlash, or contact dynamics) where no parameter fit can perfectly reproduce the target, the value of randomization breadth may increase. The current findings serve as a baseline for that harder setting, suggesting that when parameters can be identified, fidelity should precede randomization.