Technical Summary: Distinguishing Between Solving Simulators and Using Simulators as a Proxy

Problem Statement

Reinforcement Learning (RL) research predominantly relies on benchmark simulators to study general-purpose sequential decision-making. However, a critical ambiguity exists in the community regarding the fundamental goal of these experiments: are researchers attempting to solve a specific simulator (finding the optimal policy for a given simulation, often to solve an engineering or scientific problem), or are they using a simulator as a proxy for learning in deployment (using the simulation to develop algorithms intended for real-world environments where the agent cannot control the dynamics)?

The paper argues that the lack of distinction between these two use cases leads to "cheating" behaviors, misleading performance metrics, and algorithms that fail when transferred to deployment settings. Researchers often adopt techniques that exploit simulator-specific capabilities (e.g., parallel environments, state resetting) while claiming their work advances general RL for deployment, creating a gap between reported benchmark scores and real-world applicability.

Methodology and Framework

The authors propose a formal dichotomy based on the constraints imposed by the simulator's usage. They define a simulator as a software implementation of environment dynamics where interactions are consequence-free, allowing for actions impossible in the real world.

The paper establishes two distinct problem formulations:

1. Solving a Simulator:

   • Goal: Find the optimal policy for the specific simulation.
   • Constraints: The agent can utilize parallel environments (multiple asynchronous copies) and simulator-exclusive actions (resetting to arbitrary states, querying multiple outcomes from a single state-action pair).
   • Appropriate Algorithms: Multi-stream model-free algorithms (e.g., A2C, IMPALA, PQN), planning with learned models, and meta-learning for hyperparameters.
   • Evaluation Metrics: Best episodic return (expected return of the best policy found), optimality gap, and total computation used. The focus is on the quality of the final policy, not the rewards accumulated during the learning process.

2. Using a Simulator as a Proxy for Learning in Deployment:

   • Goal: Understand algorithms for learning in environments where dynamics are outside the practitioner's control (e.g., robotics, physical systems).
   • Constraints: The agent interacts with a single environment sequentially. Simulator-exclusive actions like resetting or parallel sampling are prohibited as they do not exist in deployment.
   • Appropriate Algorithms: Standard model-free algorithms (e.g., Sarsa, DQN, PPO) applied to a single stream of interaction.
   • Evaluation Metrics: Average episodic return, average reward across all learning steps, sample efficiency, and stability. The focus is on performance during learning (online return), as rewards incurred during exploration matter in deployment.

Key Contributions and Issues Identified

The paper identifies four specific issues arising from the conflation of these two settings, supported by simple experiments and theoretical analysis:

1. Parallel Environments in Proxy Settings:

   • Many modern algorithms (e.g., PPO, PQN) rely on parallel environments to accelerate training. When used as a proxy for deployment, this creates a misleading impression of speed and sample efficiency.
   • Evidence: Experiments with PQN on MinAtar show that while parallelism drastically reduces wall-clock time, switching to a single environment (to mimic deployment) results in significantly slower learning and lower performance unless the update-to-data ratio is carefully matched.

2. Suboptimal Performance from Ignoring Resetting:

   • Conversely, when the goal is solving a simulator, failing to utilize simulator-exclusive actions like resetting leads to suboptimal results.
   • Evidence: In the MountainCar-v0 environment, a PPO agent utilizing a naive resetting mechanism (resetting to random states periodically) learned a superior policy faster than a standard PPO agent. This performance gain is lost if the researcher does not recognize the goal as "solving the simulator" and thus does not employ resetting.

3. Hyperparameter Tuning Exploitation:

   • Standard hyperparameter tuning often involves discarding "bad" trials or resetting the environment for each configuration tested. This is impossible in deployment, where every trial incurs real-world cost.
   • Evidence: The paper analyzes the tuning process for the Rainbow algorithm on VizDoom. The standard practice of resetting the simulator for every hyperparameter sweep effectively turns the experiment into "solving the simulator," making the resulting baseline less comparable to an algorithm designed for deployment where such resets are impossible.

4. Evaluation Rollouts vs. Online Returns:

   • Using offline evaluation rollouts (freezing the policy and running it without exploration) to report performance is standard for solving simulators but misleading for deployment proxies.
   • Evidence: Experiments with SAC and DQN show that evaluation rollouts often report significantly higher and smoother returns than online returns. In a deployment setting (e.g., water treatment), the online return reflects the actual cost incurred during exploration, whereas the rollout metric hides the potential damage caused by exploratory actions.

Results

The paper does not propose a new algorithm but rather presents empirical demonstrations of the discrepancies caused by mixing these settings:

• Figure 2: Demonstrates that PQN's speed advantage vanishes when restricted to a single environment, highlighting the dependency on parallelism.
• Figure 3: Shows that leveraging resetting leads to a better final policy in a simulator-solving context, a gain that would be missed if the setting were treated as a deployment proxy.
• Figure 4: Illustrates the divergence between online performance and offline rollout performance, showing that relying on rollouts can mask the instability and cost of exploration inherent in learning in deployment.

Significance and Call to Action

The authors argue that the RL community must explicitly distinguish between these two use cases to ensure empirical rigor and practical relevance. The significance of this work lies in:

• Preventing Misleading Conclusions: Clarifying that an algorithm's success in a simulator does not guarantee its efficacy in deployment if the simulator was exploited via parallelism or resetting.
• Improving Algorithm Development: Encouraging researchers to select appropriate constraints and metrics for their specific goals. If the goal is deployment, algorithms should be tested under single-stream constraints with online metrics. If the goal is solving a specific simulation (e.g., for protein docking or game mastery), exploiting simulator features is valid and necessary.
• Community Standards: The paper calls for researchers to clearly state their problem setting (e.g., "solving a simulator" vs. "learning in deployment") in their papers. It suggests that venues should reward work that applies RL to intended use cases and that algorithm variants (e.g., PPO for single-stream vs. multi-stream) should be explicitly labeled to avoid confusion.

Ultimately, the paper posits that while both goals are valid, conflating them hinders the development of robust RL agents for real-world deployment and obscures the true capabilities and limitations of current algorithms.