🤖 machine learning

Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraints

This paper argues that evaluating reinforcement learning algorithms for transfer tasks requires looking beyond sample efficiency to include practical wall-clock training time and robustness under domain randomization, demonstrating that the sample-inefficient PPO can outperform more sample-efficient algorithms like SAC and TD-MPC2 in speed while all three paradigms benefit similarly from domain randomization.

Original authors: Hany Hamed, Abhishek Naik, Colin Bellinger, A. Rupam Mahmood

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hany Hamed, Abhishek Naik, Colin Bellinger, A. Rupam Mahmood

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots learn to walk, dance, or catch balls not by being programmed with rigid rules, but by playing a game of trial and error, much like a toddler learning to walk. This field is called Reinforcement Learning (RL). In this digital playground, an AI agent tries different moves, gets points for doing well, and loses points for falling over. Over time, it figures out the best way to move. But here's the catch: most of this learning happens in a perfect, fake computer world (a simulator). The real goal is to take that learned skill and use it in the messy, unpredictable real world. This is called "transfer."

To make this transfer work, scientists usually measure how "sample efficient" an algorithm is. Think of this like counting how many times a student has to flip a page in a textbook to learn a concept. If Algorithm A needs 1,000 page flips and Algorithm B needs 10,000, we usually say Algorithm A is the winner. But this paper asks a different, very practical question: What if the student has a ticking clock? What if the real constraint isn't how many pages they can flip, but how fast they can read them? In modern computing, we can run thousands of simulations at once, like having a thousand students reading the same book simultaneously. This means an algorithm that needs more "flips" might actually finish the book faster if it has a bigger team. This paper explores whether our usual way of ranking these learning algorithms is missing the bigger picture of what actually works in the real world.

The researchers, a team from universities and research institutes, decided to put three popular learning algorithms to the test: PPO (a method known for being stable and good at using many computers at once), SAC (a method famous for needing fewer practice attempts), and TD-MPC2 (a smart method that tries to predict the future to learn faster). They wanted to see if the "winner" changed depending on how you measured them.

First, they looked at the "wall-clock time" versus the "number of tries." In their simulations, they set up a race. When they counted only the number of practice attempts (interactions), SAC and TD-MPC2 were the clear champions, learning the tasks with fewer tries than PPO. It was like seeing a student who memorized the book in half the time. However, when they switched the stopwatch to measure actual time, the story flipped. Because PPO was set up to run on 2,048 parallel environments (imagine 2,048 students reading the book at the exact same second), it gathered its massive amount of practice data so quickly that it produced a working robot policy much faster in real-time than the other two. The paper suggests that if you are a robot engineer with a deadline, the "slower" learner (PPO) might actually get you to the finish line sooner because it scales up better with powerful computers.

Next, the team tackled the problem of "domain randomization." This is a technique where you teach the robot in a simulator that changes slightly every time—maybe the floor is slippery one moment, or the robot's legs are heavier the next. The goal is to make the robot robust enough to handle the real world, where things are never perfect. There was a common belief that some learning styles (like the complex, predictive TD-MPC2) might get confused or fail when the training environment is this chaotic, while others (like PPO) were thought to be the only ones that could handle it.

The researchers tested this by training all three algorithms with five different levels of chaos, from "narrow" (tiny changes) to "extensive" (wildly different physics). They found that the idea that one algorithm is inherently better at handling chaos is a myth. In their simulations, PPO, SAC, and TD-MPC2 all benefited from domain randomization, but the results were a mixed bag. Sometimes, a little bit of chaos helped SAC the most; other times, a lot of chaos helped TD-MPC2. There was no single "best" algorithm for all situations. The paper concludes that the success of using random training depends heavily on the specific task, the specific algorithm, and how much chaos you introduce. It's not about picking the "strongest" robot; it's about tuning the training environment to fit the job.

In the end, the paper argues that we need to stop looking at sample efficiency as the only scorecard. Just because an algorithm learns with fewer interactions doesn't mean it's the best choice for a real-world project. If you have a time limit and access to powerful computers, a "less efficient" algorithm might be the practical winner. And when it comes to making robots robust against real-world surprises, there isn't a one-size-fits-all solution; the best approach depends on the specific puzzle you are trying to solve. The authors suggest that future engineers should weigh the time it takes to train a robot just as heavily as the number of times it has to practice.

Technical Summary: Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraints

Problem Statement

Transfer-oriented reinforcement learning (RL) applications, particularly in sim-to-real or sim-to-sim pipelines, require evaluating algorithms along dimensions that extend beyond standard sample efficiency. Conventional evaluation protocols prioritize the number of agent-environment interactions required to reach a certain performance level. However, in practical transfer workflows, environment interactions are often inexpensive and scalable via parallel simulation, whereas wall-clock time remains a primary operational constraint.

Furthermore, the suitability of an algorithm for transfer depends on its robustness under dynamics mismatch. While domain randomization (DR) is a standard technique to improve robustness by exposing policies to variability in physical parameters (e.g., mass, friction, damping) during training, it remains unclear whether different RL paradigms (on-policy, off-policy, and model-based) respond differently to increasing variability in the training distribution. This paper addresses two core questions:

Do conclusions about algorithm suitability change when training is evaluated under wall-clock time budgets rather than interaction-based budgets?
Do different learning paradigms respond differently to increasing variability in the training distribution induced by domain randomization?

Methodology

The authors conducted a systematic, controlled comparison of three distinct RL algorithms across 18 continuous-control tasks from the DeepMind Control Suite (implemented via MuJoCo XLA). The algorithms represent three major paradigms:

PPO (Proximal Policy Optimization): An on-policy method, configured with massively parallel environments (2,048) to leverage modern simulation scalability.
SAC (Soft Actor-Critic): An off-policy, entropy-regularized actor-critic method, trained using standard single-environment configurations (CleanRL implementation).
TD-MPC2: A model-based method combining learned dynamics with online planning, trained using standard configurations (official codebase).

Evaluation Axes

The study evaluates these algorithms along two complementary axes:

Practical Efficiency (Wall-Clock Time): Performance is measured against the actual time required to train, reflecting operational constraints where iteration speed and deployment readiness are critical.
Interaction-Based Efficiency: Performance is measured against the number of environment interactions, reflecting conventional sample-efficiency benchmarks.

Transfer and Domain Randomization Setup

Transfer is modeled as generalization across context distributions within a Contextual Markov Decision Process (CMDP). The study investigates zero-shot sim-to-sim transfer under dynamics mismatch.

Training Distributions: The authors varied the breadth of the training distribution ( $p_{train}$ ) using five levels of domain randomization: Nominal (no randomization), Narrow, Moderate, Broad, and Extensive. Randomization was applied multiplicatively to physical parameters including mass, damping, actuator strength, friction, and link length.
Evaluation Regimes: Two fixed evaluation regimes were used to test generalization:
- Nominal-Centered: Contexts close to nominal dynamics (in-support for all training regimes).
- Shifted: Contexts with structured deviations from nominal parameters (ranging from out-of-support to in-support depending on training coverage).

Key Results

1. Wall-Clock vs. Interaction-Based Efficiency

The study reveals a significant divergence between interaction-based and wall-clock rankings:

Interaction-Based: Consistent with conventional benchmarks, SAC and TD-MPC2 generally demonstrate higher sample efficiency, reaching strong performance with fewer environment interactions than PPO.
Wall-Clock: Under time-based evaluation, PPO often reaches competitive or superior performance earlier than SAC and TD-MPC2. This is attributed to PPO's ability to exploit massively parallel simulation (2,048 environments), allowing it to accumulate the necessary interactions within a shorter wall-clock duration despite its lower sample efficiency per interaction.
Implication: The ranking of algorithms changes based on the evaluation metric. An algorithm deemed "sample inefficient" (PPO) can be "time-efficient" in parallelized settings.

2. Impact of Domain Randomization on Robustness

The analysis of domain randomization effects across the three algorithms yields several nuanced findings:

No Algorithm-Specific Advantage: There is no consistent evidence that any single paradigm (on-policy, off-policy, or model-based) is systematically more or less compatible with domain randomization. PPO, SAC, and TD-MPC2 all benefit from DR, but the magnitude and direction of the effect depend on the specific task, the algorithm, the coverage level, and the evaluation regime.
Non-Monotonic Gains: Increasing the breadth of domain randomization does not guarantee improved transfer performance.
- Under Nominal-Centered evaluation, Narrow randomization often outperforms Extensive randomization.
- Under Shifted evaluation, broader distributions (Broad/Extensive) do not consistently outperform narrower ones, even when the evaluation contexts fall within the training support.
Trade-offs: Broader randomization increases exposure to diverse dynamics but also increases the difficulty of the learning problem and reduces the concentration of experience in specific regions, creating a trade-off between distributional coverage and effective learning.

Significance and Claims

The paper argues that sample efficiency alone is an insufficient criterion for assessing RL algorithms in transfer-oriented applications. The authors claim two primary insights for practitioners:

Re-evaluating Efficiency: Practical constraints, specifically wall-clock time, should be a primary consideration. The common knowledge that PPO is "sample inefficient" does not preclude it from being the most practical choice in pipelines where massive parallelization is available. The "time-to-performance" metric can invert the standard algorithmic rankings derived from interaction-based benchmarks.
Context-Dependent Robustness: The effectiveness of domain randomization is not an inherent property of a specific algorithmic family (e.g., on-policy vs. off-policy). Instead, robustness is governed by the interaction between the task, the algorithm, the specific coverage of the training distribution, and the nature of the deployment mismatch. Consequently, practitioners should not assume that one paradigm is universally superior for DR; rather, they must empirically evaluate the combination of algorithm and randomization coverage for their specific transfer constraints.

The study concludes that algorithm suitability for transfer is a multi-dimensional problem requiring evaluation across both practical efficiency (time) and robustness (dynamics mismatch), rather than relying solely on interaction-based sample efficiency.