🤖 AI

Scalable Option Learning in High-Throughput Environments

This paper introduces Scalable Option Learning (SOL), a highly efficient hierarchical reinforcement learning algorithm that achieves approximately 35 times higher throughput than existing methods and demonstrates superior performance and positive scaling trends through training on 30 billion frames in complex environments like NetHack.

Original authors: Mikael Henaff, Scott Fujimoto, Michael Matthews, Michael Rabbat

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mikael Henaff, Scott Fujimoto, Michael Matthews, Michael Rabbat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to navigate a massive, complex dungeon filled with monsters, traps, and treasure. This is a classic problem in Reinforcement Learning (RL), where an agent learns by trial and error.

The problem is that the dungeon is huge. If you tell the robot, "Move your left foot, then your right foot, then turn your head," it gets overwhelmed. It's like trying to write a novel by deciding every single pixel of every page; the robot gets stuck in local loops (like walking in circles) and never learns the big picture.

Hierarchical Reinforcement Learning (HRL) is the idea of solving this by breaking the task into layers. Instead of controlling feet, the robot has a "Manager" who says, "Go fight the monsters," and a "Worker" who actually figures out how to move its feet to do that.

However, until now, these "Manager-Worker" systems were slow and clumsy. They couldn't handle the massive amounts of data needed to learn truly complex tasks. They were like a small bakery trying to bake bread for a whole city; they just couldn't scale up.

The Solution: Scalable Option Learning (SOL)

The authors of this paper built a new system called Scalable Option Learning (SOL). Think of SOL as upgrading that small bakery into a massive, automated industrial factory.

Here is how they did it, using simple analogies:

1. The "One-Size-Fits-All" Brain (Architecture)

Old hierarchical systems were like having a separate brain for the Manager and a separate brain for every single Worker. When you have 100 Workers, you need 101 brains, and they all have to talk to each other constantly. This is slow and messy.

SOL's trick: They built one single brain that can act as the Manager or any of the Workers.

The Analogy: Imagine a Swiss Army knife. It's one tool, but depending on which "flag" (a tiny switch) you flip, it becomes a screwdriver, a knife, or a corkscrew.
In SOL, the neural network (the brain) is the same, but a tiny "index" tells it: "Right now, you are the Manager deciding what to do next," or "Right now, you are the 'Fight' Worker moving the feet." This allows the computer to process thousands of scenarios at once, massively speeding things up.

2. The "Flexible Shift" (Adaptive Length)

In old systems, a Worker might be told, "Go fight for exactly 10 steps, then stop." But what if the fight ends in 3 steps? Or what if it needs 50? Being rigid causes problems.

SOL's trick: The Manager doesn't just pick what to do; it also picks how long to do it.

The Analogy: Imagine a construction foreman. Instead of saying, "Build this wall for 10 minutes," the foreman looks at the wall and says, "Build until the wall is done, or for 5 minutes, whichever comes first."
SOL learns to choose between short bursts (like checking a corner) or long stretches (like exploring a whole room), adapting automatically to the situation.

3. The "Instant Feedback" Loop (Bootstrapping)

Usually, in these systems, a Worker gets confused because it doesn't know if it did a good job until the Manager gives a final score at the very end of the day. This is like a student taking a test but not getting a grade until the end of the semester.

SOL's trick: They created a way for the Worker to get a "practice grade" immediately after it finishes its specific task, even if the whole episode isn't over.

The Analogy: It's like a video game where you get a "Combo Score" immediately after defeating an enemy, rather than waiting until you beat the final boss to see if you played well. This helps the Worker learn much faster.

The Results: Speed and Smarts

The authors tested SOL on NetHack, a notoriously difficult, old-school video game that is essentially a giant, randomly generated dungeon. It is so complex that even top AI models struggle with it.

Speed: SOL was 35 to 580 times faster than previous hierarchical methods. It could process data at a rate comparable to "flat" (non-hierarchical) agents, which was previously impossible for this type of system.
Scale: They trained SOL on 30 billion frames of experience. To put that in perspective, most previous hierarchical agents were trained on only millions of frames. It's the difference between reading a few pages of a book versus reading the entire library of Congress.
Performance: SOL significantly outperformed "flat" agents (robots that try to learn everything at once) and other hierarchical methods.
- In a test called ZombieHorde, where the agent had to fight zombies and retreat to heal, SOL learned the strategy of "fight until hurt, then run to heal." Flat agents just kept fighting until they died.
- In TreasureDash, where the agent had to choose between grabbing gold or going to the exit, SOL learned the perfect balance of gathering gold and then leaving at the right time.

Why This Matters (According to the Paper)

The paper claims that for a long time, Hierarchical RL was stuck in the "small data" era. It was a promising idea, but it couldn't handle the massive scale required for modern AI.

SOL proves that you can scale hierarchical learning. By combining a smart "one-brain" architecture with flexible timing and better feedback loops, they unlocked the ability to train complex, multi-layered agents on billions of examples.

In short: They took a slow, clunky system that tried to manage a team of workers, and turned it into a high-speed, automated factory that can learn complex strategies by reading billions of pages of experience, all while keeping the "Manager" and "Worker" roles distinct and effective.

Technical Summary: Scalable Option Learning in High-Throughput Environments

Problem Statement

Hierarchical Reinforcement Learning (HRL) offers a promising framework for solving decision-making tasks spanning long timescales by decomposing problems into sub-tasks (options) managed by a higher-level controller. However, existing hierarchical methods have largely failed to realize the benefits of large-scale training that have driven progress in flat RL and other machine learning domains. While flat agents and vision models are routinely trained on billions of samples, hierarchical agents are typically restricted to millions of samples due to significant computational bottlenecks.

The primary challenge in scaling HRL is the difficulty of parallelizing hierarchical execution. In a batch of trajectory segments, different time steps often correspond to different policies (the controller or various options), each with distinct reward functions and termination conditions. This heterogeneity prevents the straightforward application of high-throughput RL libraries (e.g., IMPALA, Sample Factory) which rely on uniform batch processing. Consequently, current hierarchical methods often process single trajectories sequentially, limiting their throughput and preventing them from leveraging the "scaling laws" observed in other areas of AI.

Methodology: Scalable Option Learning (SOL)

The authors propose Scalable Option Learning (SOL), a hierarchical policy gradient algorithm designed to optimize both performance and computational throughput. SOL addresses the scaling bottleneck through specific algorithmic and systems-level innovations.

1. Algorithmic Contributions

Joint Optimization with Value Bootstrapping: SOL jointly optimizes actor-critic objectives for the controller and all options. A key innovation is the bootstrapping of option value functions upon option switches. Instead of waiting for an option to terminate naturally to compute returns, the algorithm approximates the value of an option $\omega$ at state $s_t$ using a $K$ -step return plus a bootstrapped estimate of the value at the switch point ( $s_{t+K+1}$ ). This allows option policies to optimize independent intrinsic rewards ( $R_\omega$ ) even when trained on trajectories generated by mixed behavior policies.
Adaptive Option Lengths: The controller does not only select the next option $\omega$ but also outputs an execution length $l$ from a set of exponentially spaced values ( $L = \{2^i\}$ ). This is implemented via a softmax head trained alongside the option selection head. This mechanism allows the agent to dynamically adjust the temporal abstraction of sub-tasks based on the current observation, avoiding the "option degeneracy" problem where controllers collapse to selecting only length-1 options.
Reduced Non-Stationarity: The objective function is designed such that the advantage and value functions for a specific option depend only on its own reward and the state distribution induced by the system, rather than the specific policies of other options or the controller after the option terminates. This limits inter-dependencies and reduces non-stationarity during training.

2. Systems-Level Contributions

To achieve high throughput, SOL restructures the training pipeline to enable massive parallelization on GPUs:

Unified Neural Network: A single neural network represents all policies (controller and all options). The network takes an observation and a one-hot vector indicating the active policy as input. It outputs distributions over environment actions, options, and option lengths. This allows for batched forward passes across all policies simultaneously.
Environment Wrapper: An actor-side wrapper tracks the active policy index, handles policy switching based on termination conditions, and computes option-specific rewards. It duplicates observations when the controller is called to ensure the selected option receives the correct state input.
Parallelized Return Computation: The learner worker computes advantages and value targets for all policies simultaneously using tensorized operations. By iterating backward through the time dimension and using policy indices to route rewards, the system computes returns for mixed-policy batches in a single pass. V-trace corrections are applied to handle off-policy data from asynchronous actors.

Key Results

The authors evaluate SOL across three environments: MiniHack, NetHack, and Mujoco (PointMaze).

Throughput: On the NetHack Learning Environment (NLE), SOL achieves a throughput of 43,000 steps/second, representing a 35x to 580x speedup compared to existing hierarchical methods (HIRO, Option-Critic, MOC, Hierarchical RLLib). It retains 86% of the throughput of a flat APPO agent.
NetHack Performance: Trained on 30 billion frames, SOL significantly outperforms flat agents (APPO) and other hierarchical baselines.
- In the NetHackScore environment, SOL steadily improves and surpasses flat agents.
- When combined with Motif (an LLM-based intrinsic reward synthesizer), SOL+Motif sets a new state-of-the-art on NetHackScore.
- Ablation studies confirm that both the hierarchical structure and the specific option rewards are necessary; hierarchical agents trained only on task rewards (SOL+HiPPO) fail to outperform flat baselines.
MiniHack Credit Assignment: In ZombieHorde (requiring healing vs. fighting trade-offs) and TreasureDash (requiring balancing gold collection vs. escaping), flat agents and flat agents with option rewards fail to escape local optima. SOL successfully learns to coordinate distinct options (e.g., a "Health" option for healing and a "Score" option for combat) to achieve near-optimal performance.
Continuous Control: On a custom G-maze (PointMaze) designed to trap agents in local optima due to a deceptive reward landscape, SOL is the only method to make progress (achieving ~70% success), while flat agents and other hierarchical methods remain at zero success.
Automatic Reward Discovery: The paper demonstrates that SOL is compatible with automatic reward synthesis methods like DIAYN. SOL+DIAYN learns effective policies without hardcoded option rewards, though it converges slightly slower than the version with handcrafted rewards.

Significance and Claims

The paper claims to present the first online hierarchical RL algorithm capable of scaling to billions of samples. Its significance lies in bridging the gap between the theoretical benefits of hierarchy and the practical requirements of modern large-scale training.

Scalability: The work demonstrates that hierarchical RL can be made computationally efficient enough to compete with flat RL in terms of throughput, enabling the training of agents on datasets orders of magnitude larger than previously possible for HRL.
Necessity of Hierarchy + Rewards: The experiments suggest that for difficult credit assignment problems, hierarchy alone is insufficient; it must be paired with useful intrinsic rewards (either handcrafted or learned) to unlock performance gains.
General Applicability: The method is shown to be effective across discrete, open-ended environments (NetHack) and continuous control tasks (Mujoco), indicating its potential as a general-purpose framework for long-horizon decision-making.

The authors note that while the system is optimized for computational efficiency in sample-rich regimes (like video games and simulators), it is not currently optimized for sample efficiency in data-scarce settings. Future work is suggested to explore general methods for learning option rewards and potentially integrating these scalable designs into model-based RL frameworks.