💻 computer science

Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning

This paper provides an overview of hierarchical reinforcement learning by defining its benefits for decision-making challenges, categorizing methods for discovering temporal structure from online and offline data to large language models, and outlining current challenges and suitable application domains.

Original authors: Martin Klissarov, Akhil Bagaria, Ziyan Luo, George Konidaris, Doina Precup, Marlos C. Machado

Published 2026-09-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Martin Klissarov, Akhil Bagaria, Ziyan Luo, George Konidaris, Doina Precup, Marlos C. Machado

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where every decision you make, from tying your shoes to planning a career, required you to consciously calculate the movement of every single muscle fiber. You would be paralyzed by the sheer volume of details, unable to see the forest for the trees. This is the daily reality for artificial intelligence agents trying to learn in complex environments. They sense the world and act moment by moment, but to achieve anything meaningful, they must reason across long stretches of time. The challenge is not just learning what to do, but learning how to organize those actions into a coherent story. This is the domain of hierarchical reinforcement learning, a field dedicated to teaching machines to break down massive, overwhelming problems into smaller, manageable chunks, much like a human breaks a day into a series of distinct tasks.

A new comprehensive review by researchers from McGill University, Brown University, and the University of Alberta maps the current landscape of this field, offering a clear guide to how machines can discover these useful structures on their own. The authors argue that the key to solving complex, long-term problems lies in finding and exploiting "temporal structure"—patterns in time where certain sequences of actions naturally group together. Instead of forcing a machine to learn every tiny step from scratch, the goal is to help it discover reusable skills, or "options," that it can call upon again and again. The paper does not present a single new algorithm that solves everything; rather, it organizes a vast and diverse body of existing research to explain what makes a structure useful, how different methods discover these structures, and where the biggest hurdles remain.

The researchers begin by clarifying what makes a temporal structure "good." They draw a parallel to the way software engineers write code: a well-organized program uses modules that can be reused and combined to build complex applications. In the same way, an artificial agent benefits when it can learn a skill, like "open a door" or "pick up a key," and then use that skill as a building block for a larger goal, like "escape the maze." The paper identifies four main benefits that such structures provide. First, they help the agent explore the world more effectively by targeting specific milestones rather than wandering aimlessly. Second, they make it easier to figure out which actions led to a success or failure, a process known as credit assignment, by grouping long chains of events into single, understandable units. Third, they allow the agent to transfer knowledge from one situation to another, reusing a skill learned in one context for a different problem. Finally, they make the agent's decision-making process more transparent to human observers, allowing us to understand the "why" behind a machine's actions.

However, the authors are careful to note that this approach is not a magic bullet. There is a trade-off. Building a hierarchy of skills requires extra computation and time to discover the right structure in the first place. If the structure the agent discovers does not match the actual problem, it can actually slow down learning or lead to poor performance. The paper suggests that the best results come when the complexity of the task justifies the cost of building this internal organization. For simple, short tasks, a standard approach is often better. But for long, complex challenges where the agent must plan far into the future, the investment in discovering a hierarchy pays off.

The review then categorizes the various ways researchers have taught agents to find these structures, dividing them into three main sources of information. The first group learns directly from interacting with the world in real-time. Some of these methods look for "bottlenecks"—narrow passages or critical states that an agent must pass through to reach new areas, like a doorway in a house. By identifying these choke points, the agent can learn specific skills to cross them, effectively unlocking new parts of the environment. Other methods in this group use mathematical techniques to map the shape of the environment, finding natural groupings of states that the agent can navigate between. A third approach focuses on "empowerment," where the agent learns skills that give it the most control over its future, encouraging it to explore states where it has the most influence.

The second group of methods learns from large collections of data that already exist, rather than interacting with the world live. This is particularly useful when an agent cannot afford to make mistakes in the real world. By analyzing offline datasets, these algorithms can identify patterns and skills that were demonstrated by humans or other agents, effectively reverse-engineering a useful hierarchy from past experiences. They can relabel old data to find new goals, helping the agent learn from a wider variety of situations without needing new interactions.

The third and most recent frontier involves using foundation models, such as large language models, to provide prior knowledge. Instead of starting from zero, these methods use the vast knowledge already encoded in these models to suggest what skills might be useful or to help the agent understand the structure of a task. This allows the agent to begin its discovery process with a head start, leveraging human language and logic to guide its learning.

Despite these advances, the authors highlight significant challenges that remain. One major issue is "non-stationarity," a technical term for the fact that as the agent learns new skills, the environment it sees changes, making it difficult to learn multiple things at once without them interfering with each other. Another challenge is balancing the rewards: the agent must learn to value the immediate satisfaction of completing a subtask while keeping the ultimate goal in mind. The paper suggests that while we have many tools to discover these structures, we still lack a single, universal method that works for every situation.

The review concludes by pointing to the domains where hierarchical learning is most likely to succeed. These are open-ended environments where tasks are long, complex, and share underlying structures, such as robotics, web navigation, and complex video games. In these fields, the ability to compose and reuse skills is not just a luxury but a necessity. The authors suggest that the future of artificial intelligence lies in building agents that can autonomously ask the right questions about their world and efficiently find the answers, creating their own libraries of skills to navigate an increasingly complex reality. The work presented is a roadmap for that journey, clarifying what we know, what we are still trying to figure out, and why the effort to teach machines to think in layers is so crucial for the next generation of intelligent systems.

Technical Summary: Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning

Problem Statement

Reinforcement Learning (RL) agents face significant challenges when operating in complex environments that require reasoning over multiple timescales. While standard RL agents select primitive actions at every timestep, effective decision-making in long-horizon tasks often necessitates abstracting sequences of actions into higher-level constructs. The central problem addressed in this work is temporal structure discovery: how can agents autonomously identify and exploit useful temporal abstractions (such as options, skills, or goal-conditioned policies) without relying on predefined human expertise?

The paper posits that while Hierarchical Reinforcement Learning (HRL) offers a promising solution by decomposing tasks into modular and composable subtasks, the field lacks a unified understanding of what constitutes "good" structure. Furthermore, it is unclear in which problem settings identifying such structure yields benefits versus when the cost of discovery outweighs the gains. The authors aim to organize the diverse literature on HRL not by technical implementation details, but by the fundamental benefits these methods provide and the specific settings (online, offline, foundation models) in which they operate.

Methodology and Framework

The paper establishes a formal framework based on the Options formalism (Sutton et al., 1999b), where an option $o$ is defined as a triple $\langle I, \pi, \beta \rangle$ consisting of an initiation set, an intra-option policy, and a termination function. The authors introduce a generic algorithmic template (Algorithm 1) for HRL with option discovery, which iterates through three core operations:

Interact: Collecting experience from the environment.
OptionDiscovery: Constructing or refining the set of options and their objectives.
PolicyImprovement: Updating the high-level policy that selects among options.

The methodology categorizes existing discovery approaches based on the source of data and prior knowledge:

1. Discovery from Online Experience

These methods learn temporal structure through direct interaction with the environment. The paper reviews several families of approaches:

Bottleneck Discovery: Identifies states that act as gateways between regions of the state space (e.g., doorways). Methods include diverse density, graph partitioning (Min-Cut), and graph centrality measures (Betweenness).
Spectral Methods: Leverage the topology of the state space via the eigenvectors of the graph Laplacian or the Successor Representation (SR). These methods discover options (Eigenoptions) that facilitate efficient exploration and cover the state space.
Sequentially Composable Options: Focus on learning options that can be chained together (e.g., Skill Chaining). These methods learn backward from a goal, ensuring the termination region of one option overlaps with the initiation region of the next.
Empowerment Maximization: Frames skill discovery as maximizing the mutual information between an agent's actions (or skills) and future states. Methods like DIAYN and DADS learn diverse skills that are distinguishable by their effects on the environment.
Via Environment Rewards: Includes Feudal RL (where managers set goals for workers) and the Option-Critic framework, which learns option policies and termination functions directly by optimizing the environmental reward signal.
Curriculum and Hindsight Learning: Utilizes learning progress to select goals or employs Hindsight Experience Replay (HER) to relabel trajectories with alternative goals, effectively creating a curriculum of subtasks.

2. Discovery through Offline Datasets

These methods leverage pre-collected datasets to learn temporal abstractions without further environment interaction.

Variational Inference of Skill Latents: Treats skills as latent variables and uses variational inference to discover a set of skills that can explain the observed trajectories.
Hindsight Subgoal Relabeling: Adapts HER principles to offline settings, relabeling trajectories with subgoals to learn a diverse set of skills from a static dataset.

3. Discovery with Foundation Models

This emerging category leverages large pre-trained models (e.g., LLMs) to aid discovery.

Embedding Similarity: Uses embeddings from foundation models to identify semantically similar states or actions.
Providing Feedback: Utilizes foundation models to provide reward signals or critiques.
Reward as Code: Translates natural language or code descriptions into reward functions.
Directly Modeling the Policy: Uses foundation models to directly generate high-level policies or skill-selection logic via in-context learning.

Key Contributions

The paper makes several distinct contributions to the HRL literature:

Benefit-Centric Organization: Unlike previous surveys that categorize methods by algorithmic architecture, this work organizes the literature around the core benefits HRL provides: Exploration (targeting subgoals, extending time scales), Credit Assignment (propagating errors over abstract steps), Transfer (reusing skills across tasks), and Interpretability.
Trade-off Analysis: The authors explicitly analyze the trade-offs between performance, sample efficiency, and computational efficiency. They argue that HRL often trades some performance and computational cost for significant gains in sample efficiency, particularly in long-horizon tasks.
Unified Algorithmic Template: The introduction of Algorithm 1 provides a common language to describe diverse discovery methods, highlighting how different approaches instantiate the specification, learning, and improvement phases.
Identification of Challenges: The paper synthesizes major open challenges, including non-stationarity (learning multiple components simultaneously), the difficulty of simultaneously learning multiple behaviors, and the complexity of combining intrinsic and environmental rewards.
Domain Suitability: The authors argue that HRL is not a universal solution but is particularly well-suited for open-ended systems (e.g., Web Agents, Robotics, Open-ended games) where tasks share structural overlap (modularity and compositionality) and require long-horizon planning.

Results and Claims

The paper does not present new experimental results but synthesizes findings from the broader literature to support its claims:

Exploration: Methods like Eigenoptions and Empowerment Maximization have demonstrated the ability to significantly reduce diffusion time and improve coverage in complex environments (e.g., Atari 2600 games, MuJoCo).
Credit Assignment: Hierarchical approaches, particularly those using bottleneck discovery or option-critic, have shown improved sample efficiency in long-horizon tasks compared to flat RL, as they allow for value propagation over longer temporal jumps.
Transfer: Options learned in one context (e.g., navigating a specific room layout) have been shown to transfer effectively to new tasks with different goals or starting positions, provided the underlying transition structure remains similar.
Foundation Models: Early work suggests that foundation models can accelerate discovery by providing priors for reward functions or directly generating high-level plans, though generalization and robustness remain active research areas.

Significance and Conclusion

The paper concludes that the discovery of temporal structure is a "grand challenge" in AI, analogous to the invention of subroutines in computer science. The significance of this work lies in its attempt to clarify when and why HRL is beneficial, moving beyond the assumption that hierarchy is always superior.

The authors emphasize that HRL is most effective in domains characterized by modularity and compositionality, such as robotics and open-ended games, where tasks can be decomposed into reusable subtasks. They argue that the field is currently at an opportune moment to tackle the problem of option discovery, driven by advances in deep learning, the availability of large offline datasets, and the integration of foundation models. The ultimate goal is to build agents that can autonomously ask useful questions (define option objectives) and efficiently find answers (learn options), thereby unlocking sample-efficient reward maximization across a wide variety of complex tasks. The paper calls for future research to address the stability of learning multiple behaviors simultaneously and to develop reliable, general-purpose discovery algorithms.