🤖 machine learning

HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents

HERAKLES is a hierarchical agent that combines a high-level LLM policy for selecting achievable subgoals with a low-level controller that compiles successful behaviors into reusable skills, enabling efficient open-ended exploration and skill composition in partially observable environments with sparse rewards.

Original authors: Thomas Carta, Clément Romac, Loris Gaven, Pierre-Yves Oudeyer, Olivier Sigaud, Sylvain Lamprier

Published 2026-07-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Carta, Clément Romac, Loris Gaven, Pierre-Yves Oudeyer, Olivier Sigaud, Sylvain Lamprier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Quest for the Self-Teaching Robot

Imagine a world where computers don't just follow a strict recipe written by a human, but instead learn to cook, build, and explore on their own, getting smarter every day. This is the dream of "open-ended learning" in artificial intelligence. For a long time, AI has been like a student who memorizes a giant textbook but freezes when asked a question that isn't in the book. Real humans, however, are different. We don't just memorize; we build a library of skills. When we learn to ride a bike, we don't just learn to balance; we learn to pedal, steer, and brake as separate tools we can mix and match later to learn how to ride a motorcycle.

The paper you are about to read tackles a specific problem in this field: how do we teach an AI agent to keep learning forever without getting stuck? The key idea is "hierarchical learning." Think of it like a company with a boss and a worker. The boss (the high-level brain) doesn't do the heavy lifting; instead, they break big, scary projects into smaller, manageable tasks for the worker (the low-level brain). The worker does the tasks, and once they get really good at them, those tasks become "skills" the boss can just order with a single word. The paper explores how to build a robot that can do this automatically, turning a chaotic mess of new challenges into a neat, organized toolbox of abilities.

The Story of HERAKLES: The Robot That Builds Its Own Toolbox

Meet HERAKLES, a new kind of robot brain designed to be a lifelong learner. Imagine you are a young adventurer trying to conquer a massive, mysterious forest. At first, you only know how to take one step at a time. If you want to reach a distant mountain, you have to figure out every single step from scratch, which is exhausting and slow. You might get tired and give up before you even reach the first hill.

HERAKLES solves this by acting like a smart team with two distinct roles. The first role is the General (the high-level policy), who is a very smart, language-savvy AI. The second role is the Soldier (the low-level policy), who is a fast, efficient robot that actually moves the body and picks up items.

Here is how the magic happens:

1. The General Plans, The Soldier Executes
When the General sees a huge goal like "Build a Stone Pickaxe," they don't try to do it all at once. Instead, they look at what the Soldier can already do. Maybe the Soldier knows how to "chop wood" and "place a table." The General breaks the big goal down: "First, chop wood. Then, place a table. Finally, craft the pickaxe." The General gives these smaller orders to the Soldier.

2. The "Skill Compilation" Trick
This is where HERAKLES gets really clever. Every time the Soldier successfully finishes a task, like "chop wood," the General doesn't just forget about it. They take that whole sequence of movements and "compile" it into a single, reusable skill. It's like the Soldier learns to ride a bike, and suddenly, the General can just say "Ride to the river" instead of saying "Pedal, steer, balance, pedal, steer..."

As the robot learns more, its toolbox of skills grows. The General starts having access to bigger and bigger "options." Instead of taking 100 tiny steps to build a house, the General might just say "Build a wall" (a skill the Soldier has already mastered) and "Build a roof" (another mastered skill). This makes the robot much faster and more efficient.

3. The Safety Filter
The General is smart enough to know what the Soldier can't do yet. If the Soldier hasn't learned to "make fire" yet, the General won't ask them to do it, because that would just lead to failure and confusion. The system uses a special "competency checker" to only offer the General goals that the Soldier has a good chance of succeeding at. This keeps the learning process smooth and prevents the robot from getting stuck on impossible tasks.

What They Found

The researchers tested HERAKLES in a digital world called Crafter, which is like a 2D version of Minecraft where you have to gather resources and craft items to survive. The goal was to see if HERAKLES could learn to do harder and harder tasks over time without needing a human to teach it every step.

They compared HERAKLES to other AI methods:

The "Do-It-All" Robot: A robot that tries to learn every tiny step from scratch every time. This robot got stuck quickly because complex tasks took too long to learn.
The "Frozen Brain" Robot: A robot that uses a smart brain but keeps its body's skills fixed, never learning new physical tricks.
The "Hidden Code" Robot: A robot that tries to learn skills but uses a secret code instead of clear language, making it hard to combine skills logically.

The Results:
HERAKLES was the clear winner in the long run. While the other robots got stuck after learning a few easy tasks (like just walking to a tree), HERAKLES kept getting better. By the end of the training (after 250,000 steps in the game), HERAKLES had learned to craft complex items like a stone pickaxe, while the others were still struggling with basic tasks.

The paper suggests that HERAKLES is much better at "sample efficiency," meaning it learns more from fewer tries. It also showed amazing flexibility. When the researchers asked HERAKLES to do things it had never seen before, like "collect 4 woods" instead of just "collect wood," or to use a synonym like "acquire wood" instead of "collect wood," it succeeded almost every time. It could even figure out how to make a "wood sword" just by realizing it was very similar to the "wood pickaxe" it had already learned to make.

Why This Matters

The authors suggest that this method is a big step toward creating AI that can truly grow and adapt on its own. By letting the AI build its own library of skills and then use those skills to tackle even harder challenges, HERAKLES avoids the "stuck" feeling that plagues many other learning systems.

However, the paper also admits there are limits. Right now, the General only understands text, not pictures, so it can't see the world directly. Also, the system needs a list of goals to start with; it doesn't invent the goals itself yet. But the idea of a robot that can "compile" its own experiences into new, reusable skills is a powerful concept. It suggests a future where AI agents don't just memorize the past, but actively build a foundation for an endless future of learning.

Technical Summary: HERAKLES – Hierarchical Skill Compilation for Open-ended LLM Agents

Problem Statement

The paper addresses the challenge of open-ended learning in partially observable environments characterized by sparse rewards and large, structured goal spaces. While recent foundation models have achieved human-level performance on static tasks, they struggle with lifelong growth in complexity because they rely on fixed datasets. In contrast, humans continuously acquire diverse skills by decomposing complex tasks into simpler, previously internalized subskills.

Current approaches to open-ended agents (e.g., Voyager, AutotelicLLM) often face a combinatorial explosion in goal complexity. As agents encounter increasingly complex goals, the time required to master each new goal in isolation grows rapidly, causing learning progress to stall. Existing Hierarchical Reinforcement Learning (HRL) methods often rely on predefined skill sets or pre-trained policies, which are incompatible with truly open-ended agents that must autonomously discover and expand their skill repertoire without expert intervention.

The core problem is how to enable an agent to continuously adapt to a growing goal space by efficiently composing new skills from mastered behaviors, without relying on static datasets or expert-defined primitives.

Methodology: HERAKLES

The authors propose HERAKLES (HiERarchicAl sKill compiLation for open Ended llm agentS), a hierarchical agent that jointly learns a high-level policy and a low-level controller. The system operates on the assumption that the goal space admits prerequisite relations, allowing tasks to be decomposed into subgoals.

Architecture

The agent consists of two concurrently trained policies:

High-Level Policy ( $\pi_{HL}$ ): Implemented as a Large Language Model (LLM) (specifically Mistral 7B with LoRA adapters). It acts as a planner and explorer, selecting subgoals based on textual prompts derived from observations and task goals. It operates over a restricted set of "callable" options—subgoals the low-level policy is estimated to reliably achieve.
Low-Level Policy ( $\pi_{LL}$ ): A lightweight, computationally efficient neural network (ResNet-based) that executes primitive actions. It is trained to directly solve goals and progressively compile successful trajectories into reusable skills.

Key Mechanisms

Hierarchical Compilation: When the hierarchical agent successfully solves a goal $g$ , the full trajectory of primitive actions is "compiled" into $\pi_{LL}$ . Once mastered, $\pi_{LL}$ can execute $g$ directly as a single skill. $\pi_{HL}$ can then invoke this new skill as a primitive for more complex goals, effectively reducing the signal-to-noise ratio for higher-level planning.
Adaptive Subgoal Filtering: To prevent $\pi_{HL}$ from selecting unachievable subgoals, a competency estimator ( $C_\theta$ ) filters the goal space. This estimator, trained on low-level trajectories, predicts the probability that $\pi_{LL}$ can achieve a specific goal from the current state. $\pi_{HL}$ only samples from goals with a high estimated success probability, ensuring stable learning.
Bilevel Optimization: The system is framed as a bilevel problem with a Stackelberg structure. $\pi_{HL}$ $π_{H L}$ shapes the distribution of states and subgoals seen by $\pi_{LL}$ $π_{LL}$ , while $\pi_{LL}$ $π_{LL}$ 's execution determines the optimality of the high-level decomposition. The two policies are trained simultaneously:
- $\pi_{HL}$ is trained via on-policy RL (POAD) on high-level decision trajectories.
- $\pi_{LL}$ is trained via off-policy RL (AWR) using two types of data: full hierarchical rollouts (for skill compilation) and subgoal segments (for skill reinforcement).

Training Dynamics

The training process follows an automatic curriculum. Initially, $\pi_{LL}$ only masters atomic actions. As it masters goals, these are added to its repertoire. $\pi_{HL}$ then uses these mastered goals as building blocks to tackle more complex objectives. This creates a feedback loop where the low-level policy's growing competence expands the high-level policy's effective action space.

Key Contributions

Joint Learning Framework: A method for open-ended agents that jointly trains an LLM-based high-level policy and a low-level controller without requiring pre-specified skills or extensive pre-training.
Dynamic Skill Compilation: A mechanism where successful behaviors are automatically distilled into the low-level policy, enabling scalable skill composition as the agent's competence grows.
Adaptive Filtering: A competency-based filtering mechanism that restricts the high-level policy's search space to achievable subgoals, stabilizing learning in complex, open-ended environments.
Language-Structured Abstraction: Leveraging the compositional properties of natural language to define the goal space and facilitate the discovery of prerequisite chains between goals.

Experimental Results

The method was evaluated in the Crafter environment, a procedurally generated 2D world with an achievement tree of 17 goals requiring artifact collection and crafting.

Sample Efficiency: HERAKLES outperformed baselines (POAD, FUN, BOSS) in the long run. While baselines initially progressed faster, they plateaued after mastering easy goals. HERAKLES showed gradual, sustained improvement as more goals were compiled into $\pi_{LL}$ , eventually reaching complex goals (e.g., "make stone pickaxe") that baselines failed to discover.
Generalization: HERAKLES demonstrated superior generalization to unseen goals compared to baselines:
- Compositional Generalization: It maintained high success rates on goals requiring repetition of known tasks (e.g., "collect 4 woods" vs. "collect wood"), whereas baselines failed significantly.
- Synonymous Goals: It handled synonymous verbs effectively.
- Trajectory-Similar Goals: It successfully transferred skills to unseen goals sharing similar prerequisite structures (e.g., "make wood sword" after learning "make wood pickaxe").
Skill Expansion: The number of low-level skills callable by $\pi_{HL}$ increased steadily over training, confirming the system's ability to build a reusable skill library.

Significance and Claims

The paper claims that HERAKLES provides a viable path toward autotelic agents capable of lifelong learning. By combining the reasoning and compositional capabilities of LLMs with the execution efficiency of specialized low-level controllers, the system overcomes the "complexity barrier" that stalls open-ended learning.

The authors emphasize that their approach:

Does not rely on expert-defined skills or pre-trained policies.
Enables compositional generalization by treating mastered goals as primitives for future learning.
Improves sample efficiency in open-ended settings by amortizing past learning into reusable skills.

The work is presented as a step toward Artificial General Intelligence (AGI), specifically addressing the need for agents that can autonomously generate, select, and train on their own goals to continuously expand their capabilities. The authors acknowledge limitations, such as the current reliance on textual goals (suggesting future work with Vision-Language Models) and the computational cost of periodic retraining, but position the method as a foundational step toward fully autonomous, open-ended agents.