💻 computer science

LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks

This paper proposes LEACL, a framework that leverages large language models to automatically decompose long-horizon manipulation tasks and generate necessary specifications, enabling reinforcement learning agents to achieve superior performance through automatic curriculum learning using only sparse rewards without the need for manually designed dense reward functions.

Original authors: Faraz Heravi, James Ouyang, Zifan Xu, Arjun Kumar, Yoonchang Sung, Peter Stone

Published 2026-07-28

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Faraz Heravi, James Ouyang, Zifan Xu, Arjun Kumar, Yoonchang Sung, Peter Stone

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to make a complex meal, like a gourmet sandwich. You can't just tell the robot, "Make a sandwich," and expect it to figure out how to slice the tomato, butter the bread, and stack the layers perfectly. If you only give the robot a "good job" or "bad job" signal at the very end, it will likely wander around aimlessly for years, never learning the specific steps. This is the world of Reinforcement Learning (RL), where software agents learn by trial and error. The tricky part is the "sparse reward" problem: if the robot only gets a reward when the task is 100% done, it has no idea if it's getting closer or further away. To fix this, scientists often use Curriculum Learning, a method where the robot practices on easy versions of a task first (like just picking up the bread) before moving to harder ones (slicing the tomato). But here's the catch: designing these easy-to-hard steps usually requires a human expert to manually write out every single rule and difficulty setting, which is slow, boring, and hard to do for every new task.

Enter LEACL (LLM-Enhanced Automatic Curriculum Learning), a new approach that asks a very smart question: "Can we get a super-intelligent AI language model to do the boring planning work for us?" The researchers behind this paper wanted to see if they could use a Large Language Model (LLM)—the same kind of tech that powers chatbots—to act as a master teacher. Instead of humans manually writing the lesson plans, the LLM would read a natural language goal (like "open the drawer") and automatically break it down into a sequence of smaller, manageable steps. Even better, the LLM would then design the specific "training wheels" (the difficulty settings and parameters) for each step, allowing the robot to learn using only the simple "success/fail" signal at the end, without needing complex, hand-written reward instructions for every tiny movement.

The paper tests this idea on five tricky robot tasks, such as opening a cabinet drawer, putting a bowl on a plate, or turning on a stove. The results are quite promising. The researchers found that when they let the LLM design the curriculum, the robot learned these long, complex tasks much faster and more successfully than when it tried to learn them all at once with no help. In fact, the LLM-designed system performed just as well as, and sometimes even better than, systems where human experts had spent hours manually crafting the training steps and reward rules. The study suggests that by letting AI handle the "lesson planning," we can teach robots to do complex, multi-step jobs without needing a human to micromanage every single detail of the training process.

The Robot's School Day: How LEACL Works

Think of a robot trying to learn a long-horizon task (a "long-horizon" task is just a fancy way of saying a job that takes many steps to finish) like a student trying to pass a final exam. If the teacher only gives a grade at the very end, the student might study the wrong things or give up entirely. Automatic Curriculum Learning (ACL) is like a tutor who creates a syllabus, starting with easy questions and slowly making them harder. But usually, a human has to write that syllabus.

LEACL changes the game by bringing in an LLM as the head teacher. The process happens in three fun stages:

The Breakdown (Task Decomposition):
Imagine you tell the robot, "Open the top drawer of the cabinet." A human might think, "Okay, that's just one job." But the LLM looks at it and says, "No way! That's actually three jobs: First, reach for the drawer. Second, grab the handle. Third, pull it open." The LLM uses its massive knowledge of how the world works (like knowing you can't pull a drawer if you aren't holding the handle) to break the big goal into a logical chain of smaller subtasks. It writes these down in a special "recipe language" called PDDL, which the robot can understand.
The Lesson Plan (Meta-Task Generation):
Now that the robot has the steps, it needs to know how to practice them. Should it start with the drawer slightly open? Or completely shut? Should the handle be close or far away? This is where the LLM shines again. It acts as a creative curriculum designer, generating a "task space" for each step. It creates a Python script that can generate thousands of slightly different versions of the "grab the handle" task. Some are super easy (the handle is right in front of the robot), and some are harder (the handle is slightly further away). The LLM also figures out which versions are "harder" than others, creating a perfect ladder of difficulty for the robot to climb.
The Practice (Automatic Curriculum Learning):
Finally, the robot starts training. It uses an algorithm that watches how well the robot is doing. If the robot is crushing the "easy" versions of the task, the system automatically switches to the "medium" difficulty versions. If the robot is struggling, it goes back to the easy ones. The robot learns using only a simple "1" for success and "0" for failure at the very end of each subtask. It doesn't need a human to say, "Good job, you moved your arm 2 inches closer." The LLM's pre-planned curriculum does all the heavy lifting of guiding the robot.

The Results: Did the Robot Pass the Test?

The researchers tested this system on five distinct challenges using a simulation called LIBERO (which they upgraded to LIBERO+ to handle these new types of tasks). The tasks included:

Opening a bottom drawer.
Putting a white bowl on a plate.
Picking up ketchup and putting it in a basket.
Turning on a stove and placing a pot on it.
Putting a mug in a microwave and closing the door.

They compared their LLM-powered system (LEACL) against several other methods:

The "Do Nothing" approach: Just letting the robot try to learn the whole thing with a sparse reward. (Spoiler: It failed completely, getting 0% success on most tasks).
The "No Curriculum" approach: Breaking the task down but letting the robot learn each step without a smart difficulty ladder. (This also failed miserably, with success rates near 0% or very low).
The "Human Expert" approach: Where humans manually designed the steps and the reward functions (giving the robot extra points for getting close to the goal). This is usually considered the gold standard.
The "LEAGUE" approach: A previous method that uses LLMs to write dense reward functions (complex scoring rules) for each step.

Here is what happened:
The LEACL system, which used the LLM to plan the curriculum but relied only on the simple "success/fail" signal, outperformed the systems that tried to learn without a curriculum. More impressively, it performed better than the LEAGUE system (which used complex, hand-tuned reward rules) on four out of the five tasks.

In terms of raw numbers, the LEACL system achieved success rates like 99.8% for opening the drawer and 90.7% for putting the bowl on the plate. These numbers were very close to, and in some cases matched, the performance of the "Human Curriculum" where experts manually designed everything. For example, on the "open the bottom drawer" task, the human-designed system got 99.8 ± 0.2%, and LEACL got 99.8 ± 0.1%. On the "put the mug in the microwave" task, the human system got 89.0 ± 7.5%, while LEACL got 75.9 ± 3.4%.

Why This Matters (and What It Doesn't Do)

The big takeaway is that designing complex reward functions is hard and often unnecessary. The paper argues that trying to give the robot a "dense" reward (like "you get 0.5 points for moving the arm closer") is actually a trap. It can confuse the robot, making it prioritize the wrong things or develop "sloppy" habits where it gets close to the goal but never quite finishes the job. By letting the LLM handle the structure of the learning (the curriculum) and letting the robot learn from the simple truth of success or failure, the robot learns more precise and reliable skills.

However, the paper is careful to note a few limits. The LLM still needs a "dictionary" of what is possible (a predefined set of rules or predicates) to work. It can't invent new physics or objects out of thin air; it has to work within the rules of the simulation (like the LIBERO+ environment). Also, while the LLM did a great job, the human-designed curriculum still edged it out slightly on three of the five tasks, suggesting that human intuition still has a role to play, even if the LLM is getting very good at it.

In short, LEACL suggests that we don't need to be the ones writing the detailed instruction manuals for robots anymore. We can just give the robot a goal, let an AI language model figure out the best way to teach it, and watch the robot learn to do complex, multi-step jobs all by itself.

Technical Summary: LEACL - LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks

Problem Statement

Long-horizon manipulation tasks present significant challenges for Reinforcement Learning (RL) due to sparse reward signals and the complexity of exploring high-dimensional, continuous state spaces over extended time horizons. While Automatic Curriculum Learning (ACL) has emerged as a solution to improve sample efficiency by training agents on a sequence of tasks from easy to hard, its success heavily relies on task-dependent specifications. These specifications include well-defined task parameter spaces and difficulty measures, which are traditionally manually crafted. This manual effort limits generalization across diverse tasks.

Furthermore, existing methods that attempt to automate curriculum design often rely on Large Language Models (LLMs) to generate dense reward functions for subtasks. However, designing dense rewards is prone to bias, requires significant human supervision, and may misalign with the true sparse objectives, potentially leading to task failure or "sloppy" behaviors that fail to satisfy precise task predicates.

Methodology: LEACL Framework

The authors propose LEACL (LLM-Enhanced Automatic Curriculum Learning), a framework that integrates LLMs with ACL to eliminate the need for manually designed dense reward functions. LEACL operates in three sequential stages:

1. LLM-Based Task Decomposition

Instead of training on the target task directly, the framework first decomposes a complex, long-horizon task (described in natural language) into a sequence of semantically meaningful subtasks.

Process: An LLM generates a sequence of subtasks in natural language. These are then converted into Planning Domain Definition Language (PDDL) specifications.
Mechanism: The system employs a bi-level prompting strategy with reflection mechanisms (syntax validation and iterative refinement) to ensure the generated PDDL specifications are syntactically valid and executable.
Output: A sequence of subtasks, each defined by a set of initial literals ( $I_i$ ) and goal literals ( $G_i$ ), forming a sequence of MDPs.

2. LLM-Based Meta-Task Generation

To facilitate ACL without dense rewards, the LLM generates task-dependent specifications for each subtask.

Task Parameter Space: The LLM identifies a parameter vector (e.g., initial object positions, target opening distances) that defines a control space. It generates a Python function to map these parameters to valid PDDL specifications, effectively creating a parametric task space.
Difficulty Measure: The LLM generates a sequence of parameter vectors explicitly ordered by difficulty. This allows the curriculum to progress from easier instances (e.g., a drawer opened 25%) to harder ones (e.g., 100%).
Key Distinction: Unlike prior work, the LLM does not generate reward functions; it generates the structural specifications required for the ACL algorithm to function.

3. Automatic Curriculum Learning (ACL)

The framework utilizes established ACL algorithms (specifically Active Domain Randomization in the experiments) to train the agent.

Process: The ACL algorithm adaptively selects training tasks from the LLM-generated parameter space based on the agent's current capabilities (e.g., past episodic rewards or temporal difference errors).
Reward Signal: The agent learns using only sparse task completion rewards (reward = 1 upon goal satisfaction, 0 otherwise). No dense shaping rewards are used.
Aggregation: Once the agent masters the sequence of subtasks, the policies are aggregated to solve the original target task.

Key Contributions

Novel Framework: The introduction of LEACL, which integrates LLMs for curriculum generation with ACL methods, enabling RL agents to solve long-horizon manipulation tasks using only sparse rewards.
Elimination of Dense Rewards: The framework removes the dependency on hand-crafted or LLM-generated dense reward functions, which are often difficult to generalize and can introduce bias.
LLM as Meta-Task Generator: The paper demonstrates using LLMs to generate task-dependent specifications (parameter spaces and difficulty measures) rather than direct curricula or rewards, simplifying prompt design and reducing LLM query overhead during training.
Empirical Validation: Extensive evaluation on the LIBERO+ benchmark (an extension of the LIBERO benchmark with parameterized predicates) across five diverse manipulation tasks.

Experimental Results

The authors evaluated LEACL against several baselines on five long-horizon tasks (e.g., "open the bottom drawer," "put the mug in the microwave"):

vs. Sparse Reward RL: LEACL significantly outperformed standard RL trained with only sparse rewards (which achieved 0% success on most tasks).
vs. LEACL w/o ACL: LEACL outperformed a version using LLM decomposition but without curriculum learning, demonstrating that decomposition alone is insufficient for sparse-reward learning.
vs. LEAGUE (Dense Rewards): LEACL achieved superior or competitive performance compared to LEAGUE, a baseline using expert-designed dense reward functions. Notably, LEACL outperformed LEAGUE on 4 out of 5 tasks, suggesting that dense rewards can sometimes hinder the learning of precise behaviors required for complex conjunctions of predicates.
vs. Human Curriculum: LEACL achieved competitive performance against a "Human Curriculum" baseline (where experts designed both decomposition and task spaces). While the human baseline performed slightly better on 3 tasks, the authors note that the human baseline required significant iterative effort, whereas LEACL operates in a zero-shot manner.

Significance and Claims

The paper claims that LEACL offers a scalable and adaptable approach to robotic skill acquisition by leveraging the common-sense knowledge of LLMs to structure the learning process without human intervention in reward design.

Efficiency: The method achieves superior asymptotic performance (success rates) compared to baselines without ACL and competitive performance against human-engineered solutions.
Generalization: By avoiding task-specific dense reward engineering, the framework aims to be more generalizable across diverse manipulation tasks.
Limitations: The authors acknowledge that LEACL currently relies on a pre-defined grammar (a set of available predicates provided a priori, such as in LIBERO+). Future work is suggested to explore methods where LLMs can automatically propose task-specific grammars to further reduce human dependency.

In summary, LEACL posits that the combination of LLM-driven structural decomposition and ACL, operating on sparse rewards, is a more robust and less labor-intensive path to solving long-horizon manipulation tasks than traditional dense-reward approaches.