🤖 machine learning

Curriculum reinforcement learning with measurable task representation learning

This paper proposes a novel automatic curriculum generation approach for reinforcement learning that utilizes a variational autoencoder to learn a latent task representation with measurable similarity, enabling effective curriculum construction in complex, non-Euclidean navigation tasks where traditional interpolation methods fail.

Original authors: Yongyan Wen, Siyuan Li, Mingjian Fu, Yiqin Yang, Xun Wang, Peng Liu

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongyan Wen, Siyuan Li, Mingjian Fu, Yiqin Yang, Xun Wang, Peng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a complex, confusing maze to find a specific treasure. If you just drop the robot into the hardest version of the maze immediately, it will likely get stuck, hit walls, and give up because it has no idea what to do. This is a common problem in Artificial Intelligence called the "sparse reward" issue: the robot only gets a "good job" signal when it finally wins, which might take thousands of tries.

Curriculum Reinforcement Learning (CRL) is the idea of teaching the robot like a human teacher: start with an easy level, then a slightly harder one, and gradually work your way up to the final boss. But here's the tricky part: How does the computer know which level is "next"?

The Problem with "Straight Lines"

Most previous methods tried to create these intermediate levels by drawing a straight line between the "easy start" and the "hard finish." Imagine drawing a straight line on a map from your house to a mountain peak. If there's a giant canyon or a wall in the middle of that straight line, the path is useless. You can't walk through the wall.

In complex mazes (like the ones in this paper), the "distance" between two tasks isn't a straight line. It's more like a winding path around obstacles. If you just mix the start and end points mathematically, you might accidentally create a "task" where the robot is trapped behind a wall with no way out. This is what the authors call the Euclidean limitation—it assumes the world is flat and open, but real mazes are full of twists, turns, and dead ends.

The Solution: ACRL (The "Translator" Approach)

The authors propose a new method called ACRL (Automatic Curriculum with Representation Learning). Instead of trying to mix the start and finish directly, they use a clever trick involving a "secret language" or a Latent Space.

Think of it like this:

The Translator (VAE): The robot tries out many different mazes. The system watches what the robot does (its moves and the rewards it gets) and translates those experiences into a "latent space." This isn't a physical map; it's a mental map where similar experiences are close together, regardless of how far apart they look on the actual map.
- Analogy: Imagine two mazes that look totally different on paper. In one, you go left then right; in the other, you go right then left. But if both mazes have a similar "feel" (e.g., both require dodging a specific type of trap), the system realizes they are "cousins" and places them next to each other in this secret mental map.
The Smooth Path: Once the system has this mental map, it can draw a smooth, safe line from the "easy" tasks to the "hard" target task within this mental space. Because this space understands the true difficulty and similarity of the tasks (not just their physical coordinates), the line it draws avoids the "walls" and dead ends.
The Translator Back: The system then translates these new, smooth steps back into real, playable mazes. The result is a perfect sequence of levels that gradually get harder, guiding the robot safely to the goal without ever hitting a dead end.

How It Works in Practice

The paper tested this on two types of challenges:

MiniGrid: A grid-based world with keys, doors, and lava. The robot has to pick up a key to open a door. If you just mix the start and end positions, you might create a door that is locked but the key is on the other side of a wall. ACRL avoids this by understanding the logic of the task, not just the numbers.
U-Maze: A continuous, smooth environment with a barrier in the middle. The robot has to learn to go around the barrier. Standard methods often tried to push the robot straight through the barrier (which is impossible). ACRL learned to guide the robot around the curve.

The Results

The paper claims that ACRL is much faster and more efficient than previous methods.

Faster Learning: The robot learns the final task much quicker because it isn't wasting time on impossible or confusing intermediate steps.
Better Stability: Once the robot learns the path, it stays good at it. Other methods sometimes get confused and forget what they learned.
No External Help: The system figures out the curriculum on its own without needing a human to say, "Okay, now try this specific level."

The Catch

The paper notes one limitation: This method works best when the environment can be described by numbers (like coordinates or parameters). It doesn't yet know how to handle tasks described in words or symbols (like "go to the red house" vs. "go to the blue house" if the houses aren't defined by coordinates).

In summary: The paper introduces a smart teacher for robots. Instead of guessing the next step by drawing a straight line, it builds a mental map of what tasks are actually similar. It then uses this map to create a perfect, step-by-step training course that guides the robot from "beginner" to "expert" without ever getting stuck in a dead end.

Technical Summary: Curriculum Reinforcement Learning with Measurable Task Representation Learning

Problem Statement
Curriculum Reinforcement Learning (CRL) aims to accelerate learning on challenging target tasks by training agents on a sequence of tasks with increasing difficulty. While early CRL methods focused on sequencing pre-defined tasks, recent research has shifted toward automatic curriculum generation. A dominant paradigm in this field involves task distribution interpolation, where new intermediate tasks are generated by interpolating between an initial task distribution and a target task distribution using distance metrics like KL divergence or Wasserstein distance.

However, this interpolation-based approach relies on the assumption that task similarity can be measured via Euclidean distance in the context (task) space. This assumption fails in complex environments, particularly those with non-Euclidean geometric structures (e.g., mazes with walls). In such contexts, a straight-line interpolation in the parameter space may generate tasks that are physically infeasible (e.g., placing a goal behind a wall) or do not represent a meaningful progression in difficulty, thereby hindering the agent's ability to transfer knowledge effectively. Furthermore, existing methods often struggle to define meaningful similarity metrics in complex, high-dimensional, or sparse-reward environments without external reward signals specifically designed for curriculum generation.

Methodology: ACRL
The authors propose ACRL (Automatic Curriculum with Representation Learning), a framework that generates curricula by learning a measurable task representation in a latent space. The core insight is that task similarity should be defined by the agent's behavior (state transitions and rewards) rather than raw environmental parameters.

The methodology consists of three main components:

Task Representation Learning via VAE:
The authors employ a Variational Autoencoder (VAE) to map task contexts to a latent space.
- Encoder: Takes trajectories (sequences of states, actions, rewards, and next states) collected from a specific task context $c$ and encodes them into a latent embedding $z$ . To ensure permutation invariance, the trajectory embedding is computed as the product of embeddings for individual transitions.
- Decoders: The VAE includes a transition decoder ( $p_\theta(s'|z)$ ) and a reward decoder ( $p_\theta(r|z)$ ) to reconstruct the next state and reward, ensuring the latent space captures the dynamics and reward structure of the task.
- Task Decoder: An auxiliary decoder ( $p_c^\theta(z)$ ) maps the latent embedding back to the context space, allowing the generation of new task parameters from latent points.
- Training Constraint: To ensure meaningful representations, the VAE is trained only on trajectories where the agent achieves a return above a specific threshold $\delta$ , filtering out random or failed attempts.
Automatic Curriculum Generation:
Once task representations are learned, the curriculum is updated using two strategies based on the agent's learning progress:
- Latent Space Prediction (LSP): Used when the agent has achieved a certain performance on the target task. The method interpolates between the latent embeddings of current successful tasks and the target task's embedding in the latent space. These interpolated points are decoded back to the context space to generate new, progressively harder tasks that respect the underlying geometric constraints of the environment.
- Exploration Bound Update (EBU): Used in early stages or when target task performance is low. The method identifies the "exploration boundary"—tasks closest to the target that the agent can currently solve—and samples new tasks near this boundary with added Gaussian noise to encourage exploration.
Sampling Strategy:
The framework maintains a task buffer and uses a sampling ratio $\lambda$ to balance between sampling from the learned curriculum distribution (to exploit progress) and a uniform distribution over the context space (to ensure exploration and prevent mode collapse).

Key Contributions

ACRL Framework: The proposal of an automated curriculum RL framework that generates intermediate task distributions between source and target tasks without requiring external rewards for curriculum generation. It utilizes task representation learning to map contexts to a latent space, ensuring a smoother and more meaningful metric for task similarity.
Measurable Task Similarity: A novel task representation learning method that maps the context space to a latent space using a VAE. This approach allows for similarity measurement in non-Euclidean metric spaces by encoding state transitions and rewards, overcoming the limitations of direct Euclidean interpolation.
Empirical Superiority: Demonstrations showing that ACRL achieves superior learning efficiency and asymptotic performance on challenging navigation tasks with complex geometric structures compared to state-of-the-art baselines, including interpolation-based methods (CURROT) and generative approaches (Goal GAN, ALP-GMM).

Experimental Results
The authors evaluated ACRL on two domains:

MiniGrid (Discrete Action Space): A set of maze navigation tasks with varying difficulty (Easy, Medium, Hard) involving keys, doors, and lava. In these environments, Euclidean distance fails because walls block direct paths.
- Performance: ACRL outperformed baselines (Default, Random, ALP-GMM, Goal GAN, VDS, PLR, CURROT) in terms of time-to-threshold and stability.
- Curriculum Visualization: The generated task distributions successfully navigated around obstacles, moving from the initial position toward the target via feasible paths. In contrast, CURROT generated intermediate tasks that often led agents into walls, failing to provide a valid learning path.
- Representation Analysis: Multidimensional Scaling (MDS) of the latent space revealed that the learned embeddings effectively captured the non-Euclidean structure of the tasks, clustering similar tasks (in terms of difficulty and geometry) together.
U-Maze (Continuous Action Space): A continuous control environment where the agent must navigate a U-shaped barrier.
- Performance: ACRL demonstrated better sample efficiency than baselines. While Goal GAN achieved good asymptotic performance, it suffered in time-to-threshold due to a lack of target guidance. Other baselines failed because they generated tasks in unreachable regions.
- Ablation Studies: The study confirmed that the combination of LSP and EBU is crucial. Removing either component or setting the sampling ratio $\lambda$ too high (reducing exploration) or too low (degrading to uniform sampling) significantly reduced performance. The inclusion of both reward and transition decoders was also shown to be essential for stable representation learning.

Significance and Claims
The paper claims that ACRL addresses a fundamental limitation in automatic curriculum generation: the inability of Euclidean distance metrics to capture task similarity in complex, non-Euclidean environments. By learning a latent representation based on the agent's actual interaction (trajectories), the method can generate curricula that respect the geometric and dynamic constraints of the environment.

The authors emphasize that their approach:

Does not require external rewards for curriculum generation.
Is compatible with any existing RL algorithm (e.g., PPO, SAC).
Handles both discrete and continuous context spaces.
Provides a more robust and efficient learning path for agents tackling long-horizon, sparse-reward tasks.

The paper concludes modestly, acknowledging a limitation: the method currently assumes a parametric context space (numerical parameters) and cannot directly handle non-parametric definitions like symbolic or language-based contexts, though this is noted as a potential avenue for future work involving semantic encoders.