⚡ electrical engineering

Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization

This paper establishes a theoretical foundation for unifying goal-conditioned reinforcement learning and unsupervised skill learning by framing both as instances of control maximization, proving that specific mutual information skill learning objectives are bounded by and thus optimally aligned with distinct goal-reaching formulations.

Original authors: Alireza Modirshanechi, Benjamin Eysenbach, Peter Dayan, Eric Schulz

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alireza Modirshanechi, Benjamin Eysenbach, Peter Dayan, Eric Schulz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a maze. You want it to learn two things:

How to get anywhere: If you point to a specific spot (a "goal"), the robot should know how to get there. This is called Goal-Conditioned Reinforcement Learning (GCRL).
How to be interesting: Before you tell it where to go, you want the robot to explore the maze on its own and learn a wide variety of different moves or "skills" (like running, jumping, or sliding) without any specific instructions. This is called Unsupervised Skill Learning (MISL).

For a long time, researchers have noticed that if you teach a robot these diverse skills first (MISL), it becomes much better at reaching specific goals later (GCRL). But nobody really understood why this worked. It was like knowing that eating vegetables makes you stronger, but not understanding the biology behind it.

This paper solves that mystery by showing that both of these tasks are actually the same thing in disguise: Maximizing Control.

Here is the breakdown using simple analogies:

1. The "One Size Does Not Fit All" Problem

The authors discovered that "reaching a goal" isn't just one single task. It depends entirely on the rules of the game. They identified three different ways to play:

The "Persistent Goal" (The Lighthouse): Imagine a lighthouse that stays on forever. The robot gets points every time it visits the lighthouse. The goal is to stay there as much as possible.
The "Exact Timing" (The Train Schedule): Imagine a train that leaves at exactly 5:00 PM. The robot gets points only if it arrives at the station at exactly 5:00 PM. Arriving at 4:59 or 5:01 gets zero points.
The "Opportunity Window" (The Deadline): Imagine a paper submission deadline. The robot gets points if it arrives at the office anytime before the deadline closes.

The Big Discovery: The authors proved that the best strategy for one of these games is often a terrible strategy for the others.

Analogy: If you train a runner to stay at the finish line as long as possible (Persistent), they will be terrible at sprinting to the line exactly when the gun goes off (Exact Timing). If you train them to sprint to the line at a specific second, they might be too slow to get there before a deadline (Opportunity Window).

This means you can't just use any "skill learning" method to help with any "goal reaching" task. You have to match them up.

2. The Unifying Theory: "Control"

So, how do we connect learning random skills to reaching specific goals? The authors say the link is Control.

Think of "Control" as the robot's ability to say, "I want to go here," and have the universe listen.

If the robot has high control, it can steer its future path exactly where it wants.
If the robot has low control, it's like a leaf blowing in the wind; it can't really choose where it goes.

The paper argues that:

Goal-Conditioned Learning (GCRL) is just trying to maximize how well the robot can steer itself to a specific destination.
Skill Learning (MISL) is trying to maximize how many different destinations the robot can reach using different "keys" (skills).

If a robot learns a diverse set of skills, it is essentially learning to be in control of many different parts of its environment. If it is in control, it can naturally reach any specific goal you ask it to.

3. The "Matching" Rule

The most important practical takeaway from the paper is that not all skill-learning methods are created equal.

Because there are three different types of "Goal Reaching" (Persistent, Exact Timing, Opportunity Window), there are three different types of "Skill Learning" that match them perfectly.

If your downstream task is like a Lighthouse (stay there forever), you should use a skill-learning method that focuses on where the robot ends up after a long time.
If your downstream task is like a Train Schedule (arrive at a specific time), you should use a skill-learning method that focuses on where the robot is at that exact moment.
If your downstream task is like a Deadline (arrive before time runs out), you need a specific skill-learning method that focuses on the first time the robot visits a spot.

The Metaphor:
Imagine you are packing a suitcase for a trip.

If you are going to a beach (Persistent), you pack swimwear.
If you are going to a ski resort (Exact Timing), you pack skis.
If you are going to a business conference (Deadline), you pack a suit.

The paper says: "Don't pack skis for the beach just because you learned how to ski." You must choose the pre-training (the packing) that matches the specific type of goal you will face later.

Summary

The Mystery: Why does learning random skills help robots reach goals?
The Answer: Because both are about Control. Learning diverse skills means learning to control your environment, which makes reaching specific goals easier.
The Catch: There isn't just one way to "reach a goal." There are different rules (stay forever, arrive on time, arrive before a deadline).
The Solution: You must pick the specific "skill learning" method that matches the specific "goal reaching" rules you will face later. If you match them correctly, the robot will be a master at the task. If you mismatch them, it will fail.

The paper provides the mathematical proof that links these specific types of skill learning to specific types of goal reaching, giving engineers a clear map on which tool to use for which job.

Technical Summary: Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization

Problem Statement

Unsupervised pretraining, particularly Mutual Information Skill Learning (MISL), has driven empirical success in Goal-Conditioned Reinforcement Learning (GCRL) by discovering diverse skills that facilitate downstream goal-reaching. However, the theoretical justification for why MISL objectives (which maximize behavioral diversity) should support specific downstream GCRL tasks remains poorly understood.

A fundamental obstacle to unifying these fields is that both "GCRL" and "MISL" are umbrella terms encompassing distinct, often incompatible formulations:

GCRL Diversity: Different GCRL tasks utilize distinct criteria for measuring goal-reaching performance (e.g., persistent goals, exact timing, or opportunity windows), leading to fundamentally different optimal policies.
MISL Diversity: Different MISL methods optimize distinct notions of behavioral diversity (e.g., diversity in terminal states vs. full trajectories) and rely on different variational bounds.

The central question is: Is there a theoretical account linking specific MISL objectives to specific downstream GCRL formulations, and under what conditions do they align?

Methodology and Framework

The authors propose a unifying perspective based on Control-Maximization. They argue that both GCRL and MISL fundamentally seek to maximize an agent's degree of control over its environment.

1. Analysis of GCRL Formulations

The paper identifies three canonical GCRL formulations and proves their general inequivalence:

Persistent Goal ( $Pe(\gamma)$ ): Infinite-horizon settings where the agent is rewarded repeatedly for revisiting a goal (e.g., target-occupancy optimization).
Exact Timing ($ET(K)$): Finite-horizon settings where reward is given only if the goal is reached at an exact step $K$ .
Opportunity Window ( $OW(K, \gamma)$ ): Finite-horizon settings where reward is given if the goal is reached within a window $K$ , often with discounting.

Key Finding: These formulations induce fundamentally different optimal policies. For instance, in a stochastic environment (e.g., crossing a river with a risky bridge), the optimal strategy for $Pe(\gamma)$ (prioritizing reliability/revisits) may differ from $ET(K)$ (prioritizing speed to meet a deadline) or $OW(K, \gamma)$ . The paper provides conditions (e.g., deterministic environments, one-step horizons, or infinite windows) under which these formulations converge to the same optimal policy.

2. Defining Goal Sensitivity

To unify these formulations, the authors introduce Goal Sensitivity ( $C(s, \pi)$ ), a novel measure of an agent's control. It quantifies how much the agent's future trajectory depends on the commanded goal compared to a random goal:
$C(s, \pi) := \frac{1}{N_s^2} \sum_{g, g' \in S} \left( J(s, g, \pi_{g}) - J(s, g, \pi_{g'}) \right)$
where $J$ is the return. This metric reflects both the environment's intrinsic controllability (objective control) and the agent's competence (subjective control).

3. Theoretical Bounds Linking MISL and GCRL

The core theoretical contribution is establishing precise information-theoretic bounds between MISL objectives and downstream GCRL performance via Goal Sensitivity:

Step 1: The authors show that for $Pe(\gamma)$ and $ET(K)$, maximizing Goal Sensitivity is equivalent to maximizing the GCRL return. For $OW(K, \gamma)$ , it provides a tight lower bound on the return.
Step 2: They derive that the Goal-Behavior Mutual Information (MI between the commanded goal and a behavioral variable $S'$ $S^{'}$ ) is tightly bounded by the Goal Sensitivity of the matching GCRL formulation.
- For $Pe(\gamma)$ , the matching behavioral variable is the discounted state occupancy ( $S_{\gamma,+}$ ).
- For $ET(K)$, the matching variable is the state at step $K$ ( $S_K$ ).
- For $OW(K, \gamma)$ , they identify a novel variable, the discounted first-visit vector ( $F_{K,\gamma}$ ), as the tightest match, noting that existing trajectory-based variables ( $\tau_K$ ) yield only loose bounds.
Step 3: They prove that the standard MISL objective (MI between skills and behavior) approximates the Goal-Behavior MI, with the gap controlled by the uniformity of the skill distribution induced by the goal-to-skill mapping.

Key Contributions

Inequivalence of GCRL Formulations: The paper proves that distinct GCRL formulations ($Pe$, $ET$, $OW$) generally induce incompatible optimal policies, explaining why algorithms designed for one formulation may fail in another.
Control-Maximization Lens: It provides a precise theoretical framework defining GCRL performance as maximizing "Goal Sensitivity," a measure of control that unifies disparate formulations.
Correspondence between MISL and GCRL: The authors establish systematic bounds showing that specific MISL objectives correspond to specific downstream GCRL formulations. They demonstrate that maximizing a specific MISL objective (e.g., diversity in terminal states) theoretically bounds the performance of a specific GCRL task (e.g., Exact Timing).
Identification of Novel Variables: The work identifies that for Opportunity Window formulations, standard trajectory-based MISL objectives are suboptimal, and a discounted first-visit vector is required for tight theoretical bounds.

Results

Theoretical Bounds: The paper derives two-sided bounds for $Pe$ and $ET$ formulations, showing that Goal Sensitivity tightly bounds the Mutual Information between goals and behavior. For $OW$, it provides a lower bound and discusses conditions for an upper bound.
Policy Equivalence Conditions: The paper delineates specific conditions (e.g., deterministic environments, infinite horizons, or the presence of "waiting" actions) under which policies learned for one formulation can be safely transferred to another.
Practical Implication: The results imply that there is no universally optimal MISL objective. The choice of pretraining objective must match the expected downstream goal-reaching criterion (e.g., using terminal-state diversity for exact timing tasks vs. trajectory diversity for opportunity windows).

Significance and Claims

The paper claims to lay a theoretical foundation for RL pretraining by resolving the "theoretical mystery" of why unsupervised skill learning aids downstream goal-reaching. By framing both fields as instances of control maximization, the authors provide a principled guide for practitioners:

Selection of Pretraining Objectives: Users should select MISL objectives that match the specific "flavor" of goal-reaching (persistent, exact, or windowed) expected at test time.
Algorithmic Design: The findings explain why certain GCRL algorithms (e.g., C-Learning, which relies on stationary structures) do not transfer to finite-horizon settings without modification.

The authors remain modest, acknowledging limitations such as the reliance on finite state/action spaces and the assumption of uniform goal distributions (though they show extensions to non-uniform distributions in the appendices). They also note that for the Opportunity Window formulation, the inverse bound (from MI to performance) requires additional assumptions, leaving the identification of the "perfect" MISL objective for this specific case as an open question.

Ultimately, the work bridges the gap between unsupervised skill discovery and goal-conditioned control, suggesting that the success of pretraining is not accidental but a result of aligning the pretraining objective with the specific controllability demands of the downstream task.