🤖 AI

Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

This paper introduces CARL, a hierarchical reinforcement learning algorithm that exploits the regularity of local dynamics to align global contexts with required action sequences, thereby learning reusable skills that improve downstream performance on complex tasks.

Original authors: Sarthak Dayal, Abhinav Peri, Carl Qi, Claas Voelcker, Alexander Levine, Caleb Chuck, Amy Zhang

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sarthak Dayal, Abhinav Peri, Carl Qi, Claas Voelcker, Alexander Levine, Caleb Chuck, Amy Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Reinventing the Wheel" Issue

Imagine you are teaching a robot to navigate a giant, complex maze. In traditional Reinforcement Learning (RL), the robot often tries to learn every single move from scratch every time it enters a new room.

If the robot learns how to "walk forward" in the kitchen, it might forget how to do that exact same thing when it enters the living room, even though the physics of walking are identical. It's like a student who learns to solve a math problem in a textbook but fails to solve the exact same problem on a test just because the font is different.

This is the core challenge of Hierarchical Reinforcement Learning (HRL). HRL tries to teach robots "skills" (like walking, standing up, or grabbing) so they can reuse them. But current methods often fail to realize that a "walk forward" skill is the same whether you are in the kitchen or the living room. They treat them as totally different things, wasting time and data.

The Solution: CARL (The "Universal Translator" for Skills)

The authors introduce a new method called CARL (Contrastive Action-based Representations for Reusable Local Control).

Think of CARL as a universal translator for movement. Instead of asking, "Where am I in the world?" (which changes constantly), CARL asks, "What kind of movement sequence do I need to get from here to there?"

The Core Idea: "Local Dynamics"

The paper relies on a simple intuition: Local dynamics are regular.

Analogy: Imagine you are in a crowded city. If you want to walk three steps forward, you need to lift your left foot, then your right, then your left. It doesn't matter if you are in New York, Tokyo, or a small village; the sequence of steps required to move three feet is the same.
The Paper's Claim: Many different places in a robot's environment share these same "local rules." If a robot needs to go from Point A to Point B, and Point C to Point D, and the distance and obstacles are similar, the robot should use the exact same "skill" (action sequence) for both.

How CARL Works: The "Matchmaking" Algorithm

CARL looks at a massive database of past robot movements (offline data). It doesn't just memorize the map; it looks for patterns in how the robot moved.

The Input: It looks at a "State-Goal Pair." (e.g., "I am at the door, and I want to get to the table.")
The Secret Sauce: It looks at the Action Sequence the robot used to get there (e.g., "Step, Step, Turn, Step").
The Magic: CARL uses a technique called Contrastive Learning. It tries to group together any two situations that require the same action sequence, even if the situations look totally different on the surface.

The Metaphor:
Imagine a librarian organizing books.

Old Method: The librarian sorts books by the color of the cover or the city where they were printed. (This is like sorting by "where the robot is").
CARL Method: The librarian sorts books by the plot summary. If two books have the exact same plot (e.g., "The hero climbs a ladder to escape"), they are shelved together, even if one is a sci-fi novel and the other is a fantasy novel.

In CARL's world, "Standing up" in a maze corner and "Standing up" in the middle of a room get shelved together because the "plot" (the action sequence) is the same.

The Results: Why It Matters

The authors tested this on a benchmark called OGBench, which includes tasks like:

Locomotion: Getting a virtual ant, robot dog, or humanoid robot to walk through mazes.
Manipulation: Getting a robotic arm to stack cubes or solve puzzles.

What happened?

Better Transfer: When the robot learned a skill in one part of the maze, it could instantly use that skill in a completely different part of the maze. It didn't have to relearn how to walk.
Higher Scores: When they combined CARL with existing advanced AI methods (like HIQL), the robots solved tasks much more often. For example, in some difficult "giant maze" tests, the success rate jumped by over 30%.
Visual Proof: The authors created visualizations (like Figure 3 in the paper) showing that CARL successfully grouped "walking backward" behaviors together, even when the robots were in totally different parts of the maze.

What CARL Does NOT Do (Based strictly on the paper)

It does not create new skills out of thin air; it finds and reuses existing patterns found in the data.
It is not a magic fix for all problems. The paper notes that it struggles a bit with tasks that require very complex, long-term planning (like solving a 4x4 puzzle) or environments with very random, unpredictable physics (like teleporting portals).
It relies on having a good "library" of past data. If the data is sparse or bad, CARL can't find the patterns.

Summary

CARL is a method that teaches robots to recognize that "walking" is "walking," regardless of where they are. By mathematically grouping situations that require the same steps, it allows robots to reuse their learned skills across different parts of an environment, making them smarter, faster, and more efficient at solving long, complex tasks.

Technical Summary: Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL

Problem Statement

Hierarchical Reinforcement Learning (HRL) aims to solve long-horizon tasks by discovering and reusing temporally extended skills. However, a persistent challenge is obtaining skills that are genuinely reusable across different contexts. Existing approaches often suffer from training instability when co-learning skills and policies, or they fail to effectively leverage reusability. While goal-conditioned policies have decoupled training, they often lose the temporal abstraction and consistent reusability inherent to true skills. Specifically, reasoning directly in the global state-goal space makes it difficult for low-level policies to recognize when the same behavior can be reused in different global contexts.

Methodology

The authors propose CARL (Contrastive Action-based Representations for Reusable Local Control), a method designed to learn representations of local dynamics from offline datasets to facilitate skill reuse.

Theoretical Foundation: Local Dynamics and Bisimulation

The core intuition is local dynamics: many local regions of the state space share similar dynamics, meaning local transitions in different global contexts require similar action sequences.

k-ball MDP: The authors formalize local dynamics by defining a finite-horizon MDP (the $k$ -ball) rooted at a state $s$ , containing all states reachable within $k$ steps.
Dynamics Bisimilarity: Two states are considered to share local dynamics if their corresponding $k$ -ball MDPs are dynamics-bisimilar. This means there exists a relation where the transition probabilities under identical action sequences are equivalent, allowing the states to be treated as interchangeable for short-horizon goal-reaching behaviors.

Practical Algorithm: CARL

Directly estimating bisimulation in high-dimensional continuous domains is intractable. Instead, CARL leverages behavioral similarity as a heuristic. The hypothesis is that in near-expert offline datasets, state-goal pairs $(s_1, g_1)$ and $(s_2, g_2)$ that admit the same $k$ -step action sequence $a_k$ are behaviorally similar.

CARL learns embeddings $\phi(s, g)$ for state-goal pairs and $\psi(a_k)$ for action sequences using a contrastive learning objective (InfoNCE):
$\mathcal{L}_{InfoNCE} = -\frac{1}{B} \sum_{i=1}^B \log \frac{\exp(\langle \phi(s_i, g_i^k), \psi(a_i^k) \rangle / \tau)}{\sum_{j=1}^B \exp(\langle \phi(s_i, g_i^k), \psi(a_j^k) \rangle / \tau)}$
This objective aligns state-goal pairs with the action sequences that achieve them. Consequently, state-goal pairs requiring similar local skills are embedded nearby in the latent space, regardless of their global location.

Integration with HRL

CARL is integrated into existing hierarchical offline RL algorithms, specifically HIQL and HGCBC.

The learned representation $\phi(s, g)$ is used as the input for the low-level policy and value function.
The high-level policy outputs subgoals in this learned representation space.
This allows the low-level policy to reuse the same skill (action sequence) in different global contexts without relearning the policy from scratch for each region.

Key Contributions

Formalization of Local Dynamics: The paper defines state equivalence based on local dynamics (via $k$ -ball MDPs and bisimulation) rather than long-horizon value equivalence, providing a principled basis for skill reuse.
CARL Algorithm: A practical contrastive learning framework that approximates local dynamics structure from offline data, clustering state-goal pairs by their required action sequences.
Empirical Validation: Demonstrating that integrating CARL into HIQL and HGCBC significantly improves downstream performance on the OGBench benchmark across diverse tasks (locomotion, manipulation, and complex puzzles) and modalities (state-based and pixel-based).
Analysis of Representation Structure: Visualizations (UMAP) and nearest-neighbor analyses confirm that CARL successfully clusters distinct global states that require similar local behaviors (e.g., "walking backward" in different maze locations).

Results

The paper reports extensive experiments on the OGBench suite:

State-Based Tasks: HIQL+CARL achieved a win rate of 20/22 against standard HIQL, with significant improvements in complex environments like the "giant maze" (+30% on scene tasks) and "cube" manipulation tasks. HGCBC+CARL achieved 21/21 against HGCBC.
Pixel-Based Tasks: On visual environments, HIQL+CARL achieved 13/14 wins against HIQL, with notable gains in long-horizon antmaze tasks (e.g., +30% on large/giant variants).
Ablation Studies:
- Contrastive vs. Prediction: The contrastive objective (InfoNCE) outperformed direct regression (prediction) losses, suggesting it is more effective at organizing the latent geometry for skill reuse.
- Action Sequence vs. Single Action: Modeling full action sequences ( $a_k$ ) was superior to modeling single actions, confirming the importance of temporal structure.
- Sampling: "Interior sampling" (using intermediate states within the $k$ -step horizon) performed better than "surface sampling" (only using the endpoint).
Robustness: CARL+HIQL showed similar sensitivity to dataset coverage and imbalance as the baseline HIQL, indicating that CARL does not exacerbate data requirements.

Significance and Claims

The authors claim that local dynamics structure provides a principled basis for skill extraction and reuse, helping HRL reconnect with the temporal abstractions central to long-horizon decision-making. By learning embeddings that cluster state-goal pairs according to the short-horizon behaviors they admit, CARL provides a concrete mechanism for skill reuse that improves existing hierarchical RL frameworks.

The paper emphasizes that this approach allows high-level policies to reason about low-level skills more effectively, enabling the transfer of behaviors across globally different contexts (e.g., a humanoid standing up in different parts of a maze). While the method relies on offline datasets and fixed horizons, the results suggest that explicitly modeling local reusability unlocks performance benefits, particularly in complex, high-dimensional robotics domains where relearning low-level policies for every context is inefficient.

The work is modest about its limitations, noting that the choice of extraction horizon $k$ is environment-dependent and that performance on certain tasks (like those with stochastic dynamics or extreme combinatorial generalization) remains challenging. Future work is suggested to explore automatic skill discovery, action abstraction, and high-level policy representations.