💻 computer science

Abstraction for Offline Goal-Conditioned Reinforcement Learning

This paper proposes a hierarchical framework for offline Goal-Conditioned Reinforcement Learning that leverages relativised options and distinct state representations to exploit symmetries and shared structures, thereby enabling experience reuse across similar contexts and significantly improving performance.

Original authors: Clarisse Wibault, Alexander Goldie, Antonio Villares, Maike Osborne, Jakob Foerster

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Clarisse Wibault, Alexander Goldie, Antonio Villares, Maike Osborne, Jakob Foerster

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a giant, complex maze. You have a video recording of someone else trying to solve it, but the recording is messy. The person in the video often gets stuck, takes wrong turns, or only tries a few specific paths. This is what researchers call Offline Reinforcement Learning: teaching an AI using only a static dataset of past attempts, without letting it explore the real world to learn by trial and error.

The problem is that if the robot tries to learn the entire maze as one giant, complicated task, it gets confused by the bad parts of the video. It doesn't know how to handle the corners where the previous person failed.

This paper introduces a new way to teach the robot, called Abstractive Reinforcement Learning (ARL). Here is the core idea, broken down with simple analogies:

1. The Problem: The "Absolute" Trap

Most robots try to learn "Absolute" instructions. They memorize: "To get to the red door, I must turn left at the blue wall, then walk 50 steps, then turn right."

If the robot sees a red door in a different part of the maze (maybe the wall is green, or the distance is different), it panics. It thinks, "I've never seen this exact combination before!" It fails to realize that the action of turning left is the same, even if the location is different. This is like a student who memorizes the answer to a math problem but fails when the numbers change slightly because they didn't understand the concept.

2. The Solution: "Relativised" Options

The authors suggest teaching the robot Relativised Options. Instead of memorizing absolute coordinates, the robot learns relative instructions.

Absolute Instruction: "Go to the red door at coordinates (10, 20)."
Relativised Instruction: "Go to the nearest corner in front of you."

Think of it like giving directions to a friend.

Bad (Absolute): "Walk 47 steps, turn 90 degrees left, walk 12 steps." (This only works if you start at the exact same spot).
Good (Relativised): "Walk until you hit a wall, then turn left." (This works no matter where you start).

By learning these "relative" sub-tasks, the robot can reuse the good parts of the messy video. Even if the robot only saw one person successfully navigate a corner in the video, it can apply that "turn-left-at-the-corner" skill to every corner in the maze, not just that one specific spot.

3. The Two-Step Strategy (The Hierarchy)

The paper proposes a two-layer brain for the robot:

The Boss (High-Level): This layer looks at the big picture. It doesn't care about the specific details of the room. It just says, "Okay, the goal is over there. Let's pick a sub-task, like 'Go to the next intersection'."
The Worker (Low-Level): This layer handles the actual movement. It hears "Go to the next intersection" and figures out the specific steps to get there, regardless of whether the floor is carpet or tile.

The Innovation: The paper introduces two specific ways to train this system:

ARLi (Implicit): The robot learns to group similar actions together naturally. If two different situations require the same "turn left" move, the robot figures out they are similar without being explicitly told.
ARLe (Explicit): The robot is forced to ignore absolute positions entirely. It is taught to only look at the difference between where it is and where it needs to go (e.g., "The goal is 5 meters to my right"). This is like training a robot to only see vectors (arrows) rather than maps.

4. Why This Matters

The authors tested this on complex tasks like moving a virtual humanoid robot through mazes and manipulating objects in 3D space (like solving a puzzle with blocks).

The Result: In situations where the training data was sparse (the robot hadn't seen many examples of how to solve a specific part of the puzzle), the new method (ARL) performed significantly better than older methods.
The Analogy: Imagine trying to learn to play chess by watching a few games where the players made mistakes. A standard AI would get stuck on the specific board positions where the players messed up. The new AI (ARL) learns the principles of the moves (e.g., "control the center," "protect the king"). Even if it hasn't seen that exact board setup, it knows how to make a good move because it understands the relative relationships between the pieces.

Summary

The paper argues that to teach robots effectively from imperfect, limited data, we shouldn't force them to memorize exact locations. Instead, we should teach them to understand relative relationships (e.g., "move toward the goal," "avoid the wall"). By breaking big problems into smaller, reusable "relative" skills, the robot can generalize what it learned from a messy dataset to solve new, unseen challenges.

The authors call this Abstractive Reinforcement Learning, and their experiments show it makes robots much better at solving complex, long-distance tasks when they only have a limited amount of practice data to learn from.

Technical Summary: Abstraction for Offline Goal-Conditioned Reinforcement Learning

Problem Statement

Offline Goal-Conditioned Reinforcement Learning (GCRL) aims to train general-purpose agents to solve complex, long-horizon tasks using static datasets. However, existing methods struggle in regions where the dataset suffers from low-quality transitions or imperfect coverage of the state-action space. Standard flat offline RL algorithms, which regularize actions toward the dataset distribution to mitigate distribution shift, often fail to recover optimal policies when the dataset contains suboptimal data in critical regions. While hierarchical policies (options) have been proposed to reduce the effective horizon (temporal abstraction), they have traditionally been anchored to the absolute frame of reference, limiting their ability to reuse experience across similar but distinct contexts.

Methodology: Abstractive Reinforcement Learning (ARL)

The authors propose Abstractive Reinforcement Learning (ARL), a framework that extends hierarchy to enable absolute abstraction. This involves learning relativised options and utilizing distinct representations at different levels of the hierarchy. The core premise is that by decoupling low-level actions from redundant high-level information and defining options relative to the agent's local context (e.g., "move forward" rather than "move to coordinate X"), an agent can generalize behaviors across similar state-space contexts.

The framework introduces two specific notions of similarity:

High-level similarity: State-goal pairs that induce similar options.
Low-level similarity: State-option pairs that induce similar immediate actions.

To operationalize this, the authors propose two algorithms:

1. ARLi (Implicitly Learning Relativised Options)

This algorithm learns relativised options via action similarity rather than value similarity.

Architecture: It employs two distinct value functions: a high-level value function ( $V_h$ ) and a low-level value function ( $V_l$ ).
Option Learning: Unlike previous hierarchical methods (e.g., HIQL) that learn option embeddings via the value function, ARLi learns option embeddings ( $\phi_\omega$ ) jointly with the low-level policy ( $\pi_l$ ). This encourages the grouping of state-waypoint pairs that induce similar low-level actions.
Training: It uses Implicit Q-Learning (IQL) for value learning and Advantage-Weighted Regression (AWR) for policy extraction. The high-level policy samples options based on the high-level value, while the low-level policy executes actions conditioned on the state and the sampled option.

2. ARLe (Explicitly Enforcing Translation Invariance)

This algorithm builds on ARLi by explicitly imposing translational invariance on the low-level decision process, specifically for high-dimensional manipulation tasks.

Relativised States: It defines the low-level value function in terms of unnormalized displacement vectors ( $v = g_s - s$ ), effectively collapsing the state-waypoint space into a manifold of relative displacements.
Option Embeddings: Options are defined in terms of these relativised states. The authors use a "soft-normalisation" technique to map these embeddings onto a hypersphere, allowing for geometric regularisation while maintaining implicit temporal awareness (magnitude scales with displacement).
Goal: This inductive bias allows the agent to reuse experience across the state-space by treating geometrically similar relative configurations as equivalent, addressing issues of sparse state coverage in high-dimensional spaces.

Key Contributions

Motivation for Absolute Abstraction: The paper argues that hierarchy in offline RL should not only reduce the temporal horizon but also enable absolute abstraction. By using relativised options and distinct representations, agents can bound the maximum error in expected return compared to flat policies, particularly in regions with low-quality data.
Theoretical Error Bounds: The authors provide a derivation showing that hierarchical policies with absolute abstraction reduce the error bound through four mechanisms:
- Horizon Reduction: Reducing the effective discount factor for the high-level policy.
- Cardinality Reduction: Mapping state-goal pairs into equivalence classes.
- Option Efficiency: Reducing the option space size via relativisation.
- Concentrability Improvement: Aggregating probability mass across similar contexts, thereby reducing the concentrability coefficient ( $\kappa$ ) and mitigating support mismatch.
Two Novel Algorithms: The introduction of ARLi and ARLe, which demonstrate that learning options via action similarity and enforcing translational invariance significantly improves performance without introducing additional hyperparameters.
Practical Guidelines: The work offers specific recommendations for practitioners, such as learning options via action similarity (jointly with the low-level policy), using two value functions to decouple decision processes, and imposing translation invariance for high-dimensional manipulation.

Experimental Results

The authors evaluated ARLi and ARLe on the OGBench datasets, focusing on challenging locomotion and manipulation environments (e.g., antmaze-giant, humanoidmaze-giant, puzzle-4x4-play).

Performance: Both ARL variants outperformed flat policies (IQL) and absolute hierarchical baselines (HIQL variants). Across all tasks, ARL achieved a mean success rate at least 10 percentage points higher than the benchmarks.
Specific Gains:
- ARLi: Consistently improved over hierarchical benchmarks, more than doubling success rates in humanoidmaze environments and nearly doubling them for scene-play-v0.
- ARLe: Demonstrated superior performance in high-dimensional manipulation tasks. In puzzle-4x4-play-v0 (83 dimensions), ARLe achieved an 88% success rate, outperforming the next-best benchmark by 39 percentage points.
Generalisation: Analysis showed that performance improvements correlated with increasing state dimensionality and data sparsity. ARLe, in particular, learned substantially smoother low-level value functions by collapsing equivalent relative states.
Robustness: The algorithms achieved these results using hyperparameters tuned for the baseline HIQL, demonstrating that the performance gains stem from the inductive biases rather than extensive tuning.

Significance and Claims

The paper claims that hierarchy enables more robust policy extraction in regions where datasets suffer from low-quality transitions. By abstracting away from the absolute frame of reference, agents can reuse optimal experience across similar contexts, effectively mitigating the limitations of offline data coverage.

The authors position this work as a proof of concept that distinct representations at different hierarchy levels can improve generalisation in offline GCRL. They explicitly state that while the current results are promising, the approach opens avenues for future research, such as:

Learning relativised options via action chunking.
Incorporating more flexible inductive biases using Geometric Deep Learning (GDL).
Addressing limitations where the inductive bias (e.g., translational invariance) may not align with the environment structure (e.g., dense, low-dimensional mazes).

The work does not claim to solve all offline RL challenges but demonstrates that specific structural biases (relativisation and distinct representations) can significantly enhance policy generalisation in high-dimensional settings without the need for complex hyperparameter tuning.