🤖 AI

Goal-Conditioned Decision Transformer for Multi-Goal Offline Reinforcement Learning

This paper introduces a Goal-Conditioned Decision Transformer that leverages pre-collected offline data to efficiently solve complex, multi-goal robotics tasks with sparse rewards, demonstrating superior performance over state-of-the-art online baselines on the Franka Emika Panda platform.

Original authors: Paweł Gajewski, Dominik Żurek, Marcin Pietroń, Kamil Faber

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paweł Gajewski, Dominik Żurek, Marcin Pietroń, Kamil Faber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot arm to perform complex tasks, like picking up a block and placing it in a specific spot. Usually, you'd have to let the robot try, fail, crash into things, and learn from its mistakes in the real world. But that's dangerous, expensive, and wears out the robot.

This paper introduces a smarter, safer way to teach robots using offline learning. Think of it like this: instead of letting the robot practice in real-time, we give it a massive library of "home videos" of other robots (or experts) performing tasks. The robot then studies these videos to learn how to move, without ever touching the real world during training.

Here is the breakdown of their new method, using simple analogies:

1. The Problem: The "One-Task" Robot

Most robots are like students who only study for one specific exam. If you teach a robot to push a red block, it might not know how to push a blue block or reach for a cup. It lacks generalization.

2. The Solution: The "Goal-Conditioned Decision Transformer"

The authors created a new AI model called a Goal-Conditioned Decision Transformer. Here is how it works:

The "Transformer" Part (The Super-Reader): Imagine a student who doesn't just memorize steps but understands the story of a movement. This model reads a sequence of events (where the robot was, what it did, and what happened next) like a book. It uses a "Transformer" architecture (the same tech behind advanced chatbots) to understand the context of the whole story, not just the last sentence.
The "Goal-Conditioned" Part (The GPS): Standard robots might just be told, "Do this." This new model is told, "Do this to get to that specific place."
- The Analogy: Imagine you are giving directions to a driver.
  - Old way: "Turn left, then right." (The driver doesn't know the destination).
  - New way: "Turn left, then right to get to the Pizza Place."
- By explicitly feeding the robot the "desired goal" (the Pizza Place) into its memory along with the history of moves, the robot learns that different paths can lead to the same destination. It learns to adapt its moves based on where it wants to end up.

3. The "Hindsight" Trick

The paper mentions a technique called Hindsight Experience Replay.

The Analogy: Imagine a robot tries to push a block to the kitchen but accidentally pushes it to the living room. A normal robot thinks, "I failed."
The Hindsight Trick: This method says, "Well, you didn't get to the kitchen, but you did successfully push the block to the living room! Let's pretend that was the goal all along." This turns "failures" into "successful lessons," helping the robot learn much faster from sparse data.

4. The Experiment: The Franka Panda Robot

The team tested this on a real robotic arm (Franka Emika Panda) with three tasks:

Reach: Touching a specific spot.
Push: Sliding a cube to a spot.
PickAndPlace: Lifting an object and moving it.

They compared their new "Goal-Conditioned Transformer" against:

Behavioral Cloning: Just copying the videos without understanding the "why."
TQC+HER: A very strong, standard online learning method that learns by trial and error in real-time.

5. The Results: The Underdog Wins

The results were surprising and impressive:

Speed: Their method trained in 80 minutes, while the standard online method took 240 minutes.
Performance: In complex tasks (like PickAndPlace), their offline method actually performed better than the online method that had hours of real-world practice.
Robustness: Even when the "videos" they studied were mostly bad (random mistakes) with only a few good examples, the model still figured out how to succeed. It was like a student who could pass the test even if the textbook was 75% wrong, as long as the core logic was there.
Sparse Rewards: In situations where the robot gets no "good job!" signal until the very end (sparse rewards), this method stayed steady, while others got confused and failed.

Summary

The paper claims that by treating robot learning like reading a story with a clear destination in mind, they can teach robots complex tasks using only pre-recorded data. This is faster, safer, and often more effective than traditional methods that require the robot to physically practice in the real world. They also released the dataset they used so others can try it too.

Technical Summary: Goal-Conditioned Decision Transformer for Multi-Goal Offline Reinforcement Learning

Problem Statement

Reinforcement learning (RL) in robotics faces critical challenges regarding sample efficiency and the ability to generalize across varying goals. Standard online algorithms require continuous environment interaction, which is often impractical for real-world hardware due to safety risks, hardware wear, and time constraints. While simulations offer a partial solution, they frequently suffer from the "sim-to-real" gap.

Offline RL addresses these limitations by enabling agents to learn optimal policies entirely from pre-collected datasets, eliminating the need for risky online exploration. However, existing offline methods often focus on single fixed tasks, which is insufficient for general-purpose robotics. Furthermore, while the Decision Transformer (DT) has successfully reframed RL as a conditional sequence modeling problem, its application to multi-goal offline scenarios—specifically those involving sparse rewards and robotic manipulation—remains underexplored.

Methodology

The authors propose a Goal-Conditioned Decision Transformer (GC-DT) adapted for offline multi-goal robotics. The core methodology involves modifying the standard DT architecture to explicitly process goal information within the sequence modeling framework.

1. Architecture Adaptation

Unlike standard DTs that condition on scalar returns, the GC-DT incorporates structured goal data directly into the input sequence. In multi-goal environments, the observation space is a dictionary containing:

$o_t$ : Current observation.
$g_d$ : Desired goal.
$g_a$ : Achieved goal.

The state at timestep $t$ is defined as a tuple $s_t = (o_t, g_d, g_a)$ . To adapt the Transformer, these vectors are flattened and concatenated into a single input vector:
$s_t = [o_t^{(0)}, \dots, o_t^{(n)}, g_d^{(0)}, \dots, g_d^{(m)}, g_a^{(0)}, \dots, g_a^{(m)}]$

By including the achieved goal ( $g_a$ ) alongside the desired goal ( $g_d$ ), the model can implicitly infer progress toward the target without requiring explicit distance metrics or reward shaping. This allows the autoregressive model to perform temporally extended credit assignment regarding goal achievement across the entire trajectory.

2. Dataset and Training

The model is trained entirely offline using data generated from the Franka Emika Panda environment via the Panda-Gym suite. The dataset includes:

Expert Data: Trajectories from converged Truncated Quantile Critics (TQC) agents evaluated over 1 million timesteps.
Random Data: Trajectories generated by agents sampling actions uniformly.
Mixtures: Various combinations of expert and random data to test data efficiency and noise resilience.

The authors also release this specialized offline dataset to support reproducibility.

3. Tasks and Reward Structures

The approach is evaluated on three robotic manipulation tasks: Reach (positioning the end-effector), Push (moving a cube), and PickAndPlace (lifting and moving an object). Performance is tested under two reward structures:

Sparse Rewards: Binary feedback (0 for success, -1 otherwise).
Dense Rewards: Negative Euclidean distance between the achieved and desired goals.

Key Contributions

The paper outlines three primary contributions:

Architectural Extension: Extending the Decision Transformer to handle multi-goal settings by incorporating explicit goal-conditioning (desired and achieved goals) into the input sequence.
Dataset Release: Releasing a specialized offline RL dataset for multi-goal robotic manipulation on the Franka Emika Panda platform.
Empirical Validation: Demonstrating that the GC-DT approach outperforms state-of-the-art online baselines (TQC+HER) and Behavioral Cloning (BC) in offline settings, particularly under sparse reward signals.

Experimental Results

Experiments were conducted using three random seeds, comparing the GC-DT against Behavioral Cloning (BC) and TQC with Hindsight Experience Replay (TQC+HER).

Comparative Performance

Complex Tasks: In the more complex Push and PickAndPlace environments with dense rewards, GC-DT consistently outperformed both TQC+HER and BC in average return and success rate.
Efficiency: The GC-DT achieved these results with significantly lower training time (80 minutes) compared to the online TQC+HER baseline (240 minutes).
Sparse Rewards: The GC-DT demonstrated superior robustness in sparse reward settings. In the PickAndPlace task, GC-DT achieved a significantly higher return (-7.63 vs. -16.96 for TQC+HER) and maintained a 97.8% success rate, whereas TQC+HER's success rate dropped to 77.0%.

Data Efficiency and Robustness

Dataset Size: Performance remained invariant to dataset size for the Reach task. For complex tasks, performance declined only when data fell below 250k samples; even at 100k samples, the model maintained >96% success in dense settings.
Expert Data Ratio: The model showed surprising robustness to noise. Even when the dataset consisted of only 25% expert data (mixed with 75% random trajectories), the GC-DT maintained >80% success in complex tasks. Performance dropped sharply only below this threshold.

Significance and Claims

The authors claim that the proposed Goal-Conditioned Decision Transformer effectively bridges the gap between offline RL and multi-goal robotic control. They hypothesize that the model's robustness, particularly in sparse-reward settings, stems from the Transformer's ability to perform long-horizon credit assignment via self-attention.

The paper asserts that this approach can outperform the very expert policy used to generate the training data in complex tasks like PickAndPlace. Furthermore, it establishes that goal-conditioned sequence modeling is a viable and robust alternative to traditional online actor-critic methods for offline multi-goal robotics, capable of handling suboptimal demonstrations and sparse feedback without requiring online interaction.

Future work is identified as extending this framework to continual learning settings and training with heterogeneous expert trajectories.