Goal Sets, Not Goal States: Queryable Robot Goals through Goal-Set Hindsight Relabeling

Technical Summary: Goal Sets, Not Goal States: Queryable Robot Goals through Goal-Set Hindsight Relabeling

Problem Statement
Offline Goal-Conditioned Reinforcement Learning (GCRL) aims to derive control policies from pre-collected, reward-free datasets (e.g., demonstrations or teleoperation logs) without costly online trial-and-error. A central mechanism in this domain is Hindsight Experience Replay (HER), which relabels achieved future states as goals for earlier transitions in a trajectory.

Standard offline GCRL benchmarks, such as OGBench, typically instantiate HER using the full state space as the goal space. This approach assumes that success requires reproducing the entire future state vector. However, in many robotic manipulation tasks, success depends only on a specific subset of state variables (e.g., object position), while other variables (e.g., robot joint configurations, gripper state, velocities) are nuisance dimensions. By enforcing exact full-state matching, standard HER introduces artificial constraints that overconstrain the learning objective, leading to suboptimal performance when the task success criteria are a projection of the full state.

Methodology: Goal-Set Hindsight Relabeling (GS-HER)
The authors propose Goal-Set Hindsight Relabeling (GS-HER), a predicate-level generalization of HER. Instead of treating an achieved state $s_{t+k}$ as a singleton goal $g = s_{t+k}$, GS-HER treats it as a certificate for a goal set defined by a query.

1. Query-Defined Equivalence Classes:
   A binary query vector $q \in \{0, 1\}^D$ specifies which state variables define success for a specific task.

   • $q_i = 1$: The $i$-th coordinate is active and defines the goal.
   • $q_i = 0$: The $i$-th coordinate is a nuisance variable for this query.

   Given a reference state $g$ and a query $q$, the goal set is defined as the equivalence class of states $G(g, q) = \{s' \in \mathcal{S} : q \odot s' = q \odot g\}$. Success is determined by matching only the active coordinates, typically measured via a normalized squared distance with a tolerance $\epsilon$.

2. Query-Conditioned Interface:
   GS-HER modifies the conditioning input of the underlying offline GCRL algorithm without altering its core learning objective (e.g., Bellman updates, contrastive loss, or behavioral cloning). The conditioning input $\psi_{GS}$ is constructed as:
   $$\psi_{GS}(s, g, q) = [\bar{s}_q, \bar{g}_q, q]$$
   where $\bar{s}_q = q \odot s + (1-q) \odot e_s$ and $\bar{g}_q = q \odot g + (1-q) \odot e_g$. Here, $e_s$ and $e_g$ are nuisance-coordinate embeddings (learned or fixed) that mask out irrelevant variables. The query $q$ is explicitly provided to the network, allowing the model to learn a unified policy that adapts its notion of "success" based on the query.

3. Relabeling Wrapper:
   GS-HER acts as a wrapper around existing offline GCRL algorithms. During training, for each transition, a reference state $g$ is sampled (as per the base algorithm), and a query $q$ is sampled from a distribution $p_Q$. The transition is then relabeled with the query-conditioned goal set and the corresponding success predicate. This allows a single model to learn from a mixture of goal semantics (e.g., full-state, object-position, end-effector pose) simultaneously.

Key Contributions

1. Objective Mismatch Identification: The paper identifies that exact-state hindsight relabeling creates an objective mismatch in robotic tasks where success relies on task-relevant projections rather than the full state.
2. Formalization of GS-HER: The authors formalize GS-HER as a predicate-level generalization where achieved states certify reachability of goal sets (equivalence classes) rather than singleton states.
3. Implementation-Agnostic Wrapper: GS-HER is instantiated as a query-conditioned relabeling wrapper for vector-state environments. It enables existing offline GCRL algorithms (including behavioral cloning, contrastive RL, and implicit Q-learning) to learn queryable goal semantics without modifying their underlying optimization objectives.
4. Reusable Goal Interface: The method transforms hindsight relabeling into a reusable interface, allowing a single trained checkpoint to answer multiple goal predicates at inference time without retraining.

Experimental Results
The authors evaluated GS-HER on the OGBench suite, covering manipulation tasks (e.g., cube placement) and navigation tasks (e.g., PointMaze, AntMaze) across five different offline GCRL backbones (GCBC, CRL, GCIVL, GCIQL, GCDL).

• Performance Improvement: GS-HER (specifically the "Blockwise" variant using random coordinate blocks) consistently outperformed standard full-state HER (HER-Full) on manipulation tasks. For example, averaged over manipulation tasks, GS-HER improved the official success rate from 24.5% (HER-Full) to 55.0%, approaching the performance of an oracle "Task-Focused" baseline (HER-Task) which uses a fixed, pre-defined projection.
• Nuisance Dimension Bottleneck: The performance gains were strongly correlated with the degree to which full-state HER was bottlenecked by nuisance dimensions. In low-nuisance environments (like PointMaze), GS-HER performed comparably to HER-Full, confirming that the method specifically addresses the overconstraint problem.
• One Model, Many Predicates: In a "one-model-many-goals" experiment, a single GS-HER checkpoint was queried at inference time with various predicates (e.g., cube position, cube yaw, end-effector pose, airborne object position). The model successfully adapted to these different success criteria without retraining. In contrast, HER-Task required retraining for each new predicate, and HER-Full failed to generalize to predicates that ignored specific state variables.

Significance and Claims
The paper claims that GS-HER isolates the relabeling semantics of offline GCRL. By decoupling the definition of success (the query) from the learning algorithm, GS-HER provides a primitive that allows offline datasets to support flexible, queryable goal specifications.

The authors emphasize that this work does not solve open-ended goal specification (e.g., via language or vision) but rather demonstrates that making the goal predicate an inference-time input is a useful and necessary step for offline robot learning. It allows a single model to serve multiple task definitions, mitigating the need for separate training runs for different task projections and reducing the reliance on exact-state matching which is often semantically incorrect for robotic manipulation. The method is presented as a complementary layer to existing algorithms, enhancing their flexibility without altering their core mathematical objectives.