Technical Summary: Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

1. Problem Statement

Offline Goal-Conditioned Reinforcement Learning (GCRL) aims to learn policies that reach diverse goals from a fixed dataset of trajectories without further environment interaction. While effective for short-horizon tasks, this setting faces significant challenges in long-horizon tasks.

The primary difficulty arises from long-horizon value estimation. Under sparse goal-reaching rewards, distant state–goal pairs provide weak direct supervision. Consequently, the value function must propagate goal information through many bootstrapped updates. As the temporal distance increases, approximation errors accumulate along the backup chain, leading to inaccurate evaluations of faraway goals. Directly training a flat goal-conditioned policy on these noisy value estimates results in unreliable long-horizon decision-making.

Existing hierarchical methods attempt to mitigate this by introducing intermediate subgoals, reducing long-horizon reaching to a sequence of shorter-range decisions. However, these methods often rely on fixed temporal abstractions (e.g., fixed subgoal horizons) or fixed hierarchy depths. Such fixed scales are frequently mismatched to state–goal pairs with varying reachability horizons:

• Overly local subgoals may make only marginal progress, where small value differences are dominated by long-horizon estimation errors, leading to misleading choices.
• Overly distant subgoals may fall outside the reliable execution range of the low-level policy.

The core question addressed is whether an offline hierarchical agent can adaptively identify a suitable intermediate target for each specific state–goal pair, rather than relying on a prescribed decomposition scale.

2. Methodology: CFHRL

The authors propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. The framework operates on the principle that a subgoal need not be an exact midpoint or globally optimal; it only needs to provide reliable progress and reduce the remaining reaching difficulty.

2.1 Core Components

CFHRL utilizes three learned policies coordinated by a shared reachability cost $D_\phi(s, g)$, derived from a goal-conditioned value function:

1. Refinement Planner ($\pi_p$): Proposes intermediate targets $u$ conditioned on the current state $s$ and the current planning target $g$.
2. Abstraction Policy ($\pi_z$): Maps the refined target $\bar{g}$ to a local command $z$.
3. Low-Level Policy ($\pi_a$): Maps the command $z$ to primitive actions $a$.

2.2 Adaptive Coarse-to-Fine Refinement

Instead of a fixed hierarchy, CFHRL employs a recursive refinement process with an adaptive stopping rule:

1. Initialization: Start with the final goal $g^{(0)} = g$.
2. Executability Check: At refinement level $\ell$, check if the current target $g^{(\ell)}$ is locally executable from state $s$ using a learned reachability cost:
   $$D_\phi(s, g^{(\ell)}) \le \epsilon_{exec}$$
   where $\epsilon_{exec}$ is a threshold.
3. Refinement Loop: If the condition is met, refinement stops, and $g^{(\ell)}$ is passed to the execution module. Otherwise, the planner proposes a new intermediate target $u^{(\ell)} \sim \pi_p(\cdot | s, g^{(\ell)})$, which becomes the next target $g^{(\ell+1)}$.
4. Termination: The process repeats until the target is locally reachable or a maximum depth $K_{max}$ is reached.

This mechanism ensures the effective decomposition depth is determined by the estimated reachability of each state–goal pair, not a fixed temporal scale.

2.3 Offline Learning of the Refinement Planner

The planner is trained entirely from offline data without online interaction or manually labeled subgoals:

• Candidate Construction: For a state $s_t$ and goal $g$, a set of candidate subgoals $\mathcal{C}_D$ is sampled from the offline dataset (future states if $g$ is in the same trajectory, otherwise from the replay distribution).
• Scoring: Candidates are scored by the bottleneck reachability cost:
  $$C_\phi(\tilde{s}_i | s_t, g) = \max\{D_\phi(s_t, \tilde{s}_i), D_\phi(\tilde{s}_i, g)\}$$
  This metric favors candidates that contract the maximum residual horizon.
• Training Objective: The planner is trained via value-guided weighted imitation. Candidates are weighted by a softmax over their negative scores, encouraging the planner to imitate replay-supported subgoals that decompose the problem into easier residual reaching problems.

2.4 Execution Policies

The abstraction and low-level policies are trained using progress-weighted behavior cloning on short-horizon segments. Transitions that increase the learned value toward a local target are assigned higher imitation weights, biasing the policies toward locally progressive behavior.

3. Theoretical Analysis

The paper provides a stylized analysis to justify why adaptive refinement is more robust than fixed local-scale selection under noisy estimates:

• Fixed Scale Failure: In a fixed local-scale hierarchy, as the total horizon $T$ increases, the signal-to-noise ratio for distinguishing between correct and incorrect local moves vanishes. The probability of selecting the wrong candidate approaches 0.5 as $T \to \infty$.
• Approximate Contraction Sufficiency: The analysis shows that exact midpoint prediction is unnecessary. It is sufficient for the learned cost to preserve a coarse margin between candidates that contract the horizon (reduce the remaining distance by a constant factor $\rho < 1$) and those that do not.
• Error Bounds: The probability of overall failure is bounded by the sum of refinement failures (failing to select a contracting subgoal) and final local execution errors. This motivates the value-thresholded stopping rule: refine while the target is too distant, and stop once the target is estimated to be locally executable.

4. Experimental Results

The authors evaluate CFHRL on OGBench, a benchmark for offline goal-conditioned control, covering locomotion (PointMaze, AntMaze, HumanoidMaze, AntSoccer) and manipulation (Cube, Scene) tasks.

• Performance: CFHRL achieves the highest aggregate score (37.5) compared to baselines like HIQL (23.2), GCIVL (16.5), and flat methods.
• Long-Horizon Gains: The most significant improvements are observed in long-horizon maze-style tasks (e.g., PointMaze-Giant-Stitch), where CFHRL improves success rates from 0% (for flat/fixed-hierarchy baselines) to 57%.
• Ablation Studies:
  • Adaptive Stopping: Removing the adaptive stopping rule and forcing fixed refinement depths ($K=2, 3$) degrades performance, confirming that excessive refinement accumulates planner errors.
  • Planner Necessity: Replacing the learned planner with random dataset subgoals results in failure, proving that subgoals must be selected based on learned reachability costs.
  • Candidate Budget: Increasing the number of training candidates ($N_c$) improves performance up to a point ($N_c=16$), balancing supervision quality and training throughput.
• Limitations: CFHRL underperforms on contact-rich manipulation tasks (Cube) compared to flat value-based methods, suggesting that waypoint-like decomposition is less effective for tasks requiring precise contact dynamics.

5. Significance and Claims

The paper claims three primary contributions:

1. Formulation: It formulates long-horizon offline GCRL as adaptive stopping-time subgoal refinement, where the decomposition depth is determined by learned executability rather than a fixed horizon.
2. Algorithm: It introduces a replay-supported contraction objective that trains an amortized refinement planner without requiring test-time graph search or online interaction.
3. Validation: It provides both a stylized analysis and empirical ablations demonstrating that approximate contraction and adaptive stopping effectively improve long-horizon goal reaching in maze-style environments.

The authors maintain a modest stance, acknowledging that the method relies on the quality of the learned reachability cost and that the current abstraction interface (simple local goal representations) may be insufficient for complex manipulation tasks requiring precise contact sequences. They suggest future work should focus on combining adaptive planning with richer object-centric abstractions and extending the framework to other offline GCRL value estimators.