Technical Summary: Dual Advantage Fields

Problem Statement

Offline Goal-Conditioned Reinforcement Learning (GCRL) faces a fundamental dual challenge: the agent must simultaneously reason globally about long-horizon reachability (inferring how states connect over time to stitch together behaviors from a fixed dataset) and act locally (deciding which specific action at the current state makes the most progress toward a goal).

While recent dual goal representations (parameterizing value as a bilinear interaction $V_\theta(s, g) = \psi_\theta(s)^\top \phi_\theta(g)$) provide strong global maps of reachability, they fail to directly specify action preferences. A value surface indicates how good a state is for a goal but does not inherently rank available actions, as multiple actions may originate from the same state with identical value estimates. Existing methods often struggle to extract a local, action-comparative signal from these global fields, leading to suboptimal performance in tasks where locally correct actions (e.g., pre-grasping) differ from direct movement toward the final goal.

Methodology: Dual Advantage Fields (DAF)

The authors propose Dual Advantage Fields (DAF), a policy-extraction method that leverages the geometric properties of bilinear dual representations to generate a local advantage signal without training a separate goal-conditioned action-value function ($Q$-function).

Core Insight

Under the bilinear parameterization $V_\theta(s, g) = \psi_\theta(s)^\top \phi_\theta(g)$, the goal embedding $\phi_\theta(g)$ is mathematically equivalent to the gradient of the value field with respect to the state embedding:
$$\nabla_\psi V_\theta(s, g) = \phi_\theta(g)$$
This implies that the goal embedding defines the direction in the state-representation space along which the goal-conditioned value increases.

Mechanism

DAF operationalizes this insight through the following steps:

1. Action-Effect Modeling: The method learns an action-effect model $u_\xi(s, a)$ that predicts the discounted feature displacement induced by an action:
   $$u_\xi(s, a) \approx \mathbb{E}_{s' \sim P(\cdot|s,a)} [\gamma \psi_\theta(s') - \psi_\theta(s)]$$
2. Dual Advantage Scoring: Actions are scored by the inner product between the predicted feature displacement and the goal embedding (the value-gradient direction):
   $$z_\theta(s, a, g) = u_\xi(s, a)^\top \phi_\theta(g)$$
   This score measures whether the action moves the state representation in a direction that increases the goal-conditioned potential.
3. Policy Extraction: The method employs Advantage-Weighted Regression (AWR) to extract a policy $\pi_\omega$. The policy is trained to maximize the log-probability of actions weighted by the exponential of the DAF score:
   $$J(\pi_\omega) = \mathbb{E}_{(s,a,g) \sim D} \left[ \exp(\alpha \cdot z_\theta(s, a, g)) \log \pi_\omega(a|s, g) \right]$$
   In the realizable case, this score equals the goal-conditioned Bellman advantage, providing a standard local policy-improvement guarantee.

Training Architecture

The training loop involves:

• Learning the state and goal embeddings ($\psi_\theta, \phi_\theta$) using expectile regression on offline data, tied to pessimistic $Q$-estimates.
• Training the displacement map $u_\xi$ via an auxiliary loss to track one-step feature dynamics.
• An "actor-free" coupling between the value field and the scalar dual score to ensure stability in continuous control, avoiding brittle max-$Q$ operators.

Key Contributions

1. Geometric Interpretation: The paper demonstrates that under standard dual goal representation parameterization, the goal embedding acts as the gradient direction of the goal-conditioned value field with respect to the learned state representation.
2. DAF Algorithm: Introduction of a policy-extraction method that learns action-effect vectors and scores actions based on their alignment with the goal direction, producing a local advantage-like signal.
3. Actor-Free Extraction: A mechanism to extract policies from offline data without training a separate goal-conditioned $Q$-function, reusing the geometry of the dual critic to weight offline actions.

Experimental Results

The authors evaluated DAF on the OGBench benchmark, covering locomotion, manipulation, and puzzle tasks.

• Maze Locomotion (Long-Horizon): On humanoidmaze and antmaze (Navigate and Stitch datasets), DAF achieved competitive or superior results compared to horizon-reduction baselines (e.g., HIQL, OTA). Notably, DAF excelled in "Stitch" tasks, which require composing shorter trajectory fragments into long-horizon solutions, demonstrating that local advantage extraction can preserve long-horizon structure.
• Object Manipulation (Local Control): On cube and scene tasks, DAF significantly outperformed baselines. In the cube task, DAF successfully identified pre-grasp behaviors (moving toward the object) rather than directly moving to the final placement goal, a common failure mode for methods relying solely on global subgoals.
• Puzzle Rearrangement: On combinatorial puzzle tasks, DAF showed strong performance, effectively handling the need to compare actions based on their downstream effects on the goal configuration.
• Aggregate Metrics: Using RLiable metrics (Median, IQM, Mean, Optimality Gap), DAF improved aggregate performance across all domains, indicating robustness across different control structures and data regimes (noisy expert vs. suboptimal data).

Significance and Claims

The paper claims that Dual Advantage Fields provide a unified principle for offline GCRL: global value fields should be paired with local advantage fields. By interpreting the goal embedding as a gradient, DAF transforms dual representations from mere global maps into local compasses for action selection.

The authors emphasize that this approach is particularly significant in settings where locally correct actions differ from direct movement toward the final goal (e.g., pre-grasping). The results suggest that dual representations should not only serve as global reachability maps but also be explicitly utilized to extract local advantage signals for policy improvement, offering a more consistent performance across diverse offline GCRL benchmarks than previous hierarchical or quasimetric approaches.

The paper acknowledges limitations regarding dataset coverage; like other offline RL methods, DAF relies on the dataset covering necessary actions, and its performance depends on the accuracy of the learned dual representation and action-effect model in stochastic or poorly covered regions.