🤖 machine learning

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

This paper introduces Latent-Aligned Value Learning (LAVL), an offline goal-conditioned reinforcement learning algorithm that integrates latent-representation-based value generalization with hierarchical planning to overcome erroneous generalization in long-horizon tasks, achieving state-of-the-art performance on the OGBench.

Original authors: Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to navigate a massive, complex maze or to stack blocks into a tower. You don't want the robot to learn by trial and error (which takes forever and is dangerous); instead, you want to teach it using a giant video library of someone else's past attempts. This is called Offline Goal-Conditioned Reinforcement Learning.

The paper introduces a new method called LAVL (Latent-Aligned Value Learning) that solves a major problem with how robots currently learn from these video libraries.

Here is the breakdown of the problem and the solution, using simple analogies.

The Problem: The "Map vs. Reality" Confusion

Imagine you are teaching a robot to get from Point A to Point B in a maze. You show it a video of a person walking the maze.

The Old Way (The Flawed Map):
Most current methods try to guess how "good" a specific spot in the maze is based on how close it looks to the goal.

The Analogy: Imagine the robot has a map where distance is measured in a straight line (like a bird flying). If the goal is 10 feet away, but there is a thick wall between the robot and the goal, the robot thinks, "Oh, it's only 10 feet away! I can make it!"
The Result: The robot gets confused. It thinks it's close to the goal because it's visually close, even though it's temporally far away (it would take 100 steps to get there). This is called erroneous generalization. The robot learns a "broken map" where walls don't exist, leading to bad decisions.

The Quasimetric Fix (The Rigid Rulebook):
Some researchers tried to fix this by forcing the robot's brain to follow strict mathematical rules (called "quasimetrics") that say, "You can't cut through walls."

The Analogy: This is like giving the robot a rigid rulebook that says, "Always calculate distance using this specific, complex formula."
The Result: It works great for simple mazes, but when you give the robot a complex task like picking up a cube with a robotic arm, the rigid rulebook breaks. The robot gets confused and fails completely. It's too stiff to adapt to different types of tasks.

The Solution: LAVL (The Smart GPS)

The authors propose LAVL, which uses a new way of thinking about distance. Instead of measuring how close things look, or following a rigid rulebook, LAVL teaches the robot to understand the "hidden geometry" of the task.

1. The "Latent Alignment" (The Secret Language):
Instead of looking at the raw pixels of the maze or the arm, LAVL translates the robot's current state and the goal into a "secret language" (a latent space).

The Analogy: Imagine the robot and the goal are speaking different dialects. The old methods tried to translate them word-for-word (Euclidean distance). LAVL translates both into a universal language where the meaning of the distance is preserved.
How it works: The robot learns that "State A" and "Goal B" are far apart in this secret language, even if they look close on a screen. This prevents the "wall-leaking" error. It's like having a GPS that understands traffic and detours, not just straight-line distance.

2. The "Continuity" Glue:
When a robot tries to learn a long path, its internal map can get shaky and jittery.

The Analogy: Imagine the robot's map is a piece of paper that keeps crinkling up. One second the goal is 5 steps away; the next second, it's 500 steps away, just because of a tiny glitch.
The Fix: LAVL adds a "smoothing glue" (continuity regularization). It tells the robot: "If you are in a spot very similar to the one before, your estimate of the distance to the goal shouldn't change wildly." This keeps the map stable and smooth.

3. The "Hierarchical" Boss and Worker:
For very long tasks (like a giant maze), the robot needs a plan.

The Analogy: LAVL uses a two-level system.
- The Boss (High-level): Looks at the big picture. "We need to get to the exit. Let's aim for the corner first."
- The Worker (Low-level): Handles the details. "Okay, I'm at the corner. Now I'll turn left and walk forward."
LAVL makes sure the Boss and the Worker are speaking the same "secret language" (Latent Alignment), so they don't get confused.

The Results: Why It Matters

The authors tested this on OGBench, a giant benchmark with 22 different challenges, ranging from navigating giant mazes to stacking cubes with a robotic arm.

The Score: LAVL won on 20 out of 22 datasets.
The Long Haul: In the "giant" mazes (where the path is thousands of steps long), other methods failed or got stuck. LAVL kept performing well.
The Stitching: Some datasets were made of short, broken video clips that the robot had to "stitch" together to form a full path. LAVL was much better at connecting these dots than previous methods.

Summary

The paper argues that the biggest bottleneck in teaching robots from old data is that they use the wrong kind of "distance" to measure progress. They measure how things look rather than how hard they are to reach.

LAVL fixes this by:

Learning a "secret language" (Latent Alignment) to measure true difficulty.
Smoothing out the robot's internal map so it doesn't get jittery.
Using a Boss/Worker system to handle long, complex tasks.

The result is a robot that can learn from a video library and actually figure out how to reach complex goals without getting confused by walls or long distances.

Technical Summary: Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

1. Problem Statement

Offline Goal-Conditioned Reinforcement Learning (GCRL) aims to learn policies that reach arbitrary goals using fixed, pre-collected datasets without further environment interaction. While promising, learning reliable goal-conditioned value functions in long-horizon tasks remains a significant challenge. The primary bottleneck identified in this work is erroneous generalization in goal-conditioned value functions.

Standard value functions, particularly those parameterized by Multi-Layer Perceptrons (MLPs), tend to generalize based on Euclidean proximity in the state space rather than temporal distance. In environments with obstacles (e.g., mazes) or sparse rewards, this leads to "value leakage," where states physically close to a goal but temporally distant (separated by walls or complex dynamics) are assigned erroneously high values. This results in noisy value estimates and brittle control policies. Furthermore, existing methods attempting to address this via quasimetric architectures (e.g., IQE, MRN) often suffer from inconsistent performance across different domains (e.g., performing well in maze navigation but failing in robotic manipulation) or introduce training instability.

2. Methodology: Latent-Aligned Value Learning (LAVL)

The authors propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates a novel value function architecture with a hierarchical planning framework.

2.1. Latent Alignment Network (LAN)

The core innovation is the Latent Alignment Network (LAN), a value function architecture designed to enforce appropriate inductive bias for value generalization.

Architecture: Unlike standard MLPs or strict quasimetric models, LAN parameterizes the goal-conditioned value $V(s, g)$ as the negative Euclidean distance between learned latent representations of the state and the goal:
$V(s, g) = -\|\phi_S(s) - \phi_G(g)\|^2$
where $\phi_S: \mathcal{S} \to \mathbb{R}^d$ and $\phi_G: \mathcal{G} \to \mathbb{R}^d$ are asymmetric state and goal representation networks.
Mechanism: By breaking the strict metric structure (allowing $\phi_S \neq \phi_G$ ), LAN introduces flexibility that allows the value function to generalize based on the geometry of the learned latent space rather than raw Euclidean state proximity. This mitigates the "overgeneralization" issue where value propagates through obstacles.
Advantages: LAN avoids the training instability and domain-specific performance drops associated with strict quasimetric constraints (like IQE) while maintaining the structural bias necessary for accurate temporal distance estimation.

2.2. Value Learning with Continuity Regularization

To stabilize learning in long-horizon settings, LAVL augments the standard Implicit Value Learning (IVL) objective with a local continuity regularization term.

Objective: The total loss combines the TD objective ( $L_{TD}$ ) with a regularization term ( $L_{Reg}$ ) that penalizes sharp local fluctuations in the value function between neighboring states:
$L(V) = L_{TD}(V) + w_c L_{Reg}(V)$
Function: $L_{Reg}$ suppresses high-frequency noise in the value landscape without compromising global Bellman consistency, ensuring stable propagation of sparse rewards over long horizons.

2.3. Hierarchical Policy Framework

LAVL adopts a hierarchical policy framework similar to HIQL, consisting of:

High-level policy ( $\pi_h$ ): Generates a subgoal representation $w$ conditioned on the current state $s$ and final goal $g$ . In LAVL, the subgoal is defined as a goal-only mapping $w = \phi(g)$ , aligning naturally with the LAN parameterization.
Low-level policy ( $\pi_l$ ): Outputs actions conditioned on the state $s$ and subgoal $w$ .
Training: Both policies are trained using Advantage-Weighted Regression (AWR), leveraging the stable value estimates provided by LAN.

3. Key Contributions

The paper outlines three primary contributions:

Identification of Erroneous Generalization: The authors identify that the failure of offline GCRL in long-horizon tasks is fundamentally driven by value functions generalizing via Euclidean proximity instead of temporal distance. They demonstrate that inductive bias in the value function architecture is the critical factor for resolving this, rather than the learning objective alone.
Proposal of LAVL: They introduce LAVL, a unified framework combining the Latent Alignment Network (LAN) for reliable value generalization, continuity regularization for stable learning, and a hierarchical policy for long-horizon planning.
Empirical Superiority: Extensive experiments on the OGBench benchmark demonstrate that LAVL consistently outperforms existing state-of-the-art offline GCRL methods.

4. Experimental Results

The authors evaluated LAVL on 22 datasets from OGBench, covering maze navigation (Point, Ant, Humanoid) and robotic manipulation (Cube, Scene) tasks with varying horizons and data collection strategies (navigate vs. stitch).

Overall Performance: LAVL achieved the highest performance on 20 out of 22 datasets, significantly outperforming baselines like GCIVL, QRL, HIQL, and CGCIVL.
Long-Horizon Stability: In "giant" maze environments, LAVL maintained an 85% success rate with a relative drop of only 9.6% compared to medium-sized mazes. In contrast, baselines like HIQL and QRL suffered catastrophic performance drops (75.4% and 61.7% relative drops, respectively).
Trajectory Stitching: On "stitch" datasets (composed of short, fragmented trajectories), LAVL showed a minimal performance drop (1.1%) compared to navigate datasets, whereas other methods degraded significantly (e.g., 36.9% drop for HIQL).
Robotic Manipulation: Unlike quasimetric methods (e.g., QRL) which failed in manipulation tasks, LAVL achieved high success rates (e.g., 88% on Scene-play), demonstrating the robustness of LAN across diverse domains.
Ablation Studies: Experiments confirmed that:
- LAN outperforms quasimetric architectures (IQE, MRN) and metric-based approaches (Hilbert) within the same hierarchical framework.
- The combination of LAN and hierarchical policies yields superior results compared to flat policies or using LAN with flat policies alone.
- Continuity regularization provides significant gains on challenging tasks (e.g., increasing success rate on Pointmaze-giant-stitch from 35% to 95% in some configurations).

5. Significance and Claims

The paper claims that LAVL represents a significant advancement in offline GCRL by addressing the fundamental bottleneck of value generalization.

Architectural Insight: The work highlights that the choice of value function architecture is more critical than the learning objective for achieving reliable generalization. It argues that while optimal value functions are quasimetric, enforcing strict quasimetric constraints during training can be overly restrictive; instead, learned latent alignment offers a more effective inductive bias.
Unified Framework: LAVL successfully unifies effective generalization, stable value learning, and hierarchical planning, enabling robust performance in tasks requiring long-horizon reasoning and trajectory stitching.
Practical Impact: The method achieves state-of-the-art results without requiring extensive hyperparameter tuning or oracle representations, making it a practical solution for complex offline decision-making problems.

The authors conclude that while their work focuses on value learning and hierarchical extraction, the findings regarding architectural inductive bias may extend to broader value-based RL algorithms, and future work could explore extending these methods to high-dimensional real-world robotic control.