Technical Summary: Mollified Value Learning

Problem Statement

Offline Goal-Conditioned Reinforcement Learning (GCRL) aims to learn policies that reach specific goal states from static, pre-collected datasets without further environment interaction. A central challenge in this setting is the accurate estimation of the Goal-Conditioned Value Function (GCVF). While standard value parameterizations rely on temporal-difference (TD) recursion, they often lack the structural inductive biases necessary to capture the underlying goal-reaching geometry. This deficiency leads to uninformative or poorly structured value estimates, particularly in sparse-reward settings and high-dimensional environments.

Recent physics-informed approaches attempt to address this by imposing pointwise geometric constraints derived from the Hamilton–Jacobi–Bellman (HJB) optimality principles, often utilizing the Eikonal equation to model shortest-path geometry. However, enforcing local consistency through explicit differential operators (e.g., first-order partial differential equations) becomes unstable in complex, high-dimensional environments. These methods suffer from numerical instability due to sparse state-action coverage, complex kinematics, and the practical difficulties associated with Physics-Informed Neural Networks (PINNs).

Methodology: Mollified Value Learning (MVL)

The authors propose Mollified Value Learning (MVL), a framework that reinterprets distance-like geometric constraints as an expectation over a local spatial measure rather than enforcing them pointwise.

Core Insight

Instead of requiring the value function to satisfy strict differential equations (which are sensitive to noise and high dimensionality), MVL reframes the shortest-path consistency as a spatial expectation. The method posits that the cost-to-go relationship should hold in expectation over a local neighborhood of states. This approach acts as a "spatial mollifier," inducing distance-like value geometry without requiring expensive or unstable differential operators.

Theoretical Formulation

1. Geometric Consistency: The optimal value function $V^*(s, g)$ corresponds to the shortest feasible path (geodesic) from state $s$ to goal $g$. This satisfies a triangle inequality: $V(s, g) \geq V(s, s') + V(s', g)$.
2. Local Relaxation: For a transition from $s$ to a neighbor $s'$ within a small time step $\Delta t$, the cost is approximated as $c(s, g)\Delta t \approx c(s, g)\|s' - s\|$. The local consistency condition is defined as:
   $$V(s', g) - V(s, g) - c(s, g)\|s' - s\| \leq 0$$
3. Mollified Objective: Rather than enforcing this inequality for every specific transition, MVL defines a residual operator $\Phi(V, s, s', g) = V(s', g) - V(s, g) - c(s, g)\|s' - s\|$ and enforces its expectation over a local spatial measure $\rho_\delta(s'|s)$:
   $$\mathcal{R}_\delta[V](s, g) := \int \Phi(V, s, s', g) \rho_\delta(s'|s) ds'$$
   Here, $\rho_\delta$ is a localized transition kernel (e.g., a Gaussian distribution centered at $s$ with radius $\delta$).
4. Loss Function: The geometric regularization loss is the empirical expectation of the squared positive part of this residual (using ReLU to penalize violations):
   $$L_{MVL}(\theta) = \mathbb{E}_{s,g} \left[ \text{ReLU}(\mathcal{R}_\delta[V_\theta](s, g))^2 \right]$$

Practical Implementation

In model-free offline settings, the integral is approximated by sampling a finite set of neighbors $\{s'_1, \dots, s'_N\}$ from a Gaussian distribution $\mathcal{N}(s, \delta^2 I)$. The method is compatible with existing implicit value representation learning methods (such as GCIVL) and does not require explicit higher-order differential operators, making it computationally tractable and robust to dataset stochasticity.

Key Contributions

1. Novel Formulation: The paper introduces MVL, a value learning framework that injects shortest-path-like geometric inductive bias into offline GCRL by solving for expected cost-to-go constraints over local state distributions.
2. Stability and Tractability: By replacing explicit differential operators with a tractable integral over a local spatial measure, MVL avoids the numerical instability associated with high-dimensional Eikonal and HJB formulations.
3. Robustness to Noise: The method acts as a geometric low-pass filter, decoupling spatial routing geometry from low-level dataset noise and unmodeled dynamics. This is particularly effective in stochastic environments where pointwise constraints often fail.
4. Empirical Validation: The approach is validated across 13 state-based tasks (navigation and manipulation) and real-world robotic experiments, demonstrating improved goal-reaching performance and structured value representations.

Experimental Results

The authors evaluated MVL on the OGBench benchmark (covering navigation tasks like PointMaze, AntMaze, and HumanoidMaze, and manipulation tasks like Cube, Scene, and Puzzle) and the D4RL Franka Kitchen suite.

• Performance Gains: MVL significantly improves goal-reaching success rates when combined with implicit value learning baselines. For instance, on the cube-single-play task, adding MVL to the Dual-goal representation increased success from 89% to 91%. In the scene-play task, it improved performance from 72% to 84%.
• Comparison to Baselines: MVL outperforms standard baselines (GCIVL, VIB, VIP, TRA, BYOL) and physics-informed baselines using explicit Eikonal constraints (DUAL+EIK). Notably, in complex manipulation tasks where Eikonal-based methods struggle or fail (e.g., cube-single-play where Eikonal achieved only 1% success vs. MVL's 91%), MVL provides robust performance.
• Robustness to Stochasticity: In noisy environments (e.g., cube-single-noisy), baseline methods often collapsed to near-zero success, whereas MVL maintained high performance (99% success), demonstrating its ability to filter out transition noise.
• Real-World Deployment: The method was successfully deployed on a Franka Emika Panda robot for "Reach and Grasp" and "Pick and Place" tasks. Regularization improved success rates from 20% to 80% in Reach and Grasp and from 0% to 100% in Pick and Place, highlighting its utility in handling real-world sensor variance and friction.

Significance and Claims

The paper claims that MVL provides a simple and tractable inductive bias that improves value learning in high-dimensional manipulation and navigation tasks. The primary significance lies in demonstrating that the geometry of GCRL value functions can be defined without relying on expensive and unstable differential formulations.

The authors emphasize that their approach offers a "weak" geometric consistency that preserves large-scale routing structures while attenuating small-scale irregularities caused by sparse data or noise. They acknowledge limitations, noting that the formulation relies on Euclidean distance and isotropic kernels, which may not perfectly model non-holonomic constraints or complex contact dynamics. However, they argue that this simplification allows for effective learning in high-dimensional state spaces where exact dynamics are unknown, offering a practical alternative to exact HJB solutions.

The work suggests a broader direction for future research: extending these geometric regularization ideas to learned or manifold-aware representation spaces, particularly in visual domains where Euclidean assumptions in latent spaces may not hold.