Technical Summary: ISEP – Implicit Support Expansion for Offline Reinforcement Learning

1. Problem Statement

Offline Reinforcement Learning (RL) aims to learn effective policies from static datasets without online interaction, a capability critical for high-stakes domains like robotics and healthcare. A fundamental challenge in this setting is the tension between conservatism (staying within the dataset's support to ensure valid value estimation) and optimism (generalizing beyond suboptimal data to discover improved behaviors).

Existing methods typically resolve this by enforcing strict support constraints: limiting updates to in-sample transitions, restricting actions to the behavior policy's support, or penalizing overestimated critic values. While these constraints ensure safety, they often suppress high-reward behaviors that are sparsely represented in the dataset. This "density bias" prevents the discovery of optimal policies, particularly when the optimal trajectory lacks sufficient data density. Furthermore, standard deterministic interpolation between conservative data and optimistic proposals often leads to mode collapse, where the policy averages distinct valid actions into a low-value, invalid action lying off the data manifold.

2. Methodology

The authors propose ISEP (Implicit Support Expansion via stochastic Policy optimization), a framework designed to safely extrapolate beyond static datasets. ISEP operates through three coupled innovations:

2.1 Interpolated Value Estimation (Implicit Support Expansion)

Instead of treating the dataset distribution as a rigid boundary, ISEP implicitly expands the valid support of the value function to encompass high-value actions that are valid but sparsely represented. This is achieved via a hybrid objective function for the value network $V_\psi$:

$$L_V(\psi) = (1 - p)\mathbb{E}_{(s,a)\sim D} [L_\tau^2(\hat{Q}_\theta(s, a) - V_\psi(s))] + p\mathbb{E}_{s\sim D, \hat{a}\sim \pi_\phi} [(\hat{Q}_\theta(s, \hat{a}) - V_\psi(s))^2]$$

• Term 1: Anchors value estimates to real transitions using asymmetric expectile loss (standard IQL approach).
• Term 2: Drives expansion by querying the Q-function with actions sampled from the current policy $\hat{a} \sim \pi_\phi$.
• Parameter $p$: A controllable scalar governing the balance between safety (in-distribution) and exploration (policy-generated). This mechanism "densifies" high-reward regions, creating a navigable path for policy improvement.

2.2 Stochastic Action Selection

To address the multimodal nature of the resulting objective, ISEP employs a Stochastic Action Selection strategy. Standard deterministic interpolation (averaging gradients) forces the policy toward the arithmetic mean of a conservative action and an optimistic action, often resulting in a low-value "off-manifold" action.

ISEP avoids this by stochastically alternating between targets at each update step using a Bernoulli gate $B \sim \text{Bernoulli}(p)$:

$$L_\pi(\phi) = (1 - B) \cdot \mathbb{E}_{(s,a)\sim D}[\omega(s, a) \log \pi_\phi(a|s)] + B \cdot \mathbb{E}_{s\sim D, \hat{a}\sim \pi_\phi}[\omega(s, \hat{a}) \log \pi_\phi(\hat{a}|s)]$$

This ensures the policy is pushed toward distinct, valid modes (either the conservative data sample or the optimistic proposal) rather than collapsing into a low-value valley between them.

2.3 Instantiation: ISEP-FM

Because the stochastic strategy necessitates a policy capable of representing complex, multimodal distributions, the authors instantiate the framework using Conditional Flow Matching (CFM) with Classifier-Free Guidance (CFG).

• The flow model is conditioned on an optimality token $o \in \{\emptyset, 1\}$.
• At inference, actions are generated by integrating the velocity field with a guidance scale $w$, allowing dynamic control between the base behavior distribution and the high-value optimality condition.

3. Theoretical Guarantees

The paper provides a theoretical analysis demonstrating that ISEP guarantees bounded value error. Under assumptions of finite sample sizes (Expectile Gap) and dataset sub-optimality, the authors prove that if the interpolation parameter $p$ satisfies a specific bound derived from dataset characteristics:

$$p \leq \frac{\delta_\tau + \delta_{sub}}{V_{max} - V^*(s) + \delta_\tau + \delta_{sub}}$$

Then, with probability at least $\eta$ (governed by dataset quality), the learned value function $\hat{V}(s)$ remains upper-bounded by the optimal state value $V^*(s)$. This establishes $p$ as a rigorous safety mechanism that prevents catastrophic divergence while enabling controlled extrapolation.

4. Experimental Results

The authors evaluated ISEP and its flow-matching variant (ISEP-FM) on the D4RL benchmark, covering Gym-MuJoCo locomotion, Adroit robotic hand manipulation, and Kitchen sequential object manipulation.

• Performance: ISEP-FM achieved state-of-the-art results across multiple environments. For instance, on hopper-med-rep, it achieved a normalized score of 104.5, outperforming strong baselines like QIPO (101.2) and IDQL (82.4). On halfcheetah-med, it scored 59.9, surpassing the previous best of 54.1.
• Ablation Studies:
  • Parameter $p$: Moderate values ($p \in \{0.3, 0.5\}$) consistently yielded the highest scores. Aggressive values ($p \geq 0.9$) led to catastrophic divergence, confirming the need for a "middle-ground" approach.
  • Stochastic vs. Deterministic: Replacing stochastic selection with deterministic interpolation resulted in significant performance drops, validating that naive averaging fails in non-convex action landscapes.
  • Support Expansion: Visualizations (t-SNE) confirmed that ISEP successfully expands action support into sparse, high-reward regions that strictly in-distribution methods (like IQL with $p=0$) fail to reach due to density bias.

5. Significance and Claims

The paper claims that ISEP addresses the critical limitation of offline RL methods that treat dataset distributions as rigid boundaries. By implicitly expanding the feasible action support through a value function interpolated between in-distribution data and policy samples, ISEP "densifies" high-reward regions without requiring explicit constraints or complex two-stage pipelines.

The significance of this work lies in its ability to:

1. Bridge the gap between safety and optimism, allowing agents to discover superior actions in sparsely represented regions of the state-action space.
2. Prevent mode collapse in non-convex landscapes through a novel stochastic action selection strategy, which is essential for multimodal policy learning.
3. Provide theoretical safety guarantees, showing that value estimates remain bounded even when exploring out-of-distribution actions, provided the interpolation parameter is tuned according to dataset quality.

The authors conclude that combining support expansion with the expressivity of modern generative models (Flow Matching) enables offline RL agents to effectively discover superior behaviors without sacrificing safety.