Technical Summary: Generative Actor-Critic with Soft Bridge Policies

Problem Statement

Maximum-entropy (MaxEnt) online reinforcement learning (RL) is a dominant approach for off-policy continuous control, balancing value maximization with explicit stochasticity. While standard Soft Actor-Critic (SAC) relies on tractable Gaussian policies, complex control problems often require multimodal or highly non-Gaussian action distributions. Expressive generative policies (e.g., diffusion and flow models) offer a solution but face two critical obstacles when applied to MaxEnt RL:

1. Intractable Entropy: MaxEnt updates require the policy's endpoint density or an equivalent regularizer. For generative actors, the marginal action density is often unavailable or expensive to compute because the action is obtained by marginalizing over latent paths. Existing methods resort to entropy bounds, heuristic proxies, or approximations, creating an indirect connection to the MaxEnt objective.
2. Computational Cost: High-performance generative policies often use iterative samplers (e.g., diffusion with many denoising steps) that reuse shared network parameters. This raises inference costs at deployment and, crucially, increases training costs. During actor updates, backpropagation through time (BPTT) must traverse every refinement step, increasing memory usage and destabilizing optimization. Reducing the number of function evaluations (NFE) lowers cost but often causes a sharp performance drop.

The paper seeks a generative actor that retains the expressiveness of stochastic latent generation while exposing a tractable MaxEnt objective and avoiding the computational burden of long, shared-parameter samplers.

Methodology: SoftGAC

The authors propose SoftGAC (Soft Generative Actor-Critic), a method that lifts the MaxEnt objective from the terminal action distribution to the full latent generation path.

1. Path-Space Soft Objective

Instead of optimizing the entropy of the final action $a$, SoftGAC optimizes a path-space objective over the latent trajectory $\tau = (z_0, \dots, z_K)$. The objective is defined as:
$$J_{path}(P_\theta | s) = \mathbb{E}_{\tau \sim P_\theta} [Q(s, T(z_K))] - \alpha D_{KL}(P_\theta(\cdot|s) \| R)$$
where $R$ is a high-entropy reference path law.

• Theoretical Foundation: The authors prove that the path KL divergence decomposes into an endpoint KL term (recovering the standard MaxEnt regularizer) and a conditional path term. This path-wise relative entropy acts as a stricter regularizer that controls how the actor reaches the terminal action, not just the distribution of the action itself.
• Fixed-Base Constraint: In practice, the actor starts from a fixed base latent distribution $p_0$. Theoretical analysis shows that while a fixed base introduces a constant gap relative to the unrestricted optimum, it does not break the variational problem; it simply acts as an inductive bias.

2. Soft Bridge Policies

To make the path KL tractable and computationally efficient, SoftGAC employs a soft bridge policy:

• Architecture: The actor is a finite Markov chain in pre-tanh latent space. It starts from a fixed base law (logistic distribution) and applies $K$ lightweight, step-specific Gaussian residual transitions to reach a terminal latent $z_K$.
• Single-Pass Design: Unlike diffusion models that reuse a single network for $K$ steps, SoftGAC uses $K$ distinct transition blocks. This allows action generation via a single sampled forward pass through the network, avoiding iterative sampling and long BPTT graphs.
• Reference Process: The reference bridge $R$ is defined using a logistic base and an Euler Gaussian kernel derived from the score of the pre-tanh density that maps to a uniform action distribution.
• Tractable Regularizer: Because both the actor and reference transitions are Gaussian, the path KL divergence factorizes into a sum of local KL terms. This sum, denoted as Control Energy $C_\theta(s, \tau)$, is analytically computable.
  $$D_{KL}(P_\theta \| R) = D_{KL}(p_0 \| r_0) + \mathbb{E}[C_\theta(s, \tau)]$$
  The actor update minimizes $\alpha C_\theta(s, \tau) - Q(s, a)$, effectively trading off value against the sampled transition control energy.

3. Training Algorithm

SoftGAC follows an off-policy actor-critic loop:

• Critic: Updates a soft value function $Q$ using targets that include the sampled control energy of the next-state path: $y = r + \gamma(Q_{min}(s', a') - \alpha C_{\bar{\theta}}(s', \tau'))$.
• Actor: Updates parameters to minimize the sampled control energy while maximizing the Q-value.
• Temperature Tuning: A dual update adjusts $\alpha$ to maintain the average control energy near a target budget $\rho_{ctrl} K d_a$.

Key Contributions

1. Path-Space Formulation: The authors formulate a path-space soft objective for generative actors, demonstrating that it contains the endpoint KL regularizer as a marginal component and characterizes the effect of using a fixed actor base.
2. Soft Bridge Policy Design: They design a specific policy architecture using a short sequence of local Gaussian transitions in pre-tanh space. This makes the path-wise regularizer exactly computable as finite-step control energy, enabling a direct actor update with action generation via a single sampled pass.
3. Compute-Return Tradeoff: Experiments demonstrate that SoftGAC achieves higher or competitive returns compared to strong generative baselines (diffusion and flow-matching) while maintaining the low-latency regime of one-pass actors.

Experimental Results

The method was evaluated on challenging continuous-control benchmarks from DeepMind Control Suite (Humanoid, Dog) and HumanoidBench (H1).

• Performance: SoftGAC attained the strongest overall performance on high-dimensional, long-horizon locomotion tasks (e.g., Humanoid Run, Dog Run, H1 Hurdle), outperforming both generative baselines (FLAC, DIME, FlowRL, QSM, QVPO) and the unimodal Gaussian baseline (CrossQ-SAC).
• Efficiency:
  • Inference Time: SoftGAC operates in the low-latency regime (approx. 61–75 $\mu$s per action on CPU), comparable to one-step flow baselines and significantly faster than high-NFE diffusion baselines (which take 300–400+ $\mu$s).
  • Parameter Budget: The actor parameter count is comparable to strong baselines, avoiding the need for larger networks to hide the cost of iterative sampling.
• Ablations: Removing the soft path-space regularizer ($\alpha=0$) consistently degraded performance, confirming that the gains stem from the principled soft objective rather than just the bridge parameterization.

Significance and Claims

The paper claims that SoftGAC resolves the tension between the expressiveness of generative policies and the computational/optimization constraints of MaxEnt RL. By treating the actor as a lightweight, explicit path law rather than an endpoint distribution, the method:

• Provides a principled soft regularization (sampled transition control energy) that is analytically tractable, replacing heuristic entropy bounds.
• Avoids the optimization burden of long BPTT chains associated with iterative samplers, stabilizing actor updates.
• Delivers a superior compute-return tradeoff, achieving state-of-the-art returns on difficult tasks without the high inference latency of multi-step diffusion samplers.

The authors conclude that soft bridge policies represent a practical design for efficient generative actor-critic learning, offering a viable alternative to iterative diffusion approaches in online RL settings.