🤖 machine learning

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

This paper provides a unified closed-form analysis of post-training steering for frozen offline reinforcement learning policies, demonstrating that Product-of-Experts and KL-regularized adaptation function as "actor-anchored" safety mechanisms that offer graceful degradation rather than guaranteed performance gains when deployment objectives change.

Original authors: Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay, Ozlem Garibay, Niloofar Yousefi

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Elias Hossain, Mohammad Jahid Ibna Basher, Ivan Garibay, Ozlem Garibay, Niloofar Yousefi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a professional chef who has spent years mastering a very specific, highly regulated recipe for a classic Beef Wellington. This chef is "frozen"—meaning you aren't allowed to change their training, their tools, or their fundamental techniques because they have already been certified by a strict food safety board.

Suddenly, a customer walks in and asks for the same dish, but with a twist: "Can you make it much spicier and less buttery?"

In the world of AI, this is the problem this paper solves. Usually, if you want an AI to change its behavior, you have to "retrain" it, which is expensive, slow, and often impossible if the AI is already locked behind strict safety rules. This paper proposes a way to "steer" that frozen chef without ever changing their core training.

The Core Idea: The "Expert and the Guide"

The researchers use a method called Product-of-Experts (PoE). Think of it like this:

The Frozen Expert (The Chef): This is the original AI. It is incredibly reliable and knows exactly how to move, but it only knows how to do one specific thing.
The Goal-Conditioned Prior (The Sous-Chef/Guide): This is a second, lighter AI. It doesn't know how to cook a whole meal, but it does know a little bit about the new goal (like "spiciness").

Instead of forcing the Chef to learn a new skill, you let the Chef and the Guide work together at the very last second. When the Chef is about to make a move, the Guide whispers, "Hey, remember, we want more spice here!" The final action is a mathematical blend of the Chef's expertise and the Guide's suggestion.

The "Safety Anchor" (Why this is clever)

The most important part of this paper is how it handles mistakes.

Imagine if you just let the "Guide" take over. Since the Guide isn't a master chef, they might suggest something crazy, like putting raw chili peppers in the pastry. The dish would be ruined.

The researchers found that their method acts like a Safety Anchor. Because they use a specific mathematical formula (the "Product" in Product-of-Experts), the final decision cannot do anything that the original Chef thinks is impossible. If the Chef knows that "putting chili in pastry" is a zero-percent possibility, the Guide can't talk them into it. The Chef’s expertise acts as a boundary that keeps the Guide from driving the AI off a cliff.

The "Ceiling" (The Reality Check)

The researchers also discovered something very honest: You can't steer a car that has no engine.

They tested this on much harder tasks. They found that if the original "Chef" (the frozen AI) is already at the absolute limit of what is possible, or if the Chef is simply incompetent (like a chef who doesn't know how to use a stove), no amount of "Guiding" will help. You can steer a skilled driver toward a new destination, but you can't steer a statue.

Summary in Three Bullets:

The Problem: How do you change an AI's goal when the AI is "frozen" and cannot be retrained?
The Solution: Use a "Guide" to nudge the "Expert" at the moment of decision, using a math formula that ensures the Expert always stays in control of the safety boundaries.
The Catch: This only works if the original AI is already competent. If the base AI is bad, the "steering" won't save it.

Technical Summary: When Policies Cannot Be Retrained

Title: When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

1. Problem Statement

In many real-world offline Reinforcement Learning (RL) applications, a policy (the "actor") is trained on a fixed dataset and subsequently approved for deployment. However, deployment objectives often shift after training (e.g., a change in reward weighting or a new goal). In these scenarios, retraining the actor is often impossible due to restricted data access, high computational costs, or strict governance and re-validation requirements.

The authors address the challenge of deployment-time adaptation: how to "steer" a frozen, validated actor toward a new objective while remaining conservative enough to ensure the agent does not deviate into unsafe or unsupported action spaces.

2. Methodology

The paper proposes a modular framework for post-training steering using a Product-of-Experts (PoE) composition.

Core Mechanism: The deployment policy $\pi_{\text{ref}}$ is formed by combining a frozen offline actor $\pi_\theta(a|s)$ with a goal-conditioned deployment prior $\rho_\phi(a|s, g)$ . The refined policy is defined as:
$\pi_{\text{ref}}(a | s, g) \propto \pi_\theta(a | s)^\alpha \rho_\phi(a | s, g)^{1-\alpha}$
where $\alpha \in [0, 1]$ controls the trade-off between following the original actor ( $\alpha \to 1$ ) and following the new goal-conditioned prior ( $\alpha \to 0$ ).
Gaussian Closed-Form: For continuous control tasks where both the actor and prior are diagonal Gaussians, the PoE update has a closed-form solution. The resulting policy is a precision-weighted combination of the means and covariances of the two distributions.
Unification: A key theoretical contribution is proving that for Gaussian policies, the PoE refinement is algebraically identical to a KL-regularized update (where $\beta = \alpha / (1-\alpha)$ ). This unifies two previously treated as distinct mechanisms into a single "actor-anchored" framework.
Goal-Conditioned Prior: The prior is trained separately from the actor using contrastive goal-weighted behavioral cloning, allowing it to provide direction without requiring access to the original training data.

3. Key Contributions

Theoretical Unification: They provide a formal proof that PoE and KL-regularized adaptation are the same deterministic policy for Gaussian distributions, differing only by a global scalar in covariance.
Graceful Degradation: They demonstrate that PoE acts as a "safety layer." When the deployment prior is noisy or random, the precision-weighting naturally anchors the policy to the frozen actor, whereas additive or prior-only methods collapse.
Actor-Competence Ceiling: They identify a fundamental boundary for steering: the effectiveness of adaptation is limited by the competence of the frozen actor. If the actor cannot solve the task, no amount of steering via a prior will enable success.

4. Results

The authors evaluated their method on D4RL continuous-control environments (MuJoCo) and harder regimes (Medium-Expert and AntMaze).

Performance Split: On standard environments, they observed a 4/5/3 HELP/FROZEN/HURT split. The method helps in some goal settings, matches the frozen actor in others, and hurts when the new goal contradicts the actor's inherent capabilities (e.g., speed-dominant goals for an agent already at its velocity ceiling).
Robustness: Under "degraded priors" (noisy or random), PoE/KL-Reg remained stable and close to the frozen actor, while other methods (Additive, Prior-only) failed significantly.
The Ceiling Effect: In "Medium-Expert" regimes, the method was consistently "HURT" across all $\alpha$ values, and in AntMaze, it yielded zero success. This confirms that steering is a tool for re-weighting existing capabilities, not for generating new ones.
Baselines: While critic-guided methods (like CQL/IQL-guided steps) were competitive in some cells, they did not consistently outperform the robustness and stability of the actor-anchored PoE approach.

5. Significance

This work shifts the focus of offline RL adaptation from optimization (retraining) to composition (steering). By establishing that PoE is a mathematically sound, robust, and conservative way to modify behavior at inference time, the paper provides a practical blueprint for deploying RL agents in regulated industries (like robotics or healthcare) where the ability to modify a validated model is restricted. It defines the "regime boundary" for when such steering is useful, preventing practitioners from attempting to steer incompetent models into impossible tasks.