🤖 machine learning

Decoupling Policy Extraction for Offline Reinforcement Learning

This paper proposes a "decoupled policy extraction" paradigm for offline reinforcement learning that separates behavior modeling from policy improvement by training an actor to generate behavior-supported action candidates and using a separate critic to rerank them at inference time, thereby overcoming the limitations of coupled actor-critic training and outperforming existing methods.

Original authors: Xuyao Lin, Yixiang Shan, Jinru Duan, Tao Yang, Xinyu Zhao, Runyu Lei, Yiming Zhao, Jiaxin Fan, Zongbao Feng, Peng Jia

Published 2026-08-24

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuyao Lin, Yixiang Shan, Jinru Duan, Tao Yang, Xinyu Zhao, Runyu Lei, Yiming Zhao, Jiaxin Fan, Zongbao Feng, Peng Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a student trying to learn how to drive a car, but instead of sitting behind the wheel and feeling the road, they are forced to study only a single, static video of a perfect driver. They cannot make mistakes, they cannot try new turns, and they cannot receive feedback from the real world. This is the challenge of offline reinforcement learning, a field where artificial intelligence learns to make decisions using only a fixed collection of past data. In this setting, the computer must figure out how to act better than the examples it has seen, without ever being able to test those new ideas in reality. The traditional way to teach these systems involves two parts working in a tight loop: one part learns what actions are good, and the other part learns how to perform them. The first part constantly guides the second, telling it to try actions that look valuable. However, because the data is frozen in time, this guidance can become dangerous. If the "what is good" part makes a mistake and overestimates a risky move, the "how to perform" part will try to do it, reinforcing the error and drifting further away from safe, proven behavior.

A team of researchers at Simplexity Robotics and Rensselaer Polytechnic Institute has proposed a different way to solve this problem, suggesting that these two parts should stop talking to each other during the learning phase. In their new approach, they completely separate the task of learning from the data from the task of choosing the best action. First, they train a model solely to mimic the behavior found in the dataset, acting like a perfect copyist that never tries to improve or guess at new strategies. This model generates a list of several possible actions that are known to be safe and supported by the data. Then, a separate system, trained independently to judge value, looks at this list and picks the single best option to execute. By breaking the feedback loop that usually connects the two, the researchers found they could avoid the trap of amplifying errors. Their experiments across thirty different complex tasks, ranging from navigating virtual mazes to manipulating robotic arms, showed that this separated method consistently outperformed the traditional, tightly coupled approaches. In some cases, the success rate jumped from less than half to nearly seventy percent, proving that sometimes, the best way to improve is to stop trying to improve the learner while it is still studying.

The core of the problem the researchers identified lies in how standard artificial intelligence systems learn from fixed data. In a typical setup, the system has a "critic" that estimates the value of actions and an "actor" that learns to perform them. The critic tells the actor which moves are good, and the actor tries to do them. In a real-world scenario where the robot can keep interacting with the environment, this works well because if the critic makes a mistake, the actor's new attempts generate fresh data that corrects the error. But in offline learning, the dataset is locked. If the critic mistakenly believes a dangerous action is valuable, the actor will try to do it, and because no new data arrives to correct the critic, the mistake gets worse. The actor drifts into regions of action space that the data never covered, a phenomenon the researchers call an out-of-distribution amplification loop. To prevent this, existing methods often try to force the actor to stay close to the original data, but this creates a difficult trade-off: if you constrain the actor too much, it cannot find the best possible moves even within the safe data; if you let it go too far, it falls into the error loop.

To solve this, the researchers decoupled the process. They trained the actor to do only one thing: model the distribution of actions found in the dataset. They did not let the critic influence the actor's training at all. Once this "proposer" model was trained, it was frozen, meaning it would never change again. At the moment the robot needed to make a decision, the proposer would generate a small set of candidate actions, all of which were grounded in the safe, observed data. A separate critic, which had been trained independently to judge value, would then look at this specific list and pick the highest-scoring option. This shifted the work of improvement from the training phase to the moment of decision. Instead of trying to tweak the actor's brain to be better, the system simply generated a few safe options and let a judge pick the winner. This approach meant the critic no longer had to worry about the actor drifting into dangerous territory, because the actor was never allowed to drift in the first place. The critic only had to rank the safe options provided by the frozen proposer.

The results of this experiment were striking. The researchers tested their method on thirty different tasks involving goal-oriented behaviors, such as getting a robot ant to navigate a large maze or a humanoid robot to move through a complex environment. They compared their decoupled method against standard techniques that kept the actor and critic linked. In a navigation task called AntMaze-Large, the traditional method using a specific type of value learner achieved a success rate of about thirty-one percent. When the researchers applied their decoupled approach with the same value learner, the success rate rose to forty-nine percent. In a manipulation task involving a cube, the improvement was even more dramatic, with the decoupled method reaching a success rate of seventy-nine percent compared to twenty-one percent for the traditional approach. Perhaps most surprisingly, the researchers found that even a very simple, basic value-learning system, which usually struggles in offline settings, became highly effective when paired with this decoupled selection process. In a puzzle-solving task, a simple value learner combined with their method achieved a one hundred percent success rate, vastly outperforming more complex, standard systems.

The study also revealed why this separation works so well by examining what happens when the system is allowed to look at more options. The researchers found that having the proposer generate a few candidates, rather than just one, allowed the system to explore the safe region of the data more thoroughly. However, there was a limit. If the system generated too many candidates, the chance of including a risky, out-of-distribution action increased, and the value-judging system might mistakenly pick it. The optimal balance was found by tuning the number of candidates, a single setting that could be adjusted without retraining the entire system. This flexibility is a significant advantage over traditional methods, which often require expensive retraining to fix the balance between safety and performance. The researchers noted that this approach is particularly promising for large-scale robotic systems, where retraining a massive model is computationally expensive. By keeping the main model frozen and only using a lightweight value system to make the final choice, the method offers a computationally efficient path to better performance.

The researchers acknowledge that this method has a boundary: the system can only choose from actions that the frozen proposer can generate. If the best possible action was never seen in the original data, the system cannot invent it. However, within the bounds of what is possible, the decoupled approach consistently found better solutions than the traditional, coupled methods. The work suggests that the long-held belief that learning and improvement must happen simultaneously in a single loop may not be necessary, or even desirable, when learning from static data. By separating the generation of safe options from the selection of the best one, the researchers have provided a clearer, more stable path for artificial intelligence to learn from the past without being trapped by its own mistakes. The findings indicate that for offline learning, the most effective strategy may be to stop trying to teach the actor to be clever during training, and instead let it simply remember the past, leaving the job of improvement to a separate, critical eye at the moment of action.

Technical Summary: Decoupling Policy Extraction for Offline Reinforcement Learning

1. Problem Statement

Offline Reinforcement Learning (RL) typically employs a coupled actor-critic architecture where the critic guides the actor toward higher-value actions. While effective in online RL—where the improved actor generates new data to correct the critic—this paradigm faces fundamental limitations in the offline setting where the dataset is fixed. The paper identifies two primary issues arising from this coupled training:

OOD Amplification Loop: In offline RL, the actor cannot generate new data to validate the critic. If the critic overestimates the value of out-of-distribution (OOD) actions, the actor shifts its policy toward these actions. This shift further exposes the actor to regions where the critic is less reliable, creating a self-reinforcing feedback loop that amplifies estimation errors and drives the policy away from the data distribution.
Support–Value Trade-off: To mitigate OOD drift, existing methods often employ conservative value estimation or behavior-cloning regularization. However, this creates a difficult trade-off: weak constraints fail to suppress OOD actions, while strong constraints prevent the policy from selecting high-value actions even within the supported data region. Consequently, the optimal level of conservatism for distributional support often differs from the level that maximizes task performance.

2. Methodology: Decoupled Policy Extraction

The authors propose a Decoupled Policy Extraction paradigm that separates the modeling of the behavior distribution from the process of policy improvement. This approach consists of three distinct components:

Frozen Behavior-Cloned Actor (Proposer): The actor is trained solely via behavior cloning (maximum log-likelihood) on the offline dataset. It is frozen after training and receives no critic-derived supervision (no rewards, advantages, or gradients from the critic). Its sole function is to model the dataset's action distribution and propose $N$ candidate actions at inference time.
Separately Learned Critic: A critic is trained independently using standard offline RL objectives (e.g., Q-Learning, IQL, or TRL). Crucially, the critic does not update the actor. Its role is restricted to evaluating and ranking the candidate actions proposed by the frozen actor.
Inference-Time Reranking: Policy improvement is realized explicitly at inference time rather than through training-time actor updates. The system samples $N$ candidate actions from the frozen actor and selects the one with the highest estimated value according to the critic.

This design shifts the burden of OOD control from complex training-time regularization to a tunable inference-time parameter: the candidate budget $N$ . By restricting the critic's selection to behavior-supported candidates, the need for aggressive conservative regularization is reduced.

3. Key Contributions

Paradigm Revisit: The paper challenges the prevailing assumption that actor-side policy improvement is necessary in offline RL. It demonstrates that decoupling policy improvement from actor training effectively avoids the OOD amplification loop and the support–value trade-off.
Empirical Validation: The authors instantiate this paradigm with multiple critic backbones (Q-Learning, IQL, TRL) across 30 offline goal-conditioned tasks. They show that the decoupled approach consistently outperforms both behavior cloning and jointly learned offline RL baselines.
Critic Flexibility: The framework relaxes requirements on critic design. Because the actor restricts candidates to the data-supported region, even a standard Q-learning critic (without explicit OOD regularization) can achieve strong performance, as it only needs to provide accurate relative rankings among valid candidates.
Scalability for Large Policies: The approach offers a computationally efficient path for large-scale policies. Large, expressive models can be trained once via behavior cloning, while task-specific optimization is delegated to a lightweight critic at inference time, avoiding the cost of repeatedly updating massive actor parameters.

4. Experimental Results

Extensive experiments on the OGBench suite (including AntMaze, HumanoidMaze, AntSoccer, Cube-Double, Scene-Play, and Puzzle-4x4) yield the following findings:

Performance Gains: The decoupled policy extraction variants consistently outperform their coupled counterparts. For instance, using an IQL critic, the decoupled variant increased the average success rate on 30 tasks from 47.89% to 68.49%.
Standard Critics: A decoupled paradigm using a standard Q-learning critic outperformed the widely adopted IQL method in most environments, with a particularly significant gain (212.8% improvement) on the Puzzle-4x4 task.
Ablation Studies:
- Reranking Efficacy: Adding critic-based reranking to a coupled IQL policy improved performance, but the decoupled variant (frozen BC actor + reranking) achieved the highest overall performance, indicating that the coupled training paradigm inherently limits the actor's ability to serve as a good proposer.
- Candidate Budget ( $N$ ): Performance improves with $N$ up to a point (e.g., $N=16$ ) as the policy explores the behavior-supported region more thoroughly. However, excessively large $N$ can degrade performance due to the increased probability of sampling OOD actions that the critic erroneously ranks highly.
- Inference Cost: While GFLOPs increase with $N$ , the median latency remains low (<0.53 ms) due to parallel batched evaluations, and GPU memory usage remains nearly constant.

5. Significance and Claims

The paper claims that the decoupled policy extraction paradigm addresses the core bottlenecks of offline RL by fundamentally restructuring how policy improvement is achieved. By moving the optimization step from the training loop (actor updates) to the inference loop (action selection), the method:

Breaks the Feedback Loop: Prevents critic errors from recursively distorting the actor's distribution.
Resolves the Trade-off: Allows the actor to faithfully model the data distribution while the critic performs value maximization within that safe region, eliminating the need to balance these objectives in a single loss function.
Enables Efficient Scaling: Provides a mechanism to leverage large, pre-trained behavior models for offline RL without the computational overhead of continuous actor fine-tuning, delegating task-specific adaptation to lightweight critics.

The authors acknowledge a limitation: the performance ceiling is bound by the coverage of the behavior-cloned proposer. If high-value behaviors are missing or extremely rare in the dataset, the critic cannot select them. However, the paper posits that for many practical applications, this decoupled approach offers a more robust and efficient alternative to traditional coupled actor-critic methods.