🤖 machine learning

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation

This paper proposes a lightweight framework called Conservative Query and Adaptive Regularization under Uncertainty Estimation, which leverages a Morse network to estimate action uncertainty for guiding informative preference queries and dynamically adjusting data-level constraints, thereby enhancing offline reinforcement learning performance on the D4RL benchmark.

Original authors: Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun Huang

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to walk, but you can't let it practice in the real world. Maybe it's too expensive, too dangerous, or the robot might break something. Instead, you give the robot a giant video library of someone else walking. This is the world of Offline Reinforcement Learning. The robot has to learn a strategy just by watching these old videos, without ever taking a step on its own. The tricky part is that the robot might try to copy a move it saw in the video, but if it tries to do something slightly different that wasn't in the library, it might get confused or fall over. This is called the "distribution shift" problem: the robot is guessing about things it hasn't seen before.

To make the robot smarter, scientists sometimes let it ask a human expert for help. Imagine the robot says, "Should I step left or right?" and the expert says, "Left is better." This is called an action preference query. It's like getting a hint without actually having to go out and try the move yourself. However, there's a catch: if the robot asks about a move that is too weird or far from what it saw in the video, the expert's advice might actually confuse the robot's internal math, making it learn the wrong thing. The big question is: how do we ask for the right kind of help without getting the robot into trouble?

This paper introduces a clever new system called CQ2L (Conservative Query Q-Learning) to solve exactly that problem. The authors, Li-Rong Zhou, Qin-Wen Luo, and Sheng-Jun Huang from Nanjing University of Aeronautics and Astronautics, realized that previous methods were a bit too reckless. They would ask for advice on any move the robot thought of, even if that move was totally foreign to the training videos. This often led to the robot getting "drunk" on bad advice and stumbling.

The authors propose a two-part safety net to fix this. First, they use a special tool called a Morse neural network. Think of this as a "weirdness detector." Before the robot asks for help, this detector checks: "Is this new move something we've seen before, or is it totally alien?" If the move is too alien (out-of-distribution), the system says, "Nope, don't ask about that one; it's too risky." It only lets the robot ask about moves that are close to the ones in the video library. This is the Conservative Query.

Second, even when the robot gets good advice, the system needs to know how much to listen to it. The authors created an Adaptive Regularization system. Imagine the robot has a volume knob for the expert's advice. If the robot is doing something very familiar (safe), the volume is turned down because the robot can learn just fine from the video data alone. But if the robot is trying something slightly new but still safe, the volume is turned up to let the expert's advice guide it. If the robot tries something dangerous, the system turns the volume all the way up to force it back to safety. This dynamic adjustment prevents the robot from being too stubborn or too easily swayed.

The researchers tested this idea on a famous benchmark called D4RL, which includes tasks like making a virtual cheetah run or an ant navigate a maze. They compared their method against other top algorithms. The results showed that their "smart questioning" approach worked better than standard offline learning. In fact, it performed as well as, or sometimes even better than, methods that were allowed to practice in the real world for a long time, but they did it with far fewer "questions" (only 90,000 preference queries compared to 250,000 steps of real-world practice for other methods).

The paper explicitly argues against the old way of just asking for advice on any random move, showing that this leads to unstable learning and bad results. They suggest that by being careful about what you ask and how you listen, you can get the best of both worlds: the safety of learning from a static library and the boost of expert guidance, without the risk of the robot crashing. Their experiments on the D4RL benchmark suggest that this approach is a solid, reliable way to make offline learning smarter and safer.

Technical Summary: Conservative Query and Adaptive Regularization for Offline RL under Uncertainty Estimation

1. Problem Statement

Offline Reinforcement Learning (RL) aims to learn effective policies from static datasets without further environment interaction. However, performance is fundamentally limited by the dataset's coverage. A primary challenge is distribution shift, where the learned policy selects out-of-distribution (OOD) actions, leading to inaccurate value estimates and error propagation.

While action preference query mechanisms offer a way to improve policies using expert feedback (comparing action pairs) without online interaction, existing methods (e.g., OAP [41]) face two critical limitations:

Query Shift: Existing strategies select queried actions based solely on Euclidean distance to the dataset. This can result in selecting actions far outside the dataset distribution (OOD). In value regularization methods, this induces inaccurate value estimation and unstable policy updates, as the regularization term may erroneously elevate the Q-values of OOD actions.
Inefficient Exploitation: Current approaches treat queried preferences as static constraints throughout training. They fail to dynamically adjust the reliance on these preferences as the policy improves, often applying unnecessary constraints to reliable actions or failing to leverage the optimistic potential of Bellman updates for in-distribution actions.

2. Methodology

The authors propose a lightweight framework integrating Conservative Query and Adaptive Regularization under Uncertainty Estimation. The core components are:

A. Uncertainty Estimation via Morse Neural Networks
The method employs a Morse Neural Network [10, 4, 45, 33] as a pre-trained uncertainty estimator.

Training: The network is trained on the offline dataset $D$ to output a "Morse score" $M_\phi(s, a) \in [0, 1]$ . A score of 1 indicates the action lies on the mode submanifold of the data distribution, while lower scores indicate deviation (higher uncertainty/OOD degree).
Mechanism: The network minimizes the KL divergence between the data distribution and the network's output, effectively quantifying the certainty that a state-action pair originates from the offline dataset.

B. Uncertainty-Driven Conservative Query
To prevent query shift, the framework filters candidate actions before querying an oracle (e.g., a stronger policy's Q-function).

Selection Strategy: Instead of simple distance-based selection, the method ranks candidate actions by their divergence from the dataset but filters them using a dataset-aware threshold $\delta = \mu - n\sigma$ (where $\mu$ and $\sigma$ are the mean and standard deviation of Morse scores on the dataset).
Constraint: Only actions with $M_\phi(s, \hat{a}) > \delta$ are retained for preference querying. This ensures queried actions remain close to the dataset distribution, preserving the stability of Bellman updates and preventing the overestimation of OOD values.

C. Uncertainty-Aware Adaptive Regularization
To effectively utilize preference information, the method replaces fixed regularization coefficients with data-dependent ones.

Dynamic Modulation: The regularization strength $\alpha(s, a)$ $α (s, a)$ is dynamically adjusted based on the Morse score of the current policy's action.
- High In-Distribution Confidence: If an action is close to the dataset (high Morse score), the regularization strength is reduced (approaching 0), allowing the policy to benefit fully from optimistic Bellman updates.
- Low In-Distribution Confidence: If an action is OOD (low Morse score), stronger regularization is applied to guide the policy back toward the dataset distribution.
Nonlinear Scaling: To ensure discriminative power (as linear scaling often results in weak constraints), a nonlinear scaling function is introduced to modulate the regularization strength based on the distribution of Morse scores.

D. Integration with CQL
The framework is integrated into Conservative Q-Learning (CQL), a representative value regularization algorithm. The resulting algorithm, Conservative Query Q-Learning (CQ2L), modifies the CQL objective to include the adaptive regularization term and updates preferred actions $\tilde{a}$ based on the conservative query mechanism.

3. Key Contributions

Morse-Based Uncertainty Estimation: The introduction of Morse Neural Networks to quantify the OOD degree of candidate actions relative to the offline dataset, serving as the foundation for both querying and regularization.
Uncertainty-Driven Conservative Query: A mechanism that filters query candidates to ensure they remain within the dataset distribution, mitigating query shift and stabilizing value estimation in value regularization frameworks.
Uncertainty-Aware Adaptive Regularization: A dynamic constraint mechanism that balances pessimistic regularization with optimistic Bellman updates based on the reliability of the current policy's actions, avoiding ineffective constraints on in-distribution actions.
Extension to Value Regularization: Successfully extending the action preference query paradigm from policy-constraint methods to value regularization methods (specifically CQL), a domain where previous query methods struggled.

4. Experimental Results

The method was evaluated on the D4RL benchmark, covering MuJoCo locomotion and AntMaze navigation tasks.

Offline Performance: CQ2L achieved superior or competitive performance compared to state-of-the-art offline RL baselines (including AWAC, IQL, CQL, TD3+BC) and the previous action preference query method (OAP). Notably, it achieved an average normalized score of 91.4 on MuJoCo tasks and 75.8 on AntMaze tasks, outperforming OAP (89.9 and 69.3 respectively in the offline scheme).
Offline-to-Online (O2O) Efficiency: In the O2O setting, CQ2L used only 90,000 preference queries (significantly fewer than the 250,000 online interaction steps used by fine-tuning baselines and fewer than OAP's 100,000 queries). Despite this, it outperformed most baselines that underwent extensive online fine-tuning and achieved results comparable to or better than fine-tuned CQL.
Ablation Studies: Experiments confirmed that both the Conservative Query (CQ) and Adaptive Regularization (AR) components are essential.
- In dense-reward, low-dimensional tasks (MuJoCo), Adaptive Regularization was crucial for leveraging Bellman updates.
- In sparse-reward, high-dimensional tasks (AntMaze), Conservative Query was vital to prevent overestimation of OOD actions and facilitate "trajectory stitching" within the data manifold.

5. Significance and Claims

The paper claims that the proposed framework addresses the fundamental instability of integrating action preference queries into value regularization methods. By leveraging uncertainty estimation, the method:

Mitigates Query Shift: Ensures that preference queries do not destabilize the value function by restricting queries to the in-distribution region.
Optimizes Information Utilization: Dynamically balances the trade-off between conservative constraints and the optimistic nature of Bellman updates, ensuring that preference information is used effectively without hindering learning on reliable actions.
Broadens Applicability: Demonstrates that action preference queries can be successfully adapted to value regularization algorithms, offering a new direction for improving offline RL performance without the costs and risks of online interaction.

The authors conclude that their method provides a robust, lightweight solution that achieves superior performance across diverse tasks, validating the effectiveness of combining uncertainty-driven querying with adaptive regularization.