🤖 machine learning

Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

This paper introduces NEUBAY, a long-horizon model-based offline reinforcement learning algorithm that replaces explicit conservatism with a Bayesian perspective to effectively handle epistemic uncertainty and achieve state-of-the-art performance on diverse datasets.

Original authors: Tianwei Ni, Esther Derman, Vineet Jain, Vincent Taboga, Siamak Ravanbakhsh, Pierre-Luc Bacon

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianwei Ni, Esther Derman, Vineet Jain, Vincent Taboga, Siamak Ravanbakhsh, Pierre-Luc Bacon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to walk across a room. You have a giant video library of other robots walking, but you can't let your robot practice in the real world because it might fall and break something. This is the world of Offline Reinforcement Learning: learning only from a static dataset of past experiences.

For years, the standard advice for teaching robots from these videos has been "Be Super Cautious."

The Old Way: The "Safety First" Approach

Most previous methods act like a nervous parent. They say, "If the robot sees a situation it hasn't seen in the video library, we must assume the worst. Don't try anything new, or you might crash."

The Metaphor: Imagine you are driving a car based only on a map of a small neighborhood. If you see a road not on the map, the "cautious" approach says, "Stop! That road is dangerous. Stay on the known streets."
The Problem: This works well if the map is perfect, but if the map is missing a shortcut to a better destination, the cautious driver will never find it. They get stuck in a loop of "safe but mediocre" performance.

The New Idea: The "Bayesian Detective"

The authors of this paper, NEUBAY, ask a different question: What if we stop being so pessimistic and instead act like a detective who updates their beliefs as they go?

Instead of assuming the worst about the unknown, the Bayesian approach says: "I don't know exactly how the world works, but I have a range of possibilities. Let's try to figure out which possibility is true as I move."

The Metaphor: Imagine you are in a dark room with a flashlight. The "cautious" method refuses to move because it can't see the whole room. The "Bayesian" method says, "I'll take a step, shine the light, see what's there, and update my mental map. If I see a wall, I'll turn; if I see a path, I'll go."
The Result: In situations where the video library is messy or incomplete (low-quality data), this detective approach is much better at finding the best path because it's willing to explore the unknown rather than just sticking to the known.

The Big Challenge: The "Hallucination" Trap

There is a catch. When you stop being cautious, you risk hallucinating.

The Metaphor: If you try to predict what happens 100 steps into the future based on a shaky guess, your prediction will quickly become a fantasy. It's like playing "Telephone" with yourself; by step 50, the message is completely wrong.
The Paper's Insight: The authors discovered that to avoid these hallucinations without being overly cautious, you actually need to think further ahead, not shorter.
- Why? If you only think 5 steps ahead, you have to guess what happens at step 6. If you think 500 steps ahead, the "guess" at the very end gets discounted (it matters less), and the real, known data from the start of the plan carries more weight.
- The Analogy: It's like planning a road trip. If you only plan 10 miles, you might drive into a dead end because you didn't see the sign 20 miles away. If you plan the whole trip, you see the dead end coming and avoid it, even if your map is a little fuzzy.

How NEUBAY Solves It

The authors built a system called NEUBAY that combines three clever tricks to make this "long-term thinking" work without the robot crashing:

The "Uncertainty Speed Bump": Instead of a hard stop, the robot checks its own confidence. If it starts to feel unsure about the terrain (high uncertainty), it stops planning that specific path. It's like a hiker who stops when the trail gets too foggy, rather than refusing to hike at all.
The "Stabilizer" (Layer Normalization): They added a mathematical "shock absorber" to the robot's brain. This prevents the robot's predictions from spiraling out of control when it imagines long sequences of events. It keeps the robot's imagination grounded.
The "Memory Bank": Since the robot is trying to figure out the rules of the world as it goes, it needs to remember everything that happened before. They gave the robot a long-term memory so it can learn from its entire journey, not just the last step.

What They Found

They tested this on 33 different challenging tasks (like robots walking, swinging doors, and navigating mazes).

The Winner: NEUBAY beat the best "cautious" methods on 7 of the datasets and was competitive on almost all of them.
The Sweet Spot: It shines brightest when the training data is messy or incomplete. In those cases, the "cautious" methods get stuck, but NEUBAY's detective work finds the solution.
The Surprise: They found that using very long planning horizons (thinking hundreds of steps ahead) was actually the key to success, which is the opposite of what most other researchers do.

In a Nutshell

The paper argues that in the world of learning from old data, blind pessimism isn't always the safest bet. By trusting a robot to learn from its own history and plan far into the future—while keeping a safety net for when it gets confused—we can build agents that are smarter and more adaptable than those that are just told to "play it safe."

1. Problem Statement

Offline Reinforcement Learning (RL) aims to learn policies from static datasets without further environment interaction. The dominant paradigm in offline RL is explicit conservatism, which penalizes out-of-distribution (OOD) actions or restricts rollout horizons to prevent value overestimation and unsafe extrapolation.

The authors challenge the universality of this principle, arguing that:

Conservatism limits adaptation: By strictly penalizing OOD actions, conservative methods often fail to adapt to test-time conditions, especially in low-quality datasets where the optimal action might be underrepresented.
Bayesian potential is underutilized: A Bayesian perspective (optimizing expected return over a posterior of world models) naturally handles epistemic uncertainty without explicit pessimism but has historically struggled with compounding model errors in long-horizon settings.
The Trade-off: Removing conservatism removes the mechanism for controlling value overestimation, leading to the assumption that short horizons are necessary. The paper questions if long horizons are actually harmful or if they can be leveraged to reduce overestimation.

2. Methodology: NEUBAY

The authors propose NEUBAY (NEUtral BAYesian), a model-based offline RL algorithm grounded in the epistemic POMDP framework. Instead of optimizing for the worst-case model (conservatism), it optimizes for the expected return over a posterior distribution of world models.

Core Design Choices

To make the Bayesian approach practical and effective without explicit conservatism, NEUBAY introduces four key architectural and algorithmic innovations:

A. Adaptive Long-Horizon Rollouts (The Core Insight)

Problem: Standard model-based RL uses short horizons to avoid compounding errors. However, short horizons rely heavily on bootstrapping (using the value function to estimate future rewards), which causes severe value overestimation when conservatism is removed.
Solution: NEUBAY utilizes long-horizon rollouts (hundreds of steps). By extending the horizon, the algorithm shifts the estimation burden from bootstrapped values (high bias) to model-generated rewards (lower bias). The discount factor $\gamma^H$ exponentially down-weights the bootstrapped term at the horizon end, naturally suppressing overestimation.
Mechanism: Rollouts are not fixed-length but adaptively truncated based on epistemic uncertainty. If the model's uncertainty $U_\theta(s, a)$ exceeds a threshold (defined by a quantile $\zeta$ of the dataset's uncertainty distribution), the rollout stops. This ensures rollouts remain long where the model is confident but short where it is not.

B. World Model Architecture for Stability

Large Ensemble Size: To approximate the posterior distribution accurately and handle the variance introduced by long rollouts, NEUBAY uses a large ensemble of world models ( $N=100$ ), significantly larger than the standard $N=5$ or $N=10$ .
Layer Normalization (LayerNorm): The authors apply LayerNorm within the world model's hidden layers. Theoretically, this bounds the magnitude of predicted state deltas, preventing the exponential explosion of compounding errors over long horizons.

C. History-Dependent Agent (Recurrent RL)

Since the Bayesian objective treats the true MDP as a hidden variable, the optimal policy must be history-dependent (a POMDP solution).
NEUBAY employs a recurrent actor-critic architecture using Linear Recurrent Units (LRUs) (a type of State-Space Model). This allows the agent to maintain long-term memory (spanning entire episodes) to infer the underlying model parameters and adapt its behavior at test time without parameter updates.

D. Training Strategy

No Uncertainty Penalty: Unlike conservative methods (e.g., MOPO, COMBO), NEUBAY does not add a penalty term $\lambda U(s,a)$ to the reward.
Data Mixing: It trains on a mixture of real data and imagined rollouts, controlled by a mixing ratio $\kappa$ .

3. Key Contributions

Rejection of Universal Conservatism: The paper demonstrates that explicit conservatism is not universally optimal. In low-quality or skewed datasets, Bayesian adaptation outperforms conservative methods by exploring underrepresented but high-reward actions.
Long Horizons as a Regularizer: The authors provide a theoretical and empirical analysis showing that long-horizon rollouts act as a natural mechanism to control value overestimation in non-conservative settings, counter-intuitively replacing the need for short horizons.
NEUBAY Algorithm: A practical algorithm that combines adaptive uncertainty-based truncation, large ensembles, LayerNorm, and recurrent memory to achieve state-of-the-art performance without explicit pessimism.
Theoretical Analysis: The paper includes formal proofs comparing the "Bayes-optimal" and "Robust-optimal" policies, showing that Bayesian concentrability coefficients are strictly weaker (better) than robust ones under certain data conditions, explaining the performance gap.

4. Experimental Results

The authors evaluated NEUBAY on 33 datasets across D4RL and NeoRL benchmarks (including locomotion, Adroit manipulation, and AntMaze).

State-of-the-Art (SOTA): NEUBAY achieves new SOTA results on 7 datasets, including significant gains on walker-random-v2, walker-medium-v2, pen-cloned-v1, and hammer-cloned-v1.
Performance on Low-Quality Data: NEUBAY significantly outperforms conservative baselines on low-quality datasets (e.g., random policies) and moderate-coverage datasets. It successfully adapts to test-time conditions where conservative methods get stuck on suboptimal behaviors.
Comparison to Conservative Methods: On average, NEUBAY is competitive with leading conservative algorithms (e.g., VIPO, MOPO, COMBO) and outperforms other Bayesian-inspired methods that still rely on conservatism or short horizons.
Ablation Studies:
- Horizon Length: Removing adaptive long horizons (using short fixed horizons) causes performance to collapse due to value overestimation.
- LayerNorm: Removing LayerNorm leads to rapid error explosion in state predictions.
- Uncertainty Truncation: Using a fixed short horizon or a low uncertainty threshold ( $\zeta < 0.99$ ) leads to failure, confirming the need for adaptive, long horizons.
- Memory: While Markov agents work on some locomotion tasks, history-dependent agents are crucial for high-uncertainty tasks like AntMaze.

5. Significance and Impact

Paradigm Shift: The paper challenges the "pessimism is necessary" dogma in offline RL, suggesting a shift toward Bayesian neutrality where agents adapt to uncertainty rather than avoiding it.
Practical Guidelines: It provides a new recipe for model-based RL: use large ensembles, LayerNorm, and adaptive long horizons instead of short horizons and uncertainty penalties.
Test-Time Adaptation: The work highlights the potential of offline RL agents that can adapt their behavior intra-episode based on observed history, a capability that conservative, Markovian policies lack.
Future Direction: The authors suggest that as world modeling improves (e.g., via generative models), the limits of Bayesian offline RL can be pushed further, moving the field toward an "era of experience" where agents refine behavior through test-time interaction.

In summary, NEUBAY demonstrates that by embracing a Bayesian perspective and leveraging long-horizon planning with adaptive truncation, one can achieve robust and adaptive offline RL performance without the restrictive penalties of explicit conservatism.