🤖 machine learning

On Training in Imagination

This paper analyzes the impact of learned dynamics and reward model errors on model-based reinforcement learning, deriving optimal sample allocation strategies and characterizing the tradeoff between using expensive, low-noise rewards versus cheaper, noisy rewards for policy optimization.

Original authors: Nadav Timor, Ravid Shwartz-Ziv, Micah Goldblum, Yann LeCun, David Harel

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nadav Timor, Ravid Shwartz-Ziv, Micah Goldblum, Yann LeCun, David Harel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to play a complex video game, like a racing simulator or a strategy game. You have two main ways to teach it:

Real Practice: Let the robot play the game for real, crash into walls, collect points, and learn from the actual environment.
Imagination: Build a "dream world" inside the robot's computer. In this dream, the robot simulates playing the game, predicts what will happen next, and scores its own performance, all without ever touching the real game.

This paper is about Imagination Training. It asks: "How do we make this dream world training as effective as possible, especially when our 'dream' isn't perfect?"

Here is the breakdown of their findings using simple analogies:

1. The Two Parts of the Dream

To run a simulation, you need two things:

The Physics Engine (Dynamics Model): This predicts what happens when you take an action. If you press "left," where does the car go?
The Scorekeeper (Reward Model): This tells you how good that move was. Did you get points? Did you crash?

The authors discovered that errors in these two parts hurt the robot's learning differently.

The Analogy: Imagine you are learning to cook by reading a recipe book (the Physics) and tasting the food (the Reward).
- If your recipe book is slightly wrong (e.g., it says "bake for 10 minutes" but it's actually 12), you might burn the cake eventually.
- If your taste buds are slightly off (e.g., you think it's salty when it's sweet), you might keep adding the wrong ingredients.
- The paper proves that if your "taste buds" (Reward Model) are bad, it messes up your learning just as much as a bad recipe book, but in a specific, calculable way.

2. The "Smoothness" Rule

The paper suggests that for the robot to learn well in its imagination, the maps it uses (the Physics and the Scorekeeper) need to be "smooth."

The Analogy: Think of a bumpy road versus a smooth highway.
- If the road is bumpy (mathematically, if the "Lipschitz constant" is high), a tiny change in your steering wheel (input) causes a huge, unpredictable jump in where the car goes (output). This makes the dream world chaotic and hard to learn from.
- If the road is smooth, a tiny turn in the wheel leads to a tiny, predictable curve.
- The Finding: The authors show that if you design your AI to learn these "smooth" maps, the errors in the dream world stay small, and the robot learns much faster. They even connect this to a technique called "temporal straightening," which is like forcing the robot's dream path to look like a straight line rather than a jagged scribble, making it easier to predict the future.

3. The Budget Problem: How Much to Spend on What?

Imagine you have a fixed amount of money (a budget) to train your robot. You can spend it on:

Dynamics Data: Recording how the car moves (cheaper, easier to get).
Reward Data: Getting a human expert to say "Good job" or "Bad job" (expensive, harder to get).

The Big Question: Should you buy a huge pile of cheap movement data and a few expensive expert opinions? Or a small pile of movement data and many expert opinions?

The Finding: The paper gives a mathematical formula for the perfect split.
- It turns out that Reward data is usually much easier to learn from than movement data. In their experiments, the "Reward Model" learned nearly 9 times faster than the "Physics Model" for every bit of data added.
- Because rewards learn so fast, the paper suggests you should generally spend more of your budget on getting more Reward data (even if it's a bit noisier) rather than obsessing over perfect Physics data. It's better to have a slightly imperfect map but a very clear understanding of what "winning" looks like.

4. Dealing with Noisy Scores

Sometimes, the "Scorekeeper" isn't perfect. Maybe the human expert is tired, or the sensor is glitchy, and the score is a little bit random (noisy).

The Good News: If the noise is random (sometimes +1, sometimes -1, but averages to zero), the robot can still learn perfectly fine. It just needs to try a few more times to smooth out the randomness.
The Bad News: If the Scorekeeper is biased (always thinks the robot is doing better than it actually is), no amount of practice will fix it. The robot will learn the wrong lesson.
The Trade-off: The paper asks: "Should I pay for 100 cheap, noisy scores, or 10 expensive, perfect scores?"
- Their math shows it depends on how much the cost goes down when you accept more noise.
- Scenario A: If getting a "perfect" score costs 100x more than a "noisy" one, but the noise only goes down by 2x, you should buy the cheap, noisy scores and just run more simulations.
- Scenario B: If paying a little more makes the noise disappear completely, you should pay for the expensive, perfect scores.

Summary

This paper is a guide for building better "dream worlds" for AI. It tells us:

Smoothness is key: Make sure your AI's internal maps are predictable and not chaotic.
Reward is king: Since learning what "good" looks like is faster than learning how the world moves, spend more of your budget on reward data.
Noise is okay, bias is not: Random mistakes in scoring can be fixed by doing more practice; systematic lies cannot.

The authors tested these ideas on synthetic computer simulations and found that their mathematical formulas accurately predicted how to split the budget and how much error to expect.

Technical Summary: On Training in Imagination

Problem Statement

The paper addresses the theoretical foundations of "training in imagination," a paradigm where reinforcement learning policies are optimized using rollouts generated by a learned dynamics model and scored by a learned reward model, without querying the true environment during policy updates. While recent state-of-the-art methods (e.g., Dreamer 3, Dreamer 4) demonstrate empirical success, existing theoretical bounds (such as the simulation lemma) fail to independently quantify the contributions of dynamics-model error versus reward-model error to the total return gap. Furthermore, there is no theoretical guidance on how to optimally allocate a fixed sample budget between collecting dynamics transitions (state-action-next state) and reward annotations (state-action-reward), particularly when reward annotations are more expensive or noisy.

The authors identify four open questions:

Error Attribution: How much of the return gap stems from dynamics error versus reward error?
Representation Properties: What properties of learned representations tighten the return-error bound?
Budget Allocation: How should a fixed sample budget be split between dynamics and reward data?
Reward Fidelity: How does policy optimization (specifically REINFORCE) tolerate noisy or biased rewards, and when is it better to buy many cheap noisy annotations versus fewer accurate ones?

Methodology

The authors approach these questions through a combination of theoretical analysis under Lipschitz continuity assumptions and power-law scaling hypotheses, followed by empirical validation on synthetic and Linear-Quadratic-Gaussian (LQG) benchmarks.

Theoretical Framework

Error Decomposition (Lemma 1): The authors extend the analysis of Asadi et al. [2018b] by replacing the ground-truth reward with a learned reward model. They derive a bound on the return gap $|J(\pi, M) - J(\pi, \hat{M})|$ that decomposes the error into separable terms for dynamics error ( $\epsilon_{dyn}$ ) and reward error ( $\epsilon_{rew}$ ). The bound depends on the Lipschitz constants of the learned dynamics ( $L_f$ ), reward ( $L_r$ ), and policy ( $L_\pi$ ), and the discount factor $\gamma$ .
Representation Analysis: The authors analyze how the Lipschitz constants of the learned models affect the error bound. They connect this to the "temporal straightening" objective (Wang et al. [2026]), showing that minimizing the Lipschitz constant of the latent velocity map reduces the curvature loss, thereby tightening the error bound.
Optimal Budget Allocation (Theorem 1): Assuming power-law scaling for model errors ( $\epsilon \propto N^{-\alpha}$ ), the authors derive a closed-form expression for the optimal ratio of dynamics samples to reward samples ( $N^*_{dyn}/N^*_{rew}$ ) that minimizes the return error bound under a fixed budget. This ratio depends on the power-law exponents, the per-sample costs, and the Lipschitz coefficients.
Noisy Reward Analysis (Theorem 2 & Corollary 2): The authors analyze the REINFORCE estimator under additive zero-mean reward noise. They prove that such noise leaves the gradient estimator unbiased but increases its variance. They derive an optimization problem to determine the optimal "fidelity" (cost per annotation) versus quantity trade-off, showing that the optimal strategy depends on the functional relationship between annotation cost and noise variance.
Bias Analysis (Proposition 2): They demonstrate that systematic reward bias introduces a gradient bias that cannot be removed by averaging over more trajectories, distinguishing it from zero-mean noise.

Empirical Validation

Error Bound Calibration: The authors tested Lemma 1 on 525 configurations (synthetic and LQG). The bound held universally, though it was found to be loose in practice (typically overshooting by factors of 29x to 286x), suggesting that global Lipschitz constants are conservative estimates of realized sensitivities.
Scaling Laws: In a synthetic continuous-control environment, they fitted power-law scaling for dynamics and reward errors. They found that reward error decays significantly faster ( $\beta \approx 0.96$ ) than dynamics error ( $\alpha \approx 0.11$ ) as sample size increases.
Allocation Prediction: They evaluated the predictive power of Theorem 1. While the proportionality between the optimal sample ratio and the error ratio held, the multiplier derived from global Lipschitz constants was loose (overshooting by orders of magnitude). However, replacing global constants with realized value-function sensitivities recovered the predicted ratios accurately.
Fidelity Trade-offs: Theoretical regimes for noise-cost trade-offs (power-law, bounded, irreducible floor) were illustrated, showing that optimal strategies range from buying maximum fidelity to buying maximum quantity depending on the noise-cost curve.

Key Contributions

Error Attribution: A formal decomposition of the return gap into independently controllable dynamics and reward error terms (Lemma 1), extending previous simulation lemmas to include learned reward models.
Representation Desiderata: Identification of lower Lipschitz constants for learned dynamics, reward, and policy as a key desideratum for tightening return-error bounds (Corollary 1), and a theoretical link between this and temporal straightening objectives (Proposition 1).
Optimal Sample Allocation: A closed-form solution for the optimal ratio of dynamics to reward samples under power-law error scaling and fixed budgets (Theorem 1), providing a theoretical basis for data collection strategies.
Reward Fidelity Characterization: A characterization of how REINFORCE tolerates zero-mean noise (unbiased with bounded variance) versus systematic bias (irreducible gradient bias), and a reduction of the fidelity-quantity trade-off to a one-dimensional optimization problem (Corollary 2).

Results

Theoretical Bounds: The derived bound (Equation 1) holds for all tested configurations, confirming the validity of the decomposition. However, the bound is loose when instantiated with global Lipschitz constants, indicating that realized sensitivities are more informative for practical allocation.
Scaling Dynamics: Empirical results show that reward models learn significantly faster than dynamics models (error decays ~9x faster per decade of data), suggesting that in many settings, the budget should shift toward acquiring more reward samples or that reward models saturate quickly.
Allocation Strategy: The optimal sample ratio is proportional to the ratio of error levels and the ratio of costs, scaled by a factor involving Lipschitz constants and the discount factor.
Noise vs. Bias: Zero-mean noise can be mitigated by increasing the number of rollouts (variance reduction), whereas systematic bias persists regardless of sample size, fundamentally altering the optimization landscape.

Significance and Claims

The paper claims to provide the first theoretical characterization of the trade-offs involved in training in imagination when reward models are learned separately from dynamics models. By decoupling these error sources, the authors provide a framework for:

Quantifying Error Sources: Understanding whether performance gaps are due to poor world modeling or poor reward modeling.
Guiding Data Collection: Offering a principled method to allocate limited resources between expensive reward annotations and cheaper dynamics transitions.
Informing Representation Learning: Suggesting that regularization techniques (like spectral normalization or temporal straightening) that lower Lipschitz constants directly improve the theoretical guarantees of model-based RL.

The authors remain modest about the practical tightness of their bounds, noting that while the theoretical decomposition is sound, the global Lipschitz constants used in the bounds are often loose upper bounds on realized sensitivities. Consequently, while the proportionality of the optimal allocation rule is robust, the exact multiplier requires more precise estimation of value-function sensitivities than global constants provide. The work is presented as foundational analysis to guide the design of future model-based RL systems rather than a direct recipe for immediate implementation without further calibration.