Technical Summary: RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences

Problem Statement

Model-based reinforcement learning (RL) relies on learned world models (dynamics models) to improve sample efficiency by generating synthetic experience. However, in offline settings where agents learn from fixed datasets, these models suffer from model exploitation. When a policy optimizes against an imperfect world model, it may discover and exploit inaccurate transitions (e.g., teleporting through walls) to achieve artificially high returns that do not transfer to the real environment.

Existing solutions typically fall into two categories, both with significant limitations:

1. Collecting more data: Gathering additional expert demonstrations in regions of model failure is often expensive, unsafe, or impossible (e.g., rare failure modes in robotics or autonomous driving).
2. Conservative algorithms: Penalizing visits to uncertain states or constraining policies to the behavioral distribution trades exploration and generalization for robustness, limiting the utility of model-based methods.

The authors posit a new question: Can we correct a world model's dynamics without collecting more demonstrations? They observe that while demonstrating correct behavior requires domain expertise, identifying incorrect dynamics (violations of intuitive physics like solidity or object permanence) is often trivial for humans.

Methodology

The paper introduces two core components: a new learning framework and an algorithm to optimize it.

1. Dynamics Learning from Human Feedback (DLHF)

The authors formalize DLHF, a framework that replaces the reward model in standard Reinforcement Learning from Human Feedback (RLHF) with a dynamics model.

• Core Idea: Instead of asking humans to compare trajectories based on task success (which might reinforce exploitation, e.g., preferring a trajectory that teleports through a wall because it is faster), humans are asked to compare trajectory segments based on physical plausibility.
• Formulation:
  • Let $\hat{T}_\theta$ be the learned dynamics model.
  • The log-likelihood of a trajectory segment $\sigma$ under the model is defined as $\ell_\theta(\sigma) = \sum \log \hat{T}_\theta(s_{t+1} | s_t, a_t)$.
  • In the Bradley-Terry preference framework, the probability that segment $\sigma_0$ is preferred over $\sigma_1$ is modeled as:
    $$P(\sigma_0 \succ \sigma_1) = \text{logistic}(\ell_\theta(\sigma_0) - \ell_\theta(\sigma_1))$$
  • The model parameters $\theta$ are updated by minimizing the cross-entropy loss against human preference labels. This allows the dynamics model to be supervised directly by binary preferences without needing reward labels, expert demonstrations, or environment interaction.

2. RENEW (Repairing ExploitatioN with Elicited World-model preferences)

The authors note that naive DLHF is sample inefficient because uniformly sampling trajectory pairs wastes budget on transitions the model already predicts well. RENEW addresses this by using epistemic uncertainty to direct preference queries.

• Active Querying: RENEW identifies state-action pairs where the model has high epistemic uncertainty (e.g., via ensemble disagreement).
• Targeted Sampling: It generates trajectory segment pairs starting from these high-uncertainty regions.
• Iterative Finetuning: The process iterates:
  1. Compute epistemic uncertainty $u$ under the current model.
  2. Sample segment pairs proportional to $u$.
  3. Elicit human (or oracle) preferences for these pairs.
  4. Finetune the model using the DLHF loss.
  5. Recompute uncertainty and repeat.
• Goal: This concentrates the preference budget on the specific regions where the model is most vulnerable to exploitation, improving sample efficiency and preventing catastrophic forgetting of previously learned correct dynamics.

Key Contributions

1. DLHF Framework: A novel formulation for learning transition dynamics directly from human preferences, shifting the supervision signal from task success (reward) to physical realism (dynamics).
2. RENEW Algorithm: An active learning algorithm that leverages epistemic uncertainty to make preference-based dynamics learning sample-efficient, specifically targeting regions prone to model exploitation.
3. Empirical Validation: Demonstration that binary preferences can learn world models from scratch and repair pretrained models without additional demonstrations or environment interaction.

Experimental Results

The authors evaluated their approach on Jumanji (discrete grid worlds like Maze, Sliding Tile, Sokoban) and Classic Control (MountainCar, Acrobot) environments.

• Feasibility of DLHF: Naive DLHF can learn simple world models from scratch using only preferences, but it requires a massive budget (approx. 1M preference labels) to match the performance of 1K supervised transitions.
• Sample Efficiency of RENEW:
  • Under identical preference budgets, RENEW significantly outperforms naive DLHF. For example, on the Sliding Tile 3x3 environment, RENEW roughly halved the final error compared to uniform sampling.
  • Catastrophic Forgetting: On the Sokoban environment, naive DLHF degraded the performance of a pretrained model, whereas RENEW improved it. This suggests active querying prevents the overwriting of correct dynamics with updates from inaccurate imagined transitions.
• Repairing Exploitation: In finetuning experiments on Maze environments (5x5 to 20x20) with a fixed budget of 1,600 labels, RENEW reduced both prediction error and epistemic uncertainty more effectively than naive finetuning. It achieved higher transition accuracy (e.g., 97.2% vs 93.2% on 10x10 Maze) and tighter confidence intervals.
• Continuous Control: While learning continuous dynamics from preferences is harder due to less distinct preference signals, DLHF still achieved near-parity with supervised baselines on MountainCar and Acrobot. RENEW showed advantages in sample efficiency when gradient steps were limited (e.g., at $K=8$ candidates).

Significance and Claims

The paper claims to provide initial evidence that preferences can supervise world model dynamics directly. The significance lies in:

• Direct Dynamics Supervision: Unlike prior work that uses preferences to learn reward functions or align policies, this work uses them to learn the transition model itself.
• No Expert Data Required: The approach requires no expert demonstrations, no reward labels, and no environment interaction, addressing a major bottleneck in offline model-based RL.
• Practical Repair Mechanism: RENEW offers a pragmatic alternative to collecting more data or restricting exploration, enabling the repair of model exploitation in pretrained models using a limited budget of preference queries.

The authors remain modest about the current scope, noting that experiments used synthetic oracles rather than real human annotators and were limited to small discrete and low-dimensional continuous environments. They identify validating with noisy human labels and scaling to high-dimensional observation spaces as critical next steps.