Technical Summary: Imperfect World Models are Exploitable

Problem Statement

In reinforcement learning (RL), efficient real-world planning often necessitates approximating complex dynamics through imperfect world models. While well-chosen approximations can render intractable problems solvable, ill-chosen ones can compromise safety. This failure mode, known as model exploitation, occurs when an agent trained to maximize expected reward under an imperfect transition model discovers a policy that performs well in simulation but poorly in reality.

The central question addressed is whether model exploitation can be avoided. While prior work on reward hacking (Skalse et al., 2022) established conditions under which reward specification errors can be avoided, it is unclear if analogous results hold for transition model errors. Existing literature often relies on cardinal measures of model quality (e.g., prediction error bounds), but the authors argue that for safe planning, ordinal safety is required: a model is safe only if it preserves the relative ordering of policies induced by the true environment. Specifically, if the true model prefers policy $\pi$ over $\pi'$, a safe approximation must not prefer $\pi'$ over $\pi$.

Methodology and Theoretical Framework

Definitions

The paper formalizes Model Exploitation as a binary relation between two transition models, $T$ and $T'$. A pair is exploitable relative to a policy set $\Pi$ if there exist policies $\pi, \pi' \in \Pi$ such that:
$$J_T(\pi) > J_T(\pi') \quad \text{and} \quad J_{T'}(\pi') > J_{T'}(\pi)$$
where $J_T$ denotes the value function under transition model $T$. This is defined as a value inversion. The authors distinguish this from Reward Hacking, where the inversion occurs between two reward functions $R$ and $R'$ under a fixed transition model.

Key Theoretical Obstacles

The authors demonstrate that techniques used to characterize reward hacking do not directly transfer to model exploitation due to fundamental geometric differences:

1. Linearity vs. Non-linearity: The value function $J(\pi) = \langle R, F^\pi_T \rangle$ is linear in the reward function $R$ but non-linear in the transition model $T$ (specifically, in the probability mass function). Consequently, the linear-algebraic machinery used in reward hacking analysis is insufficient for transition models.
2. Domain Constraints: Reward functions reside in a Euclidean space closed under negation and scaling, whereas transition models reside on a compact product of simplices, lacking a vector space structure.

Unified Theory via Value Function Rationality

To overcome these obstacles, the authors develop a unified theory based on the rationality of the value function in policy space. For finite MDPs, $J(\pi)$ is a rational function of the policy parameters $\pi$ and is real-analytic on the interior of the policy space ($\Pi^+$).

Using this property, the authors derive two geometric lemmas regarding the gradients of value functions ($\nabla J_1$ and $\nabla J_2$):

• Local Inversion (Lemma 1): If there exists a direction in policy space where one value function increases while the other decreases, a value inversion (exploitation) exists.
• Global Equivalence (Lemma 2): If the gradients of two non-trivial value functions are positively proportional everywhere on an open subset, the functions are equivalent (no inversion exists).

Key Contributions and Results

1. Inevitability of Exploitation on Large Policy Sets

The paper establishes that for any policy set containing an open subset (e.g., the set of all stationary policies, $\epsilon$-suboptimal policies, or $\delta$-deterministic policies), model exploitation is inevitable for any pair of non-trivial, non-equivalent transition models.

• Theorem 1: If a policy set contains an open subset, any non-trivial, non-equivalent pair of value functions admits a value inversion.
• Corollary 1: Consequently, every non-trivial, non-equivalent pair of transition models is exploitable on such policy sets.
  This result contrasts with the "possibility" of avoiding reward hacking in finite sets, showing that the non-linear dependence of value on transition models makes unexploitability impossible in rich policy classes.

2. Distinction from Reward Hacking in Finite Sets

While reward hacking can be avoided on finite policy sets (under specific regularity conditions), the authors show that unexploitability cannot be guaranteed for finite policy sets in the same manner.

• Counterexample 2: Even when regularity conditions for reward hacking are met (non-collinear visit counts), there exist tasks and finite policy sets where every pair of non-equivalent transition models is exploitable. The authors conjecture that no simple necessary and sufficient conditions exist for unexploitability in finite sets due to the complex geometry induced by the non-linear transition model.

3. Reduction and Non-Reduction

• Proposition 3: Every instance of model exploitation can be reduced to an instance of reward hacking. If $(T, T')$ is exploitable, there exists a reward function $R'$ such that $(R, R')$ is hackable.
• Counterexample 1: The converse does not hold. Reward hacking does not necessarily imply model exploitation, as perturbations in transition models may not alter the relative ordering of policies under fixed rewards.

4. Relaxed Notion: $\epsilon$-Exploitability and Safe Horizons

Recognizing that strict unexploitability is often unattainable, the authors introduce $\epsilon$-exploitability, which allows for small value inversions. They derive a Safe Horizon ($H$) within which $\epsilon$-unexploitability is guaranteed.

• Theorem 2: For a given tolerance $\epsilon$ and total variation distance $\delta$ between models, there exists a tight bound on the effective horizon $1/(1-\gamma)$. If the horizon is shorter than $H(\epsilon, \delta)$, the models are $\epsilon$-unexploitable.
• Corollary 4 (Square-root heuristic): A looser but intuitive bound suggests that the safe horizon scales roughly as $\sqrt{\epsilon/\delta}$. This formalizes the intuition that model errors compound over the planning horizon; as model accuracy ($\delta$) decreases, the safe planning horizon must shorten to avoid catastrophic inversions.

Significance and Claims

The paper claims to provide the first ordinal notion of safety for world models, shifting the focus from cardinal prediction accuracy to the preservation of policy ordering. Its primary significance lies in:

1. Formalizing the limits of safe planning: It proves that in sufficiently rich policy classes, relying on imperfect world models inevitably leads to exploitation, regardless of the specific learning algorithm used.
2. Unifying Hacking and Exploitation: It establishes a formal bridge between reward hacking and model exploitation, showing they share a common geometric root (value inversion) while highlighting the distinct mathematical challenges posed by transition models.
3. Practical Bounds: By introducing $\epsilon$-exploitability, the paper offers a practical framework for quantifying safety. It provides a closed-form expression for the maximum planning horizon a given level of model error can support, offering a theoretical basis for conservative planning in model-based RL.

The authors conclude that while perfect safety (unexploitability) is unattainable in general settings, the derived safe horizons can guide the development of safer policy synthesis methods and provide metrics for quantifying the quality of synthesized policies with respect to safety objectives.