Technical Summary: Prospect-Theory Behavior from Bellman Optimality in MDPs with Catastrophic States

1. Problem Statement

This paper investigates whether standard, risk-neutral optimal control in Markov Decision Processes (MDPs) can generate behavioral signatures typically associated with Prospect Theory (PT), specifically when the environment contains an absorbing catastrophic state.

The authors challenge the conventional interpretation that PT phenomena—such as an S-shaped value function, loss aversion ($\lambda > 1$), and the reflection effect (risk aversion in gains, risk seeking in losses)—necessitate nonlinear utility functions, distorted probability weighting, or reference-dependent preferences. Instead, the paper asks if these patterns emerge purely from the structural dynamics of an MDP with a "catastrophe boundary" (an absorbing state representing ruin, bankruptcy, or death) and linear rewards.

The core problem is to determine if a risk-neutral agent, optimizing the standard Bellman equation, will exhibit:

1. S-shaped value functions: Convex near the catastrophe and concave in the far field.
2. Endogenous loss aversion: A state-dependent sensitivity coefficient $\lambda^*(S) > 1$.
3. Reflection effect: A policy reversal where the agent acts risk-averse in "growth" regimes (positive drift) and risk-seeking in "decline" regimes (negative drift) near the catastrophe, despite local expected value comparisons suggesting the opposite.

2. Methodology

Model Formulation

The authors define a catastrophe-boundary MDP with the following properties:

• State Space: $S = \{S_{cat}, S_{cat}+1, \dots\}$, where $S_{cat}=0$ is an absorbing catastrophe with value $V_{cat} \le 0$.
• Actions:
  • Safe: Deterministic transition $S' = S + \Delta_s$.
  • Risky: Stochastic transition $S' = S + \Delta_w$ with probability $p$, and $S' = S + \Delta_\ell$ with probability $1-p$ (where $\Delta_w > 0 > \Delta_\ell$).
• Objective: Standard risk-neutral Bellman optimality with linear rewards and discount factor $\beta \in (0, 1)$.
• Regimes:
  • Growth: $E[\text{risky}] > \Delta_s > 0$.
  • Decline: $E[\text{risky}] < \Delta_s < 0$.

Analytical and Numerical Approach

1. Value Iteration: The system is solved via value iteration (tolerance $10^{-10}$) across 495 parameter configurations to generate optimal value functions $V^*(S)$ and policies.
2. Endogenous Loss Sensitivity: The authors define a structural loss sensitivity coefficient analogous to the PT parameter $\lambda$:
   $$\lambda^*(S) = \frac{|V^*(S) - V^*(S + \Delta_\ell)|}{|V^*(S + \Delta_w) - V^*(S)|}$$
3. Asymptotic Analysis: The authors derive a closed-form expression for the asymptotic limit $\bar{\lambda} = \lim_{S \to \infty} \lambda^*(S)$ by analyzing the characteristic equation of the linear recurrence governing the value function in the far field.
4. Robustness Testing:
   • Model-Free Learning: Tabular Q-learning is employed to verify if a model-free agent reproduces the phenomena without knowledge of the transition kernel.
   • Stochastic Transitions: The deterministic transitions are perturbed with additive noise (Gaussian, heavy-tailed Student-t, and asymmetric skew-normal) to test sensitivity to distributional assumptions.

3. Key Contributions

1. Emergence of Three Prospect-Theory Signatures

The paper demonstrates that standard Bellman optimality alone produces:

• S-Shaped Value Function: In growth regimes, $V^*(S)$ is convex near the catastrophe (due to the high cost of the "downside jump" into ruin) and concave in the far field.
• Endogenous Loss Aversion: The ratio of value drops to gains, $\lambda^*(S)$, is strictly greater than 1 and converges to an asymptotic plateau $\bar{\lambda}$.
• Reflection Effect: The optimal policy reverses based on environmental drift. In growth, the agent plays safe near the catastrophe (risk aversion) to avoid ruin, even if the risky action has a higher immediate expected return. In decline, the agent plays risky near the catastrophe (risk seeking) because the safe action only delays inevitable ruin, whereas a risky win offers a chance to escape.

2. Mechanism: Continuation Values, Not Preferences

The authors establish that these behaviors are driven by continuation values. Near an absorbing boundary, the proximity to irreversible failure disproportionately weights downside risks. This dynamic effect inverts local expected-value rankings without requiring any curvature in the utility function or probability weighting.

3. Closed-Form Asymptotic Loss Aversion

The paper derives a closed-form formula for the asymptotic loss aversion plateau $\bar{\lambda}$:
$$\bar{\lambda} = \frac{z^{-r} - 1}{1 - z}$$
where $r = |\Delta_\ell|/\Delta_w$ is the payoff asymmetry ratio, and $z \in (0, 1)$ is the unique root of the characteristic equation $\beta[p z + (1-p)z^{-r}] = 1$.

• Scale Invariance: $\bar{\lambda}$ depends only on $(p, \beta, r)$, not on the absolute magnitude of payoffs.
• Boundary Dominance: The formula allows decomposition of $\bar{\lambda}$ into a "boundary contribution" (present even when $r=1$, i.e., symmetric payoffs) and an "asymmetry contribution." The authors show that the boundary contribution dominates; even with symmetric payoffs ($r=1$), $\bar{\lambda} > 1$ in growth regimes.

4. Robustness to Learning and Noise

• Q-Learning: A tabular Q-learning agent reproduces the S-shape, the safe zone, the "Hail Mary" effect, and $\lambda^* > 1$ with high correlation to the exact solution ($R=0.98$ in growth, $R=1.00$ in decline).
• Stochasticity: The phenomena persist under Gaussian, heavy-tailed, and asymmetric noise up to 50% of the step size. The asymptotic plateau deviates from the closed-form prediction by at most 9.6% under risky-channel noise.

4. Results

• Quantitative Fit: The closed-form formula for $\bar{\lambda}$ matches numerical solutions with $R^2 = 0.999$.
• Decomposition of Loss Aversion: Across 48 parameter cells, the "boundary contribution" (the value of $\bar{\lambda}$ when $r=1$) accounts for the majority of the loss aversion. The "asymmetry contribution" (the increase due to $r > 1$) has a median share of only 4.6% at $r=1.25$ and rises to 13.9% at $r=2$.
• Policy Reversal: In all 69 tested decline configurations, the agent plays risky near the catastrophe, confirming the "Hail Mary" effect. In growth regimes, the agent plays safe for a specific number of states ($|\Delta_\ell|$) near the boundary before switching to risky.
• Noise Sensitivity: Safe-channel noise has negligible impact ($\le 0.41\%$ deviation), while risky-channel noise reduces $\bar{\lambda}$ monotonically but preserves the qualitative structure.

5. Significance and Claims

The paper claims a structural and normative contribution rather than a psychological one. It does not argue that human behavior is optimal or that this mechanism fully explains Prospect Theory. Instead, it posits that:

1. Sufficiency of Structure: An absorbing catastrophic state is a sufficient structural mechanism to generate PT-like signatures (S-shape, loss aversion, reflection effect) under standard, risk-neutral optimization.
2. Reinterpretation of Empirical Data: The observed variation in loss aversion coefficients in behavioral economics (e.g., dependence on payoff asymmetry) can be partially explained by the structural properties of the decision environment (probability $p$, discount $\beta$, and asymmetry $r$) rather than solely by stable psychological parameters.
3. Rationality of Extreme Behavior: Risk-averse behavior in gains and risk-seeking behavior in losses near failure can be rational responses to continuation values, not necessarily pathological deviations from rationality.
4. Implications for RL: Safe or desperate behaviors near absorbing failures in Reinforcement Learning systems may emerge naturally from the environment structure without the need for hand-coded safety constraints or preference modifications.

The authors explicitly state that probability weighting, framing, and reference dependence remain outside the model and that their findings are meant to sit alongside, not displace, psychological accounts of human decision-making.