Merton's Problem with Recursive Perturbed Utility

1. Problem Statement

The paper addresses a fundamental discrepancy between classical financial theory and behavioral evidence:

• Classical View: Merton's continuous-time investment problem predicts that rational, risk-averse investors follow deterministic portfolio rules (a specific function of time and state).
• Behavioral Reality: Empirical and laboratory evidence suggests individuals often prefer randomized decisions, exhibiting "noisy" choices even when facing identical problems (e.g., "blind boxes," sushi omakase).
• The Challenge: Extending the Additive Perturbed Utility (APU) theory (Fudenberg et al., 2015) from static settings to the dynamic Merton problem is non-trivial.
  • Ill-posedness: In a continuous-time setting with CRRA preferences, a naive additive model (maximizing expected utility + integral of entropy) is ill-posed if risk aversion ($\gamma$) is not sufficiently strong. The agent can drive the objective value to infinity by increasing the variance of the portfolio choice without bound.
  • Intractability: Even when well-posed (high risk aversion), the additive model fails to yield closed-form solutions, making policy analysis difficult.

Objective: To develop a mathematically tractable and economically sound framework that endogenizes the preference for randomization in a dynamic investment setting, solving Merton's problem with a "preference for stochasticity."

2. Methodology

A. Recursive Perturbed Utility (RPU)

The authors introduce Recursive Perturbed Utility (RPU), a novel stochastic differential utility that incorporates an entropy-based preference for randomization into a recursive aggregator.

• Mechanism: Unlike the additive APU, RPU uses a recursive structure inspired by Epstein-Zin utility and Uzawa preferences. The "temperature" (weight on randomization) becomes endogenous and history-dependent.
• The Aggregator: The utility process $J^\pi_t$ satisfies a Backward Stochastic Differential Equation (BSDE):
  $$dJ^\pi_t = -\lambda_t H(\pi_t) [(1-\gamma)J^\pi_t + 1] dt + Z^\pi_t \cdot dB_t$$
  where $H(\pi_t)$ is the differential entropy of the randomized control $\pi_t$, and $\lambda_t$ is the primary temperature parameter.
• Economic Intuition: The term $[(1-\gamma)J^\pi_t + 1]$ acts as a state-dependent discount rate.
  • If $\gamma > 1$ (typical risk aversion), past randomization reduces the marginal utility of current randomization (dampening effect), preventing the "runaway" variance seen in additive models.
  • This creates a trade-off: the agent gains utility from randomization but faces a diminishing return as randomization accumulates.

B. Market Setup

• Environment: A general Markovian incomplete market with a risk-free bond and a risky stock.
• Factors: The stock return and volatility depend on an observable stochastic factor $X_t$ (e.g., stochastic volatility or mean reversion).
• Control: The investor chooses a randomized control $\pi_t$, a probability distribution over portfolio weights. The wealth process $W^\pi_t$ follows an "exploratory" SDE where the volatility includes a term proportional to the variance of $\pi_t$.

C. Optimization Approach

The authors apply dynamic programming to derive the Hamilton-Jacobi-Bellman (HJB) equation. They utilize a two-stage optimization:

1. Fix the mean and variance of the distribution $\pi$.
2. Maximize entropy (which implies a Gaussian distribution for the optimal $\pi$).
3. Optimize over the mean and variance parameters.

3. Key Contributions

1. Resolution of Ill-posedness: The paper proves that the RPU framework yields a well-posed optimization problem for all $\gamma > 0$, overcoming the infinite value issue of the additive APU in continuous time.
2. Closed-Form Gaussian Policy: The authors prove that the optimal randomized policy is Gaussian and can be expressed in closed form.
   • Variance: Inversely proportional to risk aversion ($\gamma$) and stock volatility ($\sigma^2$).
   • Mean: Independent of wealth but dependent on time and stochastic factors.
3. Bias Analysis: Unlike previous reinforcement learning (RL) results where randomized policies are often unbiased (mean equals the deterministic optimum), the RPU-optimal policy is generally biased. The mean deviates from the classical Merton solution due to the interaction between randomization and intertemporal hedging.
4. Asymptotic Expansion: The paper provides a rigorous asymptotic expansion in terms of the perturbation weight (temperature $\lambda$) to quantify the financial cost of randomization.

4. Main Results

A. Structure of the Optimal Policy

The optimal policy $\pi^*_t$ is a Normal distribution $N(\mu^*_t, \sigma^2_{\pi})$:

• Variance: $\text{Var}(\pi^*_t) = \frac{\lambda}{\gamma \sigma^2(t, X_t)}$.
  • Higher risk aversion or market volatility leads to less randomization (lower variance).
• Mean: $\text{Mean}(\pi^*_t) = \underbrace{\frac{\mu - r}{\gamma \sigma^2}}_{\text{Myopic Term}} + \underbrace{\frac{\rho \nu}{\gamma \sigma} u_x(t, X_t)}_{\text{Hedging Term}}$.
  • The Myopic Term is the classical Merton ratio.
  • The Hedging Term arises from market incompleteness (correlation $\rho$ between stock and factor). Crucially, $u_x$ (the derivative of the value function component) depends on the randomization parameter $\lambda$ via a non-linear PDE. Thus, the presence of randomization alters the hedging demand, causing a bias in the mean policy.

B. Conditions for Unbiasedness

The optimal policy is unbiased (Mean = Classical Merton) only in specific cases:

1. Complete Markets: $\rho = 0$ (no correlation between stock and factor) or $\nu = 0$ (deterministic factor).
2. Log Utility: $\gamma = 1$. In this case, the RPU collapses to the additive APU, and the hedging term vanishes.
3. Specific Temperature Functions: If $\lambda$ is chosen such that the PDE solution $u$ remains independent of $\lambda$.

C. Asymptotic Analysis (Financial Cost)

The authors expand the solution in powers of the temperature parameter $\lambda$:

• Policy Deviation: The deviation of the optimal mean from the classical Merton policy is of order $O(\lambda)$.
• Wealth Loss: The relative wealth loss (the cost an investor is willing to pay to enjoy randomization) is of order $O(\lambda^2)$.
  • Significance: This implies that while the strategy changes linearly with the preference for randomization, the financial penalty is quadratic and thus small for small preferences. The "pleasure" of randomization comes at a relatively low financial cost.

5. Significance and Implications

• Theoretical Bridge: The paper successfully bridges behavioral economics (preference for randomization) and continuous-time stochastic control, providing a rigorous alternative to the deterministic Merton framework.
• Distinction from RL: While the mathematical form resembles "entropy regularization" in Reinforcement Learning (RL), the motivation is fundamentally different. In RL, entropy is an artificial tool to encourage exploration in unknown environments. In this paper, entropy represents an intrinsic human preference for randomness in a known environment.
• Market Incompleteness: The results highlight that in incomplete markets, the desire for randomization interacts with hedging motives, fundamentally altering the optimal investment strategy (making it biased).
• Quantitative Insight: The asymptotic results provide a metric for the "financial cost" of behavioral noise, suggesting that moderate preferences for randomization do not drastically destroy wealth, offering a potential explanation for why such behavior persists in real markets.

In summary, this paper establishes that a rational investor with an intrinsic preference for randomization will adopt a Gaussian portfolio policy with a specific variance and a biased mean, where the bias is driven by the interplay between randomization and intertemporal hedging in incomplete markets.