Data-Driven Merton's Strategies via Policy Randomization

1. Problem Statement

The paper addresses Merton's expected utility maximization problem in an incomplete market where the functional forms of the model primitives (drift $\mu$, volatility $\sigma$, and factor dynamics) are unknown.

• Context: Traditional "plug-in" methods require estimating these parameters from historical data before solving the optimization problem (separation of estimation and optimization). This approach suffers from estimation errors, particularly for expected returns, and is highly sensitive to model misspecification.
• Constraint: The agent is a "small investor" (price taker) who observes stock prices, a market factor process, and instantaneous volatility (e.g., via VIX), but does not know the underlying stochastic differential equations (SDEs) governing them.
• Goal: To derive an optimal investment strategy in a model-free, data-driven manner without estimating the market coefficients.

2. Methodology

The authors propose a novel framework that integrates Continuous-Time Reinforcement Learning (RL) with Policy Randomization.

A. The Auxiliary Problem with Randomized Policies

Instead of seeking a deterministic policy directly, the authors introduce an auxiliary problem where the agent selects actions from a specific class of Gaussian distributions (randomized policies).

• Policy Structure: The policy $\pi_t$ at time $t$ is a normal distribution $N(u_t, \frac{\lambda}{\gamma \sigma_t^2})$, where:
  • $u_t$ is the mean (the control to be learned).
  • $\lambda \geq 0$ is a "temperature" parameter controlling the degree of randomization.
  • The variance is inversely proportional to the stock volatility and risk aversion.
• Wealth Dynamics: Under this randomized policy, the wealth process includes an additional noise term derived from the policy variance, leading to a modified SDE.

B. Theoretical Foundation

The core theoretical contribution is proving that solving this auxiliary randomized problem yields the solution to the original deterministic Merton problem.

• Theorem 1: The mean of the optimal Gaussian policy for the auxiliary problem coincides exactly with the optimal deterministic policy of the original Merton problem.
• Justification: While randomization is typically used in RL for exploration (gathering information), here it serves as a technical tool. It transforms the problem into a form solvable by continuous-time RL algorithms (specifically Actor-Critic methods) without needing to estimate model primitives.
• Cost of Randomization: The authors quantify the "Equivalent Relative Wealth Loss" (ERWL) caused by randomization, showing it is a function of $\lambda$ and time horizon $T$, but the mean policy recovers the optimal utility.

C. Actor-Critic Algorithm Design

The authors develop online and offline Actor-Critic algorithms based on the martingale orthogonality conditions established in recent continuous-time RL literature (Jia & Zhou, 2022a,b).

• Critic (Value Function): Learns a function $\phi(t, x)$ representing the value function. Due to the homothetic property of CRRA utility, the value function is parameterized as $V(t,w,x) = \frac{w^{1-\gamma}}{1-\gamma} e^{\phi(t,x) - \dots}$.
• Actor (Policy): Learns the mean function $u(t,x)$.
• Learning Mechanism: The algorithms use "test functions" (martingale residuals) to update parameters.
  • Crucial Insight: Randomization ($\lambda > 0$) is necessary for the algorithm to function. If $\lambda = 0$ (deterministic), the variance of the learning signal (gradient estimate) becomes infinite, making policy improvement impossible. The randomization stabilizes the gradient estimates, balancing bias and variance.

3. Key Contributions

1. Reframing Randomization: The paper demonstrates that policy randomization in RL is not merely for exploration in unknown environments but is a mathematical necessity to solve continuous-time stochastic control problems with unknown coefficients when using gradient-based learning.
2. Model-Free Solution: It provides the first systematic data-driven solution to the Merton problem in incomplete markets without requiring the estimation of drift or volatility parameters.
3. Convergence Analysis: The authors prove the convergence of their offline algorithm in the Black-Scholes setting, achieving the optimal convergence rate of $O(n^{-1} \log n)$, and derive error bounds dependent on the temperature parameter $\lambda$.
4. Robustness to Model Misspecification: The study highlights that traditional plug-in methods fail when estimated parameters violate theoretical constraints (e.g., leading to infinite leverage in stochastic volatility models), whereas RL learns the policy directly, avoiding these "blow-up" solutions.

4. Results

The paper validates the methodology through extensive simulations and empirical studies.

A. Simulation Studies (Synthetic Data)

• Setup: A stochastic volatility model (3/2 model) with unknown parameters.
• Comparison: The proposed RL methods (Specific Parametric Form and Neural Networks) were compared against:
  • Omniscient: Theoretical optimal solution (known parameters).
  • Plug-in (Est-SV): Estimate parameters via MLE, then solve analytically.
  • Empirical Risk Minimization (ERM): Directly learn a deterministic policy.
  • Buy-and-Hold (B-H): Naïve benchmark.
• Findings:
  • Performance: The RL methods consistently outperformed the Plug-in and ERM methods, achieving utility values close to the Omniscient solution (e.g., 97% of optimal utility vs. 95% for Plug-in).
  • Robustness: When volatility data was noisy (simulating real-world estimation errors), the Plug-in method's performance collapsed (utility dropped significantly), while RL methods remained robust.
  • Small Data: RL with randomized policies significantly outperformed ERM on small datasets, whereas ERM required large datasets to converge.

B. Empirical Study (Real Market Data)

• Setup: Dynamic allocation between the S&P 500 and a risk-free asset using VIX as a volatility proxy (1990–2025).
• Findings:
  • Risk-Adjusted Returns: RL strategies (especially the Neural Network variant) achieved superior Sharpe ratios and Sortino ratios compared to all benchmarks.
  • Drawdowns: RL strategies exhibited significantly smaller maximum drawdowns and much faster recovery times during market crashes (e.g., 2000–2002, 2008) compared to the Plug-in method and Buy-and-Hold.
  • Adaptability: The RL agent adapted quickly to changing market regimes (e.g., shifting to conservative positions during bear markets), whereas the Plug-in method, relying on historical parameter estimates, lagged in adaptation.

5. Significance

This paper resolves a long-standing debate regarding the applicability of RL to financial portfolio choice for small investors.

• Beyond Exploration: It establishes that even when market data is exogenous (no need to "explore" to learn the market reaction), randomization is a technical enabler that allows the use of gradient-based learning algorithms to solve problems that are intractable via traditional estimation-optimization pipelines.
• Practical Impact: The results suggest that for real-world portfolio management, where models are imperfect and data is limited, data-driven RL strategies are more robust and adaptive than traditional model-based approaches.
• Theoretical Bridge: It successfully bridges the gap between continuous-time stochastic control theory and modern reinforcement learning, providing a rigorous foundation for "model-free" finance.