Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty

1. Problem Formulation

The paper addresses Continuous-time Risk-sensitive Reinforcement Learning (RL) within an entropy-regularized, exploratory diffusion process framework.

• Objective: The agent aims to maximize an exponential-form objective function:
  $$J(t, x; \pi; \epsilon) = \frac{1}{\epsilon} \log \mathbb{E} \left[ \exp\left( \epsilon \int_t^T (r(s, X_s^\pi, a_s^\pi) - \lambda \log \pi(a_s^\pi|s, X_s^\pi)) ds + h(X_T^\pi) \right) \bigg| X_t = x \right]$$
  where:
  • $\epsilon \neq 0$ is the risk sensitivity coefficient (negative $\epsilon$ implies risk aversion).
  • $\lambda \geq 0$ is the temperature parameter controlling the entropy regularization (exploration).
  • The term $-\lambda \log \pi$ acts as a reward for exploration, added inside the exponential.
• Context: Unlike standard RL which optimizes expected cumulative rewards (risk-neutral), this formulation accounts for the entire distribution of rewards. It is motivated by:
  1. Risk Attitude: Modeling agents that are sensitive to the variance of outcomes (e.g., financial trading).
  2. Distributional Robustness: Interpreting the objective as a robust control problem against model uncertainty (Knightian uncertainty), where the agent optimizes against the worst-case distribution within a Kullback-Leibler (KL) divergence ball.
• Challenge: Traditional discrete-time risk-sensitive RL relies on nonlinear recursive relations (e.g., "exponential Bellman equations"), which are computationally difficult and lack tractable martingale characterizations.

2. Methodology

The core methodological contribution is the transformation of the risk-sensitive problem into a martingale problem augmented by a Quadratic Variation (QV) penalty.

A. The Quadratic Variation Penalty

Leveraging the connection between exponential martingales and Itô's lemma, the author shows that the risk-sensitive objective is equivalent to a standard additive objective with an extra penalty term: the quadratic variation of the value process.

• Key Insight: The exponential form is removed, and the problem is re-characterized by the martingale property of a process involving the value function $J$ and the Q-function $q$, augmented by $\frac{\epsilon}{2} d\langle J \rangle_t$.
• Result: This transforms the complex nonlinear recursive structure of discrete-time risk-sensitive RL into a linear martingale condition in continuous time (linear in the $q$-function, though still nonlinear in the value function due to the QV term).

B. Definition of the Risk-sensitive $q$-function

The paper defines the $q$-function for continuous-time risk-sensitive RL as:
$$q(t, x, a; \pi) = \mathcal{L}^a J(t, x; \pi) + r(t, x, a) + \frac{\epsilon}{2} |\sigma(t, x, a)^\top \nabla_x J(t, x; \pi)|^2$$
where $\mathcal{L}^a$ is the infinitesimal generator.

• Optimality Condition: The optimal policy $\pi^*$ is a Gibbs measure (Boltzmann distribution) of the optimal $q$-function:
  $$\pi^*(a|t, x) = \exp\left( \frac{1}{\lambda} q^*(t, x, a) \right)$$
  (normalized to integrate to 1).
• Martingale Characterization: The optimal value function $J^*$ and $q^*$ satisfy a specific martingale condition:
  $$J^*(s, X_s) + \int_t^s \left( r(u, X_u, a_u) - q^*(u, X_u, a_u) \right) du + \frac{\epsilon}{2} \langle J^* \rangle_s$$
  is a martingale. This allows for the use of stochastic approximation algorithms to learn $J$ and $q$ simultaneously.

C. Algorithm Design

The paper proposes Risk-sensitive Q-Learning algorithms (both on-policy and off-policy) based on minimizing the martingale loss (temporal difference error augmented by the QV term).

• Loss Function: The algorithm minimizes the squared error of the martingale condition, effectively solving for parameters that make the process a martingale.
• Handling Constraints: Since the $q$-function must satisfy an integral constraint (to ensure the policy is a valid probability distribution), the algorithm uses specific parameterizations (e.g., Gaussian forms) where this constraint is analytically tractable.

3. Key Contributions

1. Theoretical Framework: Establishes the first rigorous $q$-learning theory for continuous-time risk-sensitive RL. It proves that the problem can be solved via a linear martingale condition in the $q$-function, avoiding the "exponential Bellman equation" complexity of discrete-time counterparts.
2. Q-Learning vs. Policy Gradient: Demonstrates that the standard relationship between policy gradients and $q$-learning (valid in risk-neutral settings) breaks down in risk-sensitive settings.
   • Reason: The Feynman-Kac PDE for the value function becomes nonlinear due to the QV term ($|\nabla J|^2$). Consequently, the policy gradient contains an uncomputable cross-variation term between the value process and its gradient, making unbiased policy gradient estimation difficult.
   • Implication: This highlights a theoretical advantage of Q-learning over policy gradient methods for risk-sensitive problems.
3. Ergodic Extension: Extends the theory to infinite-horizon (ergodic) tasks, providing martingale characterizations for long-term average risk-sensitive objectives.
4. Convergence Analysis (Merton's Problem): Provides a theoretical convergence analysis for Merton's investment problem with power utility.
   • Reveals the role of the temperature parameter ($\lambda$): It acts analogously to a learning rate. Higher $\lambda$ increases exploration (faster convergence of parameters) but introduces noise; lower $\lambda$ reduces noise but slows learning.
   • Derives conditions on the step size and temperature schedules to achieve optimal convergence rates ($\tilde{O}(1/n)$).
5. Finite-Sample Robustness: Through numerical experiments on Linear-Quadratic (LQ) control, shows that risk-sensitive RL with finite data yields lower estimation errors than risk-neutral RL. It acts as a regularizer against distributional shift, with the optimal risk-sensitivity coefficient $\epsilon$ diminishing as sample size increases.

4. Results

• Merton's Investment Problem:
  • The algorithm successfully learns the optimal portfolio allocation without knowing the drift ($\mu$) or volatility ($\sigma$).
  • Convergence: The Mean Squared Error (MSE) of learned parameters converges at a rate matching state-of-the-art bounds (ignoring logarithmic factors).
  • Regret: The accumulated Equivalent Relative Wealth Loss (ERWL) grows sublinearly. The paper distinguishes between executing a deterministic policy (faster convergence, lower regret) vs. the stochastic policy required during learning (higher regret due to exploration noise).
• Off-Policy Linear-Quadratic Control:
  • Simulations on a system with controlled volatility demonstrate that risk-sensitive RL outperforms risk-neutral RL when data is scarce.
  • Optimal $\epsilon$: For small sample sizes, a significant negative $\epsilon$ (high risk aversion) minimizes estimation error. As sample size grows, the optimal $\epsilon$ approaches 0 (risk-neutral), confirming the link between risk sensitivity and distributional robustness.
  • Temperature Impact: In off-policy settings, lower temperature parameters generally lead to smaller estimation errors because the fixed dataset limits the benefit of exploration, and high temperature deviates the objective too far from the true distribution.

5. Significance

• Bridging Theory and Practice: The paper bridges the gap between continuous-time stochastic control and discrete-time RL, providing a tractable framework for high-frequency or real-time decision-making under uncertainty (e.g., algorithmic trading, robotics).
• Algorithmic Efficiency: By replacing nonlinear exponential recursions with additive martingale conditions involving QV, the proposed method is computationally more efficient and numerically stable than discrete-time risk-sensitive approaches.
• Understanding Exploration: It provides a formal theoretical understanding of the temperature parameter in entropy-regularized RL, showing its dual role in balancing exploration (learning speed) and exploitation (noise reduction).
• Robustness to Data Scarcity: The findings suggest that risk-sensitive formulations are a natural and effective mechanism for handling model uncertainty and limited data, offering a data-driven approach to robust control without requiring explicit uncertainty sets.

In summary, this paper establishes a foundational theory for continuous-time risk-sensitive RL, proving that the Quadratic Variation penalty is the key to simplifying the problem structure, enabling efficient Q-learning algorithms that outperform traditional methods in both theoretical convergence and finite-sample robustness.