💰 quantitative finance

Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study

This paper proposes a continuous-time reinforcement learning framework for mean-variance portfolio selection that learns optimal investment strategies directly from data without estimating unknown market coefficients, demonstrating through theoretical regret bounds and extensive empirical studies on S&P 500 constituents that it consistently outperforms traditional model-based approaches, particularly in volatile bear markets.

Original authors: Yilie Huang, Yanwei Jia, Xun Yu Zhou

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yilie Huang, Yanwei Jia, Xun Yu Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a ship trying to reach a distant island (your financial goal) as quickly and safely as possible. You want to maximize your speed (return) while minimizing the risk of hitting a rock or getting lost (variance/loss). This is the classic "Mean-Variance" problem that investors have been trying to solve for decades.

For a long time, the standard way to do this was like trying to navigate by drawing a perfect map of the ocean first. You would spend years measuring the currents, wind speeds, and wave patterns (estimating market coefficients like drift and volatility) to build a mathematical model. Once you had the map, you would calculate the perfect route.

The Problem: The ocean is chaotic. Your map is always wrong because you can't measure the currents perfectly, and the weather changes faster than you can update your map. If your map is slightly off, your "perfect" route might lead you straight into a storm. This is why traditional investment strategies often fail in real life.

The New Approach: "Learning by Doing" (Reinforcement Learning)

This paper introduces a smarter way to sail. Instead of trying to draw a perfect map of the ocean, the authors propose an AI captain that learns by sailing.

Here is the core idea broken down with simple analogies:

1. The "Model-Free" Captain

Traditional methods are like a student who memorizes a textbook on oceanography before ever stepping on a boat. If the textbook is wrong, the student fails.

The method in this paper (called CTRL) is like a pilot who learns to fly by actually flying. It doesn't care about the physics of the wind or the exact density of the air. It just looks at the horizon, tries a turn, sees if it gets closer to the destination, and adjusts. It learns the best strategy directly from the experience of the market, without ever trying to guess the "rules" of the market.

2. The "Exploration" Trick (The Dice Roll)

How does the AI learn if it doesn't know the rules? It uses a clever trick called Exploration.

Imagine the AI is playing a video game. To learn the best move, it doesn't just play the same move over and over. It occasionally rolls a dice to try random, slightly crazy moves.

In the paper: The AI randomly "jitters" its investment decisions. It might buy a little more of Stock A or a little less of Stock B than it normally would.
Why? This helps it discover hidden opportunities or dangers that a rigid, "perfect" plan would miss. It's like a chef tasting a dish and adding a pinch of salt, then a pinch of pepper, to find the perfect flavor, rather than following a recipe blindly.

3. The "Regret" Score (The Report Card)

The authors prove mathematically that this learning method works. They use a concept called Regret.

The Oracle: Imagine a magical being who knows the future perfectly and knows the exact currents of the ocean. This being has the "perfect" strategy.
The Regret: This is the difference between how much money the Oracle made and how much your AI captain made.
The Result: The paper proves that as the AI sails longer (over time), the "Regret" grows very slowly. Eventually, the AI becomes almost as good as the Oracle, even though it never knew the rules of the ocean. It catches up to the perfect strategy just by learning from its own mistakes and successes.

4. The Real-World Test (The S&P 500 Race)

The authors didn't just write theory; they put their AI captain in a race against 13 other famous captains (including traditional math models, simple "buy-and-hold" strategies, and other AI methods) using real stock data from the S&P 500 over 20 years.

The Results:

The Winner: The new AI strategy (CTRL) won almost every time.
The Storm Test: It performed especially well during "bear markets" (when the market crashes, like in 2008). While other strategies got wrecked or took a long time to recover, the AI captain recovered quickly.
The Secret Sauce: It didn't win because it had a better map. It won because it didn't rely on a map at all. It adapted to the chaos in real-time.

Why This Matters to You

Think of traditional investing like driving a car with a GPS that relies on old, static maps. If a road is closed or a bridge is out, the GPS sends you into a ditch because it thinks the road is still there.

This new approach is like a self-driving car with super-human reflexes. It doesn't need to know the road layout in advance. It sees the pothole, swerves, and keeps going. It learns from every bump in the road.

In short: This paper shows that in the chaotic, unpredictable world of finance, trying to predict the future (modeling) is less effective than learning from the present (reinforcement learning). The best way to navigate the market isn't to know the rules; it's to learn how to play the game better than anyone else.

1. Problem Formulation

The paper addresses the continuous-time Mean-Variance (MV) portfolio selection problem in a market where asset prices and observable market factors follow Itô diffusion processes.

The Challenge: The agent (investor) has no knowledge of the market coefficients (drift and volatility) governing these diffusion processes.
The Goal: To find an investment strategy that maximizes the Sharpe ratio (or minimizes variance for a given expected return) over a finite horizon $T$ , without relying on statistical estimation of market parameters.
Context: Traditional approaches rely on a "plug-in" paradigm: first estimating model parameters (e.g., via Maximum Likelihood or Kalman filtering) and then solving the stochastic control problem. This paper argues that this approach suffers from severe estimation errors (especially for expected returns) and sensitivity, leading to poor out-of-sample performance.

2. Methodology: Continuous-Time Reinforcement Learning (RL)

The authors propose a model-free, data-driven RL framework based on the theory of controlled diffusion processes (Wang et al., 2020; Jia and Zhou, 2022a,b).

Core Concepts

Stochastic Policies & Entropy Regularization: Instead of learning a deterministic policy, the agent learns a stochastic policy $\pi(\cdot|t, x, F)$ $π (\cdot ∣ t, x, F)$ , which maps states (time, wealth, factors) to probability distributions over actions (portfolios). An entropy regularizer is added to the objective function to enforce exploration.
- Objective: Minimize the entropy-regularized variance of terminal wealth.
Martingale Approach: The algorithm avoids estimating the Hamilton-Jacobi-Bellman (HJB) PDE. Instead, it relies on martingale conditions. The value function is characterized by the property that a specific process involving the value function and the entropy term is a martingale.
Actor-Critic Architecture:
- Critic (Policy Evaluation): Estimates the value function $J(t, x, F)$ by solving moment conditions derived from the martingale property using Generalized Method of Moments (GMM) or Temporal Difference (TD) learning.
- Actor (Policy Gradient): Updates the policy parameters by estimating the gradient of the objective function with respect to the policy, utilizing the "policy gradient theorem" adapted for continuous time.
Lagrange Multiplier: A specific Lagrange multiplier $w$ is introduced to handle the mean constraint of the MV problem, which is learned simultaneously with the policy parameters.

Specific Algorithm for Black-Scholes Market

For the specific case of a multi-stock Black-Scholes market (constant drift and volatility, no factors), the authors devise a concrete algorithm (CTRL):

Parametrization: They use specific functional forms for the value function and the stochastic policy (Gaussian distribution with affine mean and time-dependent covariance) inspired by the theoretical optimal solution, but treat the parameters as learnable variables.
Stochastic Approximation: The algorithm updates parameters ( $\theta$ for value, $\phi$ for policy, $w$ for the multiplier) iteratively based on observed wealth trajectories.
Convergence: The algorithm uses iterative projections to ensure parameters remain within valid sets (e.g., positive definite covariance matrices) and employs specific learning rate schedules to guarantee convergence.

3. Key Contributions

General RL Framework for MV: The paper extends general continuous-time RL theory to the specific, challenging setting of Mean-Variance optimization, which involves time-inconsistency and a non-standard objective function.
First Sublinear Regret Bound: The authors provide the first theoretical proof of a sublinear regret bound for a model-free learning algorithm in continuous-time MV portfolio selection.
- The regret is defined in terms of the Sharpe ratio.
- The bound is of the order $O(\sqrt{N \log N \log \log N})$ , where $N$ is the number of learning episodes. This implies that as training time increases, the algorithm's performance converges to the theoretical "oracle" optimum (which knows the true market parameters).
Resolution of the "Estimation Error" Problem: By bypassing the estimation of drift and volatility entirely, the method avoids the "curse of dimensionality" and sensitivity to estimation errors that plague classical plug-in methods.
Comprehensive Empirical Validation: A rigorous comparison against 13 alternative strategies (including static/dynamic MV, shrinkage estimators, factor models, and other RL algorithms) on S&P 500 data.

4. Empirical Results

The study evaluates the CTRL algorithm against benchmarks (e.g., Equal Weight, Sample-based MV, Black-Litterman, DDPG, PPO) using S&P 500 constituents from 2000–2020 (with 1990–2000 as burn-in).

Performance Metrics:
- Returns & Risk-Adjusted Metrics: CTRL achieved the highest annualized return (12.21%) and the highest Sharpe ratio (0.531), significantly outperforming all competitors.
- Robustness in Bear Markets: The strategy was particularly dominant during the 2000–2010 period (including the 2008 crisis), where many model-based strategies failed or suffered massive drawdowns. CTRL had the shortest recovery time (521 days vs. >800 for others).
- Transaction Costs: Even with high transaction costs (50 bps), CTRL maintained superior performance due to its low turnover (5.68%), whereas model-based MV strategies suffered from high turnover and concentration risks.
Comparison with Model-Based Counterparts:
- Classical continuous-time MV (ctmv) and static MV (mv) performed poorly, often resulting in negative returns or bankruptcy, highlighting the danger of parameter estimation errors in dynamic settings.
- Other RL algorithms (DDPG, PPO) underperformed CTRL, suggesting that generic RL algorithms lack the specific structural understanding required for financial diffusion processes.
Trading Characteristics: CTRL exhibited low concentration (max single-asset position <20%), low leverage, and zero bankruptcy rate across 100 simulation runs, demonstrating inherent stability.

5. Significance and Conclusion

Paradigm Shift: The paper demonstrates that "learning the policy without learning the model" is a superior paradigm for dynamic portfolio management compared to the traditional "estimate-then-optimize" approach.
Theoretical Rigor: It bridges the gap between theoretical RL (regret bounds) and practical finance, proving that model-free learning can achieve near-optimal performance in continuous-time settings.
Practical Impact: The proposed algorithm offers a robust solution for asset allocation in volatile and uncertain markets, outperforming both traditional econometric methods and existing machine learning baselines.
Future Work: The authors acknowledge limitations, such as the current restriction to Itô diffusions (excluding jumps or stochastic volatility in the regret proof), and identify extending these results to more complex market dynamics as a major open challenge.

In summary, this paper establishes a new benchmark for continuous-time portfolio optimization, proving that reinforcement learning can effectively solve the mean-variance problem without the pitfalls of statistical model estimation, backed by both rigorous mathematical proofs and extensive empirical evidence.