💰 quantitative finance

Adaptive Partitioning and Learning for Stochastic Control of Diffusion Processes

This paper proposes an adaptive partitioning model-based reinforcement learning algorithm for controlled diffusion processes in unbounded continuous state spaces, establishing regret bounds that depend on a novel zooming dimension and demonstrating effectiveness in high-dimensional financial applications like multi-asset portfolio selection.

Original authors: Hanqing Jin, Renyuan Xu, Yanzhao Yang

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanqing Jin, Renyuan Xu, Yanzhao Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to navigate a vast, endless ocean to find the best fishing spots. The ocean represents the state space (where the robot is), and the robot's decisions on which way to steer represent the action space.

In many traditional learning problems, the ocean is a small, fenced-in pool. You can easily map every inch of it. But in the real world—especially in finance and economics—the ocean is unbounded. It stretches forever, and the "rewards" (like profits) can grow incredibly large if you get lucky.

This paper introduces a new way for a robot (or an algorithm) to learn how to navigate this infinite ocean without getting lost or overwhelmed. Here is the breakdown of their approach using simple analogies:

1. The Problem: The "Infinite Map" Dilemma

If you try to draw a map of an infinite ocean with a fixed grid (like graph paper), you run into two problems:

Too Fine: If the grid squares are tiny to be accurate, you need infinite paper and time.
Too Coarse: If the squares are huge, you miss the important details (like a hidden reef or a school of fish).

Most existing methods assume the ocean is a small, bounded pool. This paper tackles the much harder problem of an infinite ocean where the rewards can grow polynomially (like compound interest, where small gains can eventually become massive).

2. The Solution: The "Smart Zoom" Camera

The authors propose an algorithm called APL-Diffusion (Adaptive Partitioning and Learning for Diffusions). Think of this algorithm as a smart camera with a zoom lens that only focuses where it matters.

Instead of trying to map the whole ocean at once, the algorithm does the following:

Starts with a Rough Sketch: It divides the ocean into large, manageable chunks (partitions).
Explores and Learns: As the robot moves, it collects data about the water's current (drift) and how choppy it is (volatility).
The "Zoom" Mechanism: This is the core innovation. If the robot enters a chunk of the ocean where the data is confusing or the "guess" about the current is shaky, the algorithm splits that chunk in half. It zooms in to create a finer map just for that specific area.
Stays Focused: If an area is well-understood or rarely visited, it stays as a large chunk. It doesn't waste time drawing tiny details for empty, calm waters.

3. Handling the "Infinite" and "Growing" Parts

Since the ocean is infinite, the algorithm has a safety net. It focuses its learning on a large, central "safe zone" (a big circle).

The Boundary: If the robot wanders too far outside this safe zone, the algorithm uses a rough, "best guess" estimate rather than trying to learn the impossible.
Growing Rewards: In finance, a small mistake early on can lead to a huge loss later. The paper accounts for rewards that can grow very large (polynomial growth). The algorithm is designed to handle these "explosive" numbers without breaking, ensuring the robot doesn't panic when the stakes get high.

4. The Result: A Better Map with Less Effort

The paper proves mathematically that this "Smart Zoom" approach works efficiently.

Regret: In learning terms, "regret" is the difference between how well the robot did and how well it could have done with a perfect map.
The Finding: The authors show that their algorithm keeps this "regret" low. It learns almost as fast as if the ocean were small and bounded, even though it is infinite.
The "Zooming Dimension": They introduce a new concept called "zooming dimension." Think of this as measuring how "complicated" the ocean really is. Even if the ocean is huge, the important parts might only exist on a simple path (like a narrow river). The algorithm is smart enough to realize it only needs to map that river, not the whole ocean, making learning much faster.

5. Real-World Testing

The authors didn't just do math; they tested it.

Test 1: A simple 1D problem (like navigating a straight line). The algorithm successfully zoomed in on the best areas and ignored the rest.
Test 2: A Multi-Asset Portfolio. Imagine an investor trying to balance money across 5 different stocks and a risk-free bank account. This is a high-dimensional, complex problem. The algorithm successfully learned how to allocate money to maximize returns, even though the math behind it is incredibly complex.

Summary

In short, this paper teaches a computer how to learn in a world that is too big to map completely and too risky to guess blindly. By using a smart, adaptive zooming strategy, the algorithm focuses its energy only on the areas that need attention, allowing it to learn optimal strategies for complex, infinite problems like managing a financial portfolio, all while providing mathematical guarantees that it won't get lost.

Technical Summary: Adaptive Partitioning and Learning for Stochastic Control of Diffusion Processes

Problem Formulation
This paper addresses the challenge of Reinforcement Learning (RL) for controlled diffusion processes characterized by unbounded continuous state spaces, bounded continuous action spaces, and rewards with polynomial growth. These settings are prevalent in finance, economics, and operations research (e.g., portfolio optimization, dynamic hedging, and inventory management) but remain theoretically underdeveloped compared to bounded or tabular MDPs.

The authors formulate the problem as a discrete-time Markov Decision Process (MDP) induced by an Euler–Maruyama discretization of a continuous-time controlled stochastic differential equation (SDE). The state transition is governed by:
$X_{h+1} - X_h = \mu_h(X_h, A_h)\Delta + \sigma_h(X_h, A_h)B_h\sqrt{\Delta}$
where $\mu$ and $\sigma$ are unknown drift and volatility functions dependent on the state-action pair, and $B_h$ are i.i.d. Gaussian noise. The reward function $\bar{R}_h(x, a)$ is allowed to grow polynomially with the state, relaxing the standard bounded reward assumption. The goal is to minimize cumulative regret over $K$ episodes of horizon $H$ .

Methodology: APL-Diffusion Algorithm
The authors propose APL-Diffusion, a model-based algorithm that integrates adaptive partitioning with local estimation of system dynamics. Key components include:

State Localization: To handle the unbounded state space, the algorithm restricts learning to a bounded ball $S_1 = \{x : \|x\| \le \rho\}$ . Trajectories exiting this ball are handled via coarse estimations that do not dominate the regret bound.
Adaptive Partitioning: The joint state-action space is partitioned into hypercubes. Unlike uniform grids, the partition is refined adaptively. A block $B$ is split if its estimated bias exceeds a statistical confidence bound. This balances exploration and approximation error.
Local Estimators: Within each partition block, the algorithm maintains estimators for:
- Drift ( $\hat{\mu}$ ) and Volatility ( $\hat{\sigma}$ ): Constructed from observed state transitions.
- Reward ( $\hat{R}$ ): Constructed from observed rewards.
- Transition Kernel: Modeled as a Gaussian distribution conditioned on the estimated drift and volatility.
Optimistic Planning: The algorithm constructs optimistic Q-values and value functions using Upper Confidence Bounds (UCB) that account for estimation uncertainty in the drift, volatility, and reward, as well as the bias introduced by the partition size.
Policy Execution: Actions are selected greedily based on the optimistic Q-function estimates, with the action space within a block sampled uniformly from the block's projection.

Key Contributions

Theoretical Framework for Unbounded Diffusions: The paper extends RL theory to unbounded diffusion processes with polynomially growing rewards, a setting where existing theoretical guarantees (often relying on boundedness) fail.
Novel Zooming Dimension: A new notion of "zooming dimension" ( $z_{max,c}$ ) is defined specifically for unbounded state spaces. This metric quantifies the complexity of the near-optimal set and allows the regret bounds to depend on the intrinsic dimension of the problem rather than the ambient dimension.
Concentration Inequalities: The authors derive concentration inequalities for the estimators of drift and volatility under Lipschitz assumptions. A significant technical challenge addressed is establishing concentration for covariance matrices under only Lipschitz regularity of the volatility, requiring the introduction of intermediate terms not present in standard bounded MDP analyses.
Regret Bounds: The paper establishes a high-probability regret bound of order:
$\tilde{O}\left( H K^{\frac{1 - p^2 - (m+1)^2(z_{max,c}+2) - (m+1)(2d_S + 2m + 4)}{p(p+m+1)(z_{max,c}+2) + p(2d_S + 2m + 4)}} \right)$
where $p$ $p$ is the bounded moment order of the initial state distribution, $m+1$ $m + 1$ is the polynomial growth order of the reward, $d_S$ $d_{S}$ is the state dimension, and $z_{max,c}$ $z_{ma x, c}$ is the worst-case zooming dimension.
- As $p \to \infty$ (bounded initial moments), the bound recovers the order $\tilde{O}(H K^{\frac{z_{max,c}+1}{z_{max,c}+2}})$ , consistent with existing results for bounded settings (e.g., [Sinclair et al., 2023]).
- The bound explicitly captures the impact of heavy-tailed initial distributions and fast reward growth on learning efficiency.

Results and Validation

Theoretical Analysis: The regret analysis demonstrates that the algorithm achieves sublinear regret in the number of episodes $K$ . The bounds are shown to recover existing results for bounded settings as a special case while extending guarantees to a broader class of diffusion problems.
Numerical Experiments:
- 1D Example: A tractable one-dimensional control problem demonstrates that the algorithm adaptively refines partitions in regions of high optimal Q-values, leading to rapid convergence of the estimated value function. The empirical regret order (0.69) was found to be better than the worst-case theoretical bound (0.75).
- Mean-Variance Portfolio Optimization: A high-dimensional application involving 5 risky assets and 1 risk-free asset. Despite the problem violating strict Lipschitz assumptions on the action variable (due to volatility becoming arbitrarily small), the algorithm exhibited strong empirical performance with a regret order of 0.78, outperforming the worst-case theoretical bound of 0.875.

Significance
The paper claims to provide the first rigorous theoretical framework and algorithm for RL in controlled diffusion processes with unbounded states and polynomially growing rewards. By introducing a localized adaptive partitioning scheme and a tailored zooming dimension, the work bridges the gap between the theoretical tractability of bounded MDPs and the practical necessity of modeling unbounded financial and economic systems. The results suggest that efficient learning is possible in these complex, high-dimensional settings, provided the near-optimal regions possess a lower intrinsic dimensionality. The numerical experiments further validate the algorithm's robustness and effectiveness in realistic, high-dimensional financial applications.