📊 statistics

Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

This paper proposes a model-free continuous-time reinforcement learning algorithm, Hawkes-CT DDPG, that solves non-Markovian stochastic control problems driven by multivariate Hawkes jump-diffusions by first approximating the system with a finite-dimensional Markovianized representation and then applying deterministic policy gradient learning.

Original authors: Tomasz R. Bielecki, Thibaut Mastrolia, Haoze Yan

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomasz R. Bielecki, Thibaut Mastrolia, Haoze Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, threats rarely arrive as isolated, independent events. Instead, they tend to cluster, where one breach or attack makes a second one more likely to follow, creating a cascading chain of damage. This behavior, known as self-excitation, is a fundamental pattern in everything from earthquakes to financial market crashes, and it is increasingly central to understanding cyber risk. To manage such risks, defenders need to make dynamic decisions about how to allocate their limited security resources as the threat environment shifts. However, the mathematical tools traditionally used to optimize these decisions struggle when the system's memory is complex. If the likelihood of an attack depends on the entire history of past events rather than just the current moment, the problem becomes too tangled for standard methods to solve efficiently.

A team of researchers has developed a new approach to untangle this complexity, creating a method that allows computers to learn optimal defense strategies in real-time, even when the underlying rules of the threat are unknown. Their work focuses on a specific type of mathematical model called a Hawkes process, which is designed to capture that clustering behavior. The challenge they faced was that these models are "non-Markovian," meaning the system's future depends on its entire past, making it impossible to use standard, efficient learning algorithms that rely only on the present state. To overcome this, the researchers devised a way to compress the system's infinite memory into a finite set of observable signals, effectively turning a history-dependent problem into one that can be solved with modern machine learning.

The core of their solution involves a technique called "Markovianization." Imagine trying to predict the weather not just by looking at the current temperature, but by remembering every single raindrop that fell in the last century. That is the level of memory a Hawkes process holds. The researchers realized that instead of trying to remember every past event, one could approximate the system's memory using a collection of simple, decaying filters. They constructed a new state for the learning algorithm that includes the current system status plus a set of these filters, each tracking how past events have faded over time. By doing this, they transformed the complex, history-dependent problem into a manageable, finite-dimensional one that a computer can navigate.

Once the problem was reshaped into this simpler form, the team applied a continuous-time reinforcement learning algorithm, which they named Hawkes CT-DDPG. Unlike traditional methods that learn in discrete steps, this algorithm learns continuously, adjusting its strategy as events happen in real time. The system operates in a model-free manner, meaning it does not need to know the specific mathematical formulas governing the attacks or the defense mechanisms. Instead, it learns purely by observing the timing of events, the state of the system, and the costs associated with different actions. It uses a neural network to act as a "critic" that evaluates how good a decision was, and another network as an "actor" that decides what action to take next, constantly refining its policy to minimize the total cost of security incidents.

To test their method, the researchers simulated three different types of threat environments, each with a different pattern of how past events influence the future. The first scenario used a simple exponential pattern, where the influence of past events fades quickly and predictably. The second used an Erlang pattern, which represents a more complex, multi-stage fading process. The third and most difficult scenario used a power-law pattern, where the influence of past events decays very slowly, creating a long, heavy tail of memory that is notoriously difficult to model. In each case, they compared their continuous-time learning method against standard discrete-time learning techniques and against a theoretical "oracle"—a perfect solution that knows all the underlying rules in advance.

The results showed that the new method was highly effective. In the simple exponential case, the algorithm performed nearly as well as the perfect oracle, reducing costs significantly compared to static defense strategies. When the researchers moved to the more complex Erlang and power-law scenarios, the advantage of their approach became even clearer. The algorithm that used the memory filters to approximate the system's history consistently outperformed the standard learning methods that ignored the memory structure. In the power-law scenario, which has no exact simple representation, the filtered version of their algorithm reduced the average cost by nearly five percent compared to the version that did not use the memory filters. This demonstrated that capturing the system's history through these filters was crucial for making good decisions.

Furthermore, the study proved that this approach works even when the specific details of the threat are unknown. The algorithm successfully learned to minimize costs without ever being told the exact mathematical shape of the threat's memory or the specific coefficients that govern how attacks spread. By observing only the arrival times of events and the resulting system states, it was able to construct an internal model that was sufficient to find near-optimal defense strategies. The researchers verified their findings by comparing their results to analytical solutions where they existed and to high-fidelity numerical benchmarks where they did not, confirming that their method consistently produced the best results among all the learning techniques tested.

This work represents a significant step forward in managing complex, self-exciting systems. It shows that even when a problem seems too tangled to solve because it depends on too much history, it is possible to find a practical solution by approximating that history with a finite set of tools. For organizations facing the constant, evolving threat of cyberattacks, this means there is now a viable path to using artificial intelligence to dynamically allocate security resources, learning from every incident to better protect against the next wave of threats, all without needing to fully understand the complex mathematics of the danger itself.

Technical Summary: Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

Problem Statement

This paper addresses the stochastic control of multivariate Hawkes-driven stochastic differential equations (SDEs) in a non-Markovian setting. The core challenge arises from the path dependence inherent in Hawkes processes: the intensity of future events depends on the entire history of past events, rendering the system non-Markovian unless the kernel is a simple exponential. This memory dependence prevents the direct application of classical stochastic control theory and standard dynamic programming principles (Hamilton-Jacobi-Bellman equations) which rely on finite-dimensional state spaces.

The specific objective is to develop a model-free reinforcement learning (RL) framework to minimize costs associated with cascade incidents in such systems. The setting involves a controlled jump-diffusion process where the control action influences both the drift/volatility of the diffusion and the "readout" of the Hawkes memory (the kernel coefficients), while the kernel coefficients themselves remain unknown to the learner.

Methodology

1. Finite-Dimensional Markovianization

To overcome the non-Markovian nature of general Hawkes kernels, the authors propose a Markovianization procedure.

Approximation: They approximate the general kernel $\Phi(\tau, a)$ using a mixture of exponentials:
$\Phi(\tau, a) \approx \Phi_K(\tau, a) = \sum_{k=1}^K Q_k(a) e^{-\beta_k \tau}$
where $Q_k(a)$ are matrix coefficients (potentially with negative entries) and $\beta_k$ are decay rates.
State Lifting: This approximation allows the infinite-dimensional memory to be represented by a finite set of auxiliary state variables $Z^{K,k}_t$ , which are exponentially weighted integrals of past jumps. These variables satisfy linear jump-ODEs:
$dZ^{K,k}_t = -\beta_k Z^{K,k}_t dt + dN_t$
Online Construction: Crucially, the state vector $S^K_t = (t, X_t, Z^{K,1}_t, \dots, Z^{K,K}_t)$ can be constructed online using only observed event times and component labels, without knowledge of the underlying kernel coefficients, baseline intensity, or system dynamics. This enables a model-free approach.

2. Hawkes CT-DDPG Algorithm

The paper formulates a Continuous-Time Deep Deterministic Policy Gradient (CT-DDPG) algorithm adapted for the Markovianized Hawkes system.

Actor-Critic Architecture:
- Critic: Learns the value function $V(t, x, z)$ and an advantage-rate function $Q(t, x, z, a)$ . The training relies on the martingale property of the stochastic integral associated with the Bellman equation. The loss function minimizes the residual of the generalized moment method.
- Actor: Updates the deterministic policy $\pi_\zeta$ using the policy gradient theorem derived from the advantage rate.
Advantage Rate: The algorithm utilizes an advantage rate function defined as:
$A^\zeta_K(t, x, z, a) = \mathcal{L}^a_K J^\zeta_K - \rho J^\zeta_K + c(t, x, a)$
where $\mathcal{L}^a_K$ is the extended generator of the Markovianized system.
Training: The method employs a replay buffer to store transitions $(S^K_t, a_t, \text{cost}, S^K_{t+1})$ and uses multi-step temporal difference (TD) targets to update the networks.

Key Contributions

Convergence Theory: The authors prove that the Markovianized approximation of the Hawkes process, its intensity, and the value function converge to the original non-Markovian system and the primal problem value as the number of exponential filters $K \to \infty$ . This provides a theoretical foundation for replacing the infinite-dimensional memory with a finite-dimensional state.
Model-Free Continuous-Time RL: The paper introduces Hawkes CT-DDPG, a model-free algorithm capable of solving controlled Hawkes jump-diffusions. It operates by observing only event times, the realization of the SDE, and a chosen set of decay filters, without requiring knowledge of the kernel coefficients or the system's drift/diffusion parameters.
Handling Non-Markovian Kernels: The framework successfully extends continuous-time RL to systems with complex, non-Markovian kernels (specifically Erlang and power-law kernels) by lifting the state space, a capability not present in standard discrete-time RL methods applied directly to raw event data.

Numerical Results

The authors evaluate Hawkes CT-DDPG against three types of kernels: simple exponential, Erlang, and power-law.

Benchmarks:
- For the exponential and Erlang kernels, "Oracle" benchmarks were constructed using the Deep Galerkin Method (DGM) to solve the exact finite-dimensional HJB equations (since these kernels admit exact Markovian representations).
- For the power-law kernel (which has no exact finite-dimensional representation), a "Mixture DGM/HJB" benchmark was used, based on approximating the kernel with a finite exponential mixture.
Comparisons: The proposed method was compared against:
- A validation-selected static policy.
- Discrete-time RL methods: Soft Actor-Critic (SAC) and Deep Deterministic Policy Gradient (DDPG).
- Ablation studies comparing the "filtered" (Markovianized) approach against "current-state only" (no memory) approaches.
Findings:
- Performance: Hawkes CT-DDPG consistently outperformed discrete-time RL methods (SAC and DDPG) and static policies across all kernel types.
- Oracle Proximity: In all scenarios, Hawkes CT-DDPG remained the closest approximator to the respective Oracle benchmarks.
- Impact of Markovianization: The inclusion of the exponential filter bank (Markovianization) significantly reduced costs compared to using only the current physical state. For the Erlang kernel, filtering reduced costs by 13.4% for CT-DDPG; for the power-law kernel, the reduction was 4.67%.
- Robustness: The method demonstrated effectiveness even when the underlying kernel parameters and system dynamics were completely unknown to the agent.

Significance and Claims

The paper claims to provide a rigorous theoretical and practical framework for controlling self-exciting jump processes in continuous time. Its primary significance lies in bridging the gap between the non-Markovian nature of Hawkes processes and the requirements of modern continuous-time reinforcement learning.

The authors assert that their method:

Successfully extends the findings of continuous-time RL (specifically CT-DDPG) to Hawkes-driven diffusions.
Offers a viable solution for controlling cyber-risk and cascade incidents where attacks cluster and memory effects are critical, and where system parameters are unknown.
Demonstrates that finite-dimensional Markovianization via exponential mixtures is a robust strategy for approximating general Hawkes kernels, enabling the use of standard actor-critic architectures in continuous time.

The work does not claim to solve the general existence theory for nonlinear closed-loop Hawkes systems but focuses specifically on the approximation and control aspects. It positions the Hawkes CT-DDPG as a superior alternative to discrete-time methods for problems characterized by continuous-time event arrivals and memory dependence.