📊 statistics

Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

This paper proposes a model-free, continuous-time Actor-Critic reinforcement learning framework for extended mean field control with deterministic policies, which leverages a refined policy gradient formula involving measure derivatives to enable efficient and robust solutions for problems where dynamics and rewards depend on the joint state-action distribution.

Original authors: Ziheng Cheng, Xin Guo, Huyên Pham, Yufei Zhang

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziheng Cheng, Xin Guo, Huyên Pham, Yufei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, chaotic dance floor where thousands of dancers move in sync, but no single dancer knows the whole choreography. Each dancer only sees their own feet and the general vibe of the crowd. Now, imagine a "Central Choreographer" who wants to teach them the perfect routine to avoid collisions and move beautifully together. This is the world of Extended Mean Field Control.

In this paper, the authors (Ziheng Cheng, Xin Guo, Huyˆen Pham, and Yufei Zhang) tackle a tricky problem: How do you teach this Central Choreographer to learn the best moves when the rules of the dance floor are a mystery? The dancers' movements and the rewards they get depend not just on where they are, but on the entire mix of where everyone is and what everyone is doing.

The Old Way vs. The New Way

Previously, researchers tried to teach the Choreographer using "stochastic policies." Think of this as telling the dancers, "At this moment, there's a 30% chance you should spin left, a 40% chance you should jump, and a 30% chance you should slide." It's like rolling dice for every single move. The authors argue this is a bad idea for this specific type of dance. Why? Because when the reward depends on the group's collective action, rolling dice for everyone creates a messy, unpredictable cloud of actions that is incredibly hard to calculate and learn from. It's like trying to predict the weather by flipping a coin for every single raindrop.

Instead, the authors propose a Deterministic Policy. This is like giving the dancers a strict, clear rule: "If you are at position X and the crowd looks like Y, you must step Z." No dice, no guessing. The authors show that by sticking to these clear, direct rules, the "state-action distribution" (the map of where everyone is and what they are doing) becomes a direct, predictable reflection of the crowd's current state. It's a straight line from the rule to the result, rather than a tangled web of probabilities.

The Magic Formula: The "Advantage-Rate"

The paper's main finding is a new mathematical recipe called a deterministic policy gradient. Imagine the Choreographer is trying to improve the dance. They need to know: "If I tweak my rule just a tiny bit, how much better will the dance look?"

The authors derived a formula that answers this without needing to know the exact physics of the dance floor (the "model-free" part). They introduced a concept called the advantage-rate function. Think of this as a "scorecard" that tells the Choreographer exactly how much better a specific move is compared to the average move, considering the whole crowd's behavior.

They proved that by looking at how this scorecard changes as the crowd moves, the Choreographer can figure out the perfect direction to tweak their rules. They didn't just guess this; they proved it mathematically using a "martingale-based learning principle," which is a fancy way of saying they found a reliable, fair way to track progress over time.

The Algorithm: CT-DDPG

To put this into practice, the authors built a computer algorithm called Continuous-Time Deep Deterministic Policy Gradient (CT-DDPG).

Here's how it works in their simulation:

The Actors and Critics: They use neural networks (computer brains) to act as the "Actor" (the Choreographer making the rules) and the "Critic" (the judge scoring the dance).
The Crowd Simulation: They simulate a crowd of 50 particles (dancers) to mimic the real, infinite crowd.
Learning by Doing: The Actor tries a rule, the crowd dances, and the Critic scores it. The Critic doesn't just say "good" or "bad"; it uses the new "advantage-rate" formula to give specific feedback on how to tweak the rule.
Exploration: To avoid getting stuck in a boring routine, they add a little bit of "noise" (randomness) to the training. They tested two ways to do this:
- Action Space: Randomly nudging the dancers' moves.
- Parameter Space: Randomly tweaking the Choreographer's brain (the neural network weights).

The Results: Does it Work?

The authors didn't just write theory; they ran numerical experiments to see if it actually works. They tested it on two specific scenarios:

The Cucker-Smale Consensus: A model of birds or fish trying to flock together. They tested a case where the birds naturally flock (a "Linear-Quadratic" case) and a harder case where the interaction is complex (non-linear).
- The Finding: In the simulations, their new method (CT-DDPG) learned the optimal dance faster and more stably than older methods that relied on knowing the math beforehand. It even worked well when they used simple, generic features to understand the crowd, without needing to know the specific physics of the interaction.
- The Caveat: They found that "Action Space" exploration (nudging the dancers) was generally more robust and less sensitive to the size of the random noise than "Parameter Space" exploration.
Optimal Liquidation: A financial scenario where a trader tries to sell a huge pile of stock without crashing the price, while knowing that everyone else is also trying to sell.
- The Finding: Again, the method was efficient and robust. Interestingly, in this specific financial case, "Parameter Space" exploration (tweaking the brain) sometimes converged faster if tuned perfectly, showing that the best exploration strategy might depend on the specific problem.

The Bottom Line

The paper demonstrates (through rigorous math and computer simulations) that using clear, deterministic rules is a powerful way to teach a central planner how to manage a massive, interacting crowd. It avoids the computational nightmares of randomizing every single move.

While the authors show this works beautifully in their simulations of flocking birds and stock trading, they present this as a new, efficient framework for these specific types of problems. They don't claim it solves every control problem in the universe, but they do show it is a significant step forward for continuous-time, extended mean field problems, offering better stability and faster convergence than previous approaches that relied on stochastic (randomized) policies.

Technical Summary: Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

Problem Formulation

This paper addresses continuous-time extended Mean Field Control (MFC) problems. In this setting, a central planner optimizes the collective behavior of a large population of homogeneous, interacting agents. The controlled state of a representative agent follows a McKean-Vlasov stochastic differential equation (SDE):
$dX^\alpha_s = b(s, X^\alpha_s, \alpha_s, \mathcal{P}(X^\alpha_s, \alpha_s))ds + \sigma(s, X^\alpha_s, \alpha_s, \mathcal{P}(X^\alpha_s, \alpha_s))dW_s$
Crucially, unlike standard MFC where coefficients depend only on the state distribution, extended MFC allows the drift $b$ , diffusion $\sigma$ , and reward functions to depend on the joint distribution of the state and the control, denoted as $\mathcal{P}(X^\alpha_s, \alpha_s)$ .

The objective is to maximize a reward functional over admissible control processes. A primary challenge in this domain is the infinite-dimensional nature of the problem, where the value function depends on time, the individual state, and the state distribution. Furthermore, existing Reinforcement Learning (RL) approaches for continuous-time MFC often rely on stochastic (exploratory) policies. The authors argue that stochastic policies face significant algorithmic hurdles in extended MFC: randomizing actions alters the control distribution, breaking the linearity of the generator with respect to the policy and complicating the derivation of auxiliary dynamics. Additionally, stochastic policies require high-frequency sampling and Monte Carlo approximations over continuous action spaces, leading to computational inefficiency and instability.

Methodology

The paper proposes a model-free Actor-Critic framework utilizing deterministic feedback policies. Instead of optimizing over stochastic kernels, the method optimizes over a parameterized family of deterministic policies $\phi_\theta(t, x, \mu)$ , which map the current time, state, and state law directly to an action.

1. Theoretical Foundations

The core of the methodology is the derivation of a deterministic policy gradient (DPG) formula for continuous-time extended MFC.

Model-Free Sensitivity: The authors first establish a model-free sensitivity formula for parameterized McKean-Vlasov dynamics. They derive a gradient formula for the value function with respect to policy parameters $\theta$ without requiring explicit knowledge of the drift or diffusion coefficients.
Advantage-Rate Function: The gradient is expressed as an integral involving the derivative of an advantage-rate function along the flow of state distributions.
Local Decomposition (Theorem 3.2): A key theoretical refinement involves decomposing the lifted value and advantage-rate functions into local functions of the state, action, and joint state-action distribution. This decomposition yields a policy gradient containing:
1. A derivative with respect to the local action.
2. A measure-derivative term with respect to the control distribution.
Martingale Characterization: The authors establish a martingale condition for these local representations. This allows the value and advantage functions to be learned directly from observed state and control trajectories, bypassing the need to learn from the flow of state laws alone.

2. Algorithm: CT-DDPG

Based on the theoretical results, the paper proposes the Continuous-Time Deep Deterministic Policy Gradient (CT-DDPG) algorithm.

Architecture: The algorithm employs measure-dependent neural networks. The value function $V_\phi$ and advantage-rate function $q_\psi$ are approximated using neural networks that take as input not only the state and time but also finite-dimensional embeddings of the state distribution (e.g., moments or learnable feature maps).
Critic Update: The critics are trained using a temporal-difference (TD) learning scheme derived from the martingale characterization. The loss function enforces the Bellman-like constraints and terminal conditions.
Actor Update: The policy is updated via gradient ascent using the derived DPG formula (Eq. 3.22), which utilizes the local advantage-rate function and its derivatives.
Exploration: Since the policy is deterministic, exploration is introduced externally via two mechanisms:
1. Action-space exploration: Adding noise to the actions selected by the policy.
2. Parameter-space exploration: Perturbing the policy network parameters.
Simulation: The McKean-Vlasov dynamics are simulated using a particle approximation (Euler-Maruyama discretization) with $M$ agents to estimate the empirical distributions.

Key Contributions

Deterministic Policy Framework for Extended MFC: The paper introduces a model-free RL framework specifically for continuous-time extended MFC that avoids the limitations of stochastic policy approaches (such as the loss of linearity in the generator and high computational cost).
Novel Policy Gradient Formula: It derives a deterministic policy gradient formula that explicitly accounts for the dependence on the joint state-action distribution. This includes a novel measure-derivative term with respect to the control distribution, which is essential for extended MFC problems.
Local Advantage Representation: By decomposing the value and advantage functions into local components, the authors enable more informative learning signals derived from individual trajectories rather than just the aggregate state flow.
Algorithmic Implementation: The paper presents the CT-DDPG algorithm, integrating particle approximations, measure-dependent neural networks, and TD learning to solve these high-dimensional, infinite-dimensional control problems.

Numerical Results

The proposed method is evaluated on two benchmark problems:

Stochastic Cucker-Smale Consensus Control:
- Linear-Quadratic (LQ) Case ( $\gamma=0$ ): CT-DDPG converges faster to the optimal value than existing model-aware baselines (Actor-Critic and Q-Learning), despite not exploiting the known LQ structure.
- Non-LQ Case ( $\gamma=1$ ): The method demonstrates robustness and efficiency. Variants using learnable feature maps for distribution embedding perform comparably to those using hand-crafted kernel features, validating the model-agnostic nature of the approach.
- Exploration: Action-space exploration is found to be more robust to noise scale tuning than parameter-space exploration, though both can achieve comparable performance with careful tuning.
Optimal Liquidation with Trade Crowding:
- This problem explicitly depends on the control distribution (permanent market impact).
- CT-DDPG exhibits superior convergence speed and stability compared to baselines.
- In this specific setting, parameter-space exploration showed potential advantages over action-space exploration when tuned correctly, as action-space noise did not sufficiently perturb the price process dynamics.

Significance and Claims

The paper claims that the proposed deterministic policy approach offers significant computational advantages over stochastic-policy methods in continuous-time extended MFC settings. By avoiding optimization over stochastic kernels and directly utilizing the push-forward structure of deterministic policies, the method achieves:

Improved Stability: The deterministic nature of the policy reduces the variance associated with stochastic action sampling.
Faster Convergence: The martingale-based learning principle and local advantage representations provide more informative gradients.
Model-Free Capability: The algorithm operates without knowledge of the system dynamics ( $b, \sigma$ ) or reward functions, making it applicable to complex, real-world scenarios where such models are unknown or intractable.

The authors emphasize that their approach removes restrictive assumptions found in prior works (e.g., separable structures or known explicit dependence on controls) and provides a unified framework for learning in extended MFC problems where the control distribution plays a critical role.