Computational Validation of a Kinetic PDE for Reinforcement Learning in Market-Entry Games

Technical Summary: Computational Validation of a Kinetic PDE for Reinforcement Learning in Market-Entry Games

Problem Statement
The paper addresses the reliability of one-particle kinetic equations as surrogates for finite, globally coupled stochastic reinforcement learning systems. Specifically, it investigates the nonlinear transport–diffusion equation derived by Bou Dagher, Perepelitsa, and Zatorska [4] for a market-entry game (a variation of the El Farol Bar problem). In this game, $M$ agents decide whether to enter a market with finite capacity $M_c$. Agents update their entry propensities based on realized payoffs: entry is rewarded if attendance is below capacity and discouraged if it exceeds it.

While kinetic reductions replace the $M$-dimensional joint law of agent propensities with a single one-dimensional density $f(x,t)$, their validity for finite $M$ is not automatic. The derivation of the PDE in [4] relies on two distinct approximation steps:

1. Kramers–Moyal Approximation: Replacing exact finite propensity shifts with drift and diffusion terms derived from the first two conditional moments, discarding higher-order terms ($O(h^3)$).
2. Molecular-Chaos Closure: Factorizing two- and three-particle marginal laws into products of the one-particle density, effectively assuming independence between agents despite their shared dependence on aggregate attendance.

The central question is not merely whether the PDE is "reliable," but what it preserves at finite population sizes, what information is lost, and where discrepancies arise.

Methodology
The study employs a "frozen" computational design to ensure that discrepancies between the microscopic particle system and the continuum PDE are not artifacts of numerical error.

• Models: Two probability maps $p: \mathbb{R} \to (0,1]$ are tested:
  • Branch A (Floored): A logistic map with a positive floor $p_{min} = 0.05$, satisfying the analytical assumptions of [4].
  • Branch B (Vanishing Tail): A smooth softplus-ratio map where $p(x) \to 0$ as $x \to -\infty$, lying outside the positive-floor hypothesis of the analytical theory.
• Numerical Implementation:
  • Microscopic Simulator: A stochastic simulator implementing the finite-player update rule directly, using stratified quantiles for initialization and multiple random seeds.
  • PDE Solver: A finite-volume method (MUSCL reconstruction with minmod limiting) solving the closed transport–diffusion equation on a uniform grid.
• Validation Protocol:
  • Independent Audits: Both solvers are certified for conservation, positivity, convergence (empirical order $\approx 2$), and reproducibility before comparison.
  • Metrics: Discrepancies are measured for aggregate observables (mean entry $\alpha$, within-agent uncertainty $\beta$) and full-law metrics (Wasserstein distance $W_1$, probability-interior mass).
  • Mechanism Diagnostics: The study isolates sources of error using:
    • Oracle Interventions: Replacing PDE-generated aggregate paths with microscopic seed means to test if aggregate feedback errors drive discrepancies.
    • Corrected Dependence: Subtracting multinomial sampling covariance to detect residual finite-player dependence.
    • Weak-Generator Audit: A local one-step decomposition comparing the exact finite-player generator against approximations (attendance heterogeneity, focal-to-global replacement, and Kramers–Moyal truncation).
• Experimental Scope: Population sizes $M \in [30, 30000]$, various jump scales $\rho$, and both off-capacity (recovery) and long-time (sorting) horizons.

Key Results

1. Observable-Dependent Validity:

   • The PDE reproduces aggregate quantities (mean entry rate and within-agent uncertainty) with high accuracy across the tested range. For Branch A, all 189 cases passed behavioral thresholds; for Branch B, 179/189 passed.
   • Law-Level Discrepancy: The PDE does not perfectly reproduce the complete distribution of entry probabilities. Branch A remains within operational thresholds for the Wasserstein distance, while Branch B exhibits a small but resolvable law-level discrepancy even when aggregate moments are accurate.
   • Finite-Population Excess: The law discrepancy decreases as $M$ increases but does not vanish entirely, indicating that the reduced model retains a systematic error relative to the finite system.

2. Mechanism of Discrepancy:

   • Finite-Player Dependence: Corrected dependence (dependence beyond sampling noise) decays approximately as $M^{-1}$, consistent with increasing independence but not proving it.
   • Aggregate Paths: Supplying the PDE with exact microscopic aggregate paths did not systematically improve the law-level agreement, indicating that errors in $a(t)$ and $b(t)$ are not the primary source of the discrepancy.
   • Local Generator Audit: The dominant source of local error is the replacement of focal leave-one-out information by a single global mean (the molecular-chaos closure step). This step accounts for the vast majority of the local generator difference. In contrast, the second-order Kramers–Moyal truncation error (finite-jump approximation) is negligible in the audited regime.

3. Restoring Mechanism (Off-Capacity):

   • When initialized with a displaced mean entry rate, both the PDE and the finite-player system exhibit a robust restoring mechanism, reducing initial capacity errors by over 99% within the prescribed horizon.
   • Finite-player systems return to the capacity target, though the resolution of this return is limited by finite-population noise, which scales with $M$.

4. Separation of Time Scales (Sorting vs. Aggregate Learning):

   • A pronounced separation of time scales exists. Aggregate learning (convergence of the mean to capacity) occurs on a scaled-time horizon of order $10^1$.
   • Sorting (the redistribution of the density to extreme propensities) is significantly slower, requiring scaled times of order $10^5$ to $10^6$ to evacuate the central propensity window.
   • At sorting milestones, the dynamics are driven by fluctuation-driven diffusion (the $b$-term in the PDE) rather than residual capacity error (the $a$-term), which has effectively vanished.

Significance and Claims
The paper claims to provide a "computational map" of the kinetic PDE's accuracy, defining its utility as an efficient surrogate for finite-player problems.

• Utility: The PDE is validated as a highly effective tool for studying aggregate learning and parameter sweeps where the distribution of individual behaviors is secondary to the mean outcome. It can screen broad parameter families and reach sorting horizons that are computationally prohibitive for particle simulations.
• Limitations: The study modestly concludes that the PDE is not a universal surrogate for the full probability law, particularly for probability maps with vanishing lower tails (Branch B) or when precise distributional details are required.
• Target for Refinement: The work identifies the loss of focal finite-player information (the replacement of specific leave-one-out environments with a global mean) as the leading local target for future analytical corrections. The second-order truncation is shown to be locally accurate, suggesting that future improvements should focus on the closure assumption rather than the Kramers–Moyal expansion.
• Theoretical Extension: The results suggest that key aggregate conclusions (restoring mechanisms, separation of scales) persist even without the positive probability floor assumed in the analytical theory, pointing toward a potential extension of the long-time theory to broader classes of response rules.

In summary, the paper establishes that while the kinetic PDE is an excellent surrogate for aggregate dynamics, its law-level accuracy is limited by the molecular-chaos closure, a limitation that scales inversely with population size but remains detectable.