Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games

1. Problem Statement

The intersection of Mean Field Games (MFGs) and Reinforcement Learning (RL) has produced numerous algorithms for solving large-scale multi-agent systems. However, the field suffers from a critical lack of standardized evaluation protocols.

• Fragmentation: Researchers currently rely on bespoke, isolated, and often overly simplistic environments (e.g., simple Linear-Quadratic models or small grid worlds).
• Lack of Generalization: This fragmentation makes it difficult to assess the robustness, generalization capabilities, and failure modes of new algorithms. An algorithm might succeed in a monotonic setting but fail catastrophically in games with cyclic dynamics or complex state-population coupling.
• Goal: The authors propose Bench-MFG, a comprehensive benchmark suite designed to standardize the evaluation of learning algorithms in discrete-time, discrete-space, stationary MFGs.

2. Methodology and Framework

The paper introduces a structured framework comprising a taxonomy of problem classes, a procedural generation method, and a suite of benchmarked algorithms.

A. Taxonomy of MFG Classes

The authors categorize MFGs into distinct classes based on their mathematical structure, providing prototypical environments for each to isolate specific algorithmic challenges:

1. No-Interaction (NI-MFG): The transition and reward are independent of the population distribution ($\mu$). Solving this reduces to a standard Markov Decision Process (MDP).
   • Example: "Move Forward" (1D grid world).
2. Contractive Games (C-MFG): The feedback loop between the population distribution and the best response is a contraction mapping, guaranteeing a unique equilibrium.
   • Example: "Coordination Game" (agents pay a high cost to switch states, leading to a trivial fixed point).
3. Lasry-Lions Monotone Games (LL-MFG): The reward discourages agents from crowded states (negative interaction). This class often guarantees uniqueness and convergence.
   • Example: "Beach Bar Problem" (agents want to reach a center but are penalized for congestion).
   • Counter-Example: "Two Beach Bars" (agents prefer crowded states, violating monotonicity and leading to multiple equilibria).
4. Potential Games (P-MFG): The reward function can be derived from a global potential function.
   • Example: "4-Rooms Exploration" (agents avoid crowded areas).
   • Counter-Example: "Rock-Paper-Scissors" (cyclic dynamics with a skew-symmetric interaction matrix, non-potential).
5. Dynamics-Coupled Games (DC-MFG): The transition probability kernel $p$ explicitly depends on the population distribution $\mu$, making the MDP dynamics time-varying as the population evolves.
   • Example: "SIS Epidemic" (infection probability depends on the density of infected agents).
   • Example: "Kinetic Congestion" (movement success depends on target state density).

B. Procedural Generation: MF-Garnets

To move beyond fixed examples and test statistical robustness, the authors introduce MF-Garnets.

• Concept: An extension of the "Garnet" benchmark for MDPs, adapted for MFGs.
• Mechanism: It generates random MFG instances by parameterizing transition probabilities and reward functions.
• Coupling Types: It supports both Additive (mean field adds bias) and Multiplicative (mean field gates/scales) interactions for both dynamics and rewards.
• Utility: Allows for rigorous stress-testing of solvers across a distribution of problem instances rather than single fixed points.

C. Algorithm Suite

The benchmark evaluates a wide range of algorithms, including:

• Fixed Point Methods: Fixed Point (FP), Damped FP, Fictitious Play (FP).
• Policy Evaluation Methods: Policy Iteration (PI), Smoothed PI, Boltzmann PI.
• Optimization Methods: Online Mirror Descent (OMD).
• Novel Contribution: MF-PSO, a black-box approach adapting Particle Swarm Optimization (PSO) to directly minimize the exploitability of a policy (the gap between the current policy's value and the best response).

D. Implementation

• Built using JAX for Just-In-Time (JIT) compilation, offering a 2000x speedup over classical Python implementations.
• Includes a pure Python version for reproducibility.
• Uses Hydra for configuration management.

3. Key Contributions

1. Unified Benchmark Suite (Bench-MFG): The first standardized suite for stationary MFGs covering diverse interaction types (no-interaction, monotone, potential, dynamics-coupled).
2. Taxonomy and Prototypes: A formal classification of MFGs with open-source, reproducible environments that isolate specific mathematical challenges (e.g., cyclic dynamics vs. congestion).
3. MF-Garnets: A novel generator for random MFG instances, enabling statistical significance testing and moving beyond "toy" examples.
4. MF-PSO Algorithm: A new black-box method for exploitability minimization that treats the MFG solver as an optimization problem over policy parameters.
5. Systematic Evaluation & Guidelines: Extensive empirical results leading to a set of best practices for future MFG research.

4. Experimental Results

The authors benchmarked the algorithms across the proposed environments and MF-Garnet instances. Key findings include:

• Algorithm Performance:
  • Fictitious Play (FP): Demonstrated surprising robustness, converging even in non-monotone settings where theoretical guarantees are absent.
  • Online Mirror Descent (OMD): Typically achieved the lowest exploitability, particularly in cyclic games (e.g., Rock-Paper-Scissors), but was highly sensitive to hyperparameter tuning.
  • MF-PSO: Achieved superior performance compared to Fictitious Play in many scenarios, though at a higher computational cost that scales linearly with the number of particles.
  • Fixed Point (FP) Baseline: In simple contractive or no-interaction games, simple FP often solves the problem in very few iterations, suggesting that complex Deep RL methods may be unnecessary for these regimes.
• MF-Garnet Results:
  • Performance varied significantly based on the game structure (additive vs. multiplicative) and dimensionality.
  • No single algorithm dominated all categories; the choice of solver depends heavily on the specific interaction type and problem complexity.
• Hyperparameter Sensitivity: The study highlighted that algorithm performance is highly sensitive to parameters (e.g., learning rates, damping coefficients), reinforcing the need for rigorous hyperparameter sweeps in future research.

5. Significance and Guidelines

The paper concludes by synthesizing findings into guidelines to foster reproducibility and rigor in MFG research:

1. Sanity Check: Always compare new algorithms against simple Fixed Point methods. If FP solves the problem quickly, complex methods may be over-engineered.
2. Baseline Comparison: New algorithms should be compared against efficient baselines like Fictitious Play, OMD, and MF-PSO.
3. Regime Testing: Do not rely on a single game type. Robustness must be demonstrated across different MFG classes (Monotone, Cyclic, Dynamics-Coupled) and MF-Garnet instances.
4. Parameter Sensitivity: Researchers must test different parameter configurations within a single environment, as small changes can drastically alter difficulty.
5. Open Source: Emphasizes the necessity of open-sourcing code (specifically efficient JAX implementations) to allow for exhaustive sweeps and reproducibility.

Future Work: The authors plan to extend the benchmark to Finite Horizon MFGs, investigate large-scale environments for scalability, and incorporate sample-based Deep RL agents to address exploration and variance challenges.

Conclusion

Bench-MFG fills a critical gap in the Mean Field Game literature by providing a standardized, diverse, and computationally efficient testing ground. By moving beyond isolated examples to a structured taxonomy and procedural generation, it enables researchers to rigorously evaluate the robustness and generalization of learning algorithms, ultimately accelerating the development of scalable multi-agent systems.