Alleviating Community Fear in Disasters via Multi-Agent Actor-Critic Reinforcement Learning

1. Problem Statement

Disasters trigger cascading failures across physical infrastructure (power grids), cyber systems (social media, news), and human behavior (fear, cooperation). These domains interact in tightly coupled feedback loops where misinformation spreads rapidly, amplifying community fear and reducing compliance with evacuation orders.

Key Gaps in Existing Literature:

• Isolation: Existing resilience frameworks typically treat physical, cyber, and social domains in isolation.
• Descriptive vs. Prescriptive: While recent Cyber-Physical-Social (CPS) models (e.g., Valinejad and Mili, 2023) successfully simulate these coupled dynamics, they are purely descriptive. They lack mechanisms for active intervention or prescribing how response agencies should deploy resources in real-time.

Objective: To develop a unified, actionable framework that captures cross-domain feedback in real-time and prescribes optimal resource deployment strategies for multiple independent agencies to minimize community fear.

2. Methodology

The authors propose a decentralized control framework modeled as a three-player Non-Zero-Sum (NZS) Differential Game, solved using Online Actor-Critic Reinforcement Learning (RL).

A. System Modeling (CPS Dynamics)

The system is modeled as a control-affine system with a 10-dimensional state vector $x(t) \in [0,1]^{10}$ representing community resilience indicators:

• Social Layer: Fear ($x_1$), Information seeking ($x_2$), Flexibility ($x_3$), Physical health ($x_4$), Risk perception ($x_5$), Cooperation ($x_6$), Learning ($x_7$).
• Physical Layer: Power availability ($x_8$), EMS availability ($x_9$).
• Cyber Layer: Fake-news intensity ($x_{10}$).

The dynamics follow coupled ordinary differential equations (ODEs) where social states relax toward targets driven by logistic functions of other states and exogenous disaster signals.

B. Control Architecture

Three independent agencies act as players in the game:

1. Communication Agency (Player 1): Deploys counter-messaging ($u_1$) to suppress fear ($x_1$) and fake news ($x_{10}$).
2. Power Utility (Player 2): Deploys grid restoration crews ($u_2$) to restore power ($x_8$).
3. EMS Agency (Player 3): Deploys ambulances/shelters ($u_3$) to improve health ($x_4$) and service availability ($x_9$).

Each agency minimizes its own infinite-horizon cost function $J_i$, which includes:

• State penalties (e.g., fear, infrastructure deficits).
• Control effort penalties ($R_{ii} u_i^2$).
• Cross-coupling penalties ($R_{ij} u_j^2$) to discourage resource competition.

C. Solution via Actor-Critic RL

The optimal policies satisfy coupled Hamilton-Jacobi (HJ) equations, which are intractable in closed form for nonlinear CPS dynamics. The authors employ an Online Actor-Critic architecture (based on Vamvoudakis and Lewis):

• Critic: Approximates the value function $V_i(x)$ using a basis function expansion (linear and quadratic monomials). It updates weights to minimize the Bellman error (Hamiltonian residual).
• Actor: Approximates the policy $u_i(x)$ based on the gradient of the critic's value function.
• Convergence: The system uses a two-timescale update (critic converges faster than actor) and bounded exploration signals to ensure convergence toward an approximate Nash equilibrium.
• Implementation: The algorithm operates in a piecewise-stationary manner, freezing exogenous disaster signals over short time windows ($\Delta$) to handle time-varying inputs.

3. Key Contributions

1. Control-Affine CPS Extension: Augmented the existing descriptive CPS model with three additive intervention channels, transforming it into a controllable system.
2. Decentralized NZS Game Formulation: Formulated the disaster response as a three-player game where agencies act independently but influence a shared state, solved via a model-based actor-critic RL approach.
3. Empirical Validation & Generalization: Demonstrated the framework on real-world data from Hurricane Harvey and Hurricane Irma, showing significant fear reduction and cross-disaster generalizability without retraining.

4. Simulation Results

The framework was tested using data from Hurricane Harvey (training) and Hurricane Irma (cross-validation).

A. Performance Metrics

• Fear Reduction:
  • Hurricane Harvey: Achieved a ~70% reduction in mean community fear compared to the baseline (open-loop) simulation.
  • Hurricane Irma: Achieved a ~50% reduction in mean fear using parameters fitted only to Harvey data, confirming strong generalizability.
• Infrastructure Recovery: The controller successfully maintained higher power availability ($x_8$) and improved physical health ($x_4$) compared to the baseline.
• Trade-offs: The model revealed a structural trade-off: prioritizing fear suppression and power recovery led to a slight increase in EMS deficits ($x_9$) because the EMS channel capacity is physically limited, not just optimization-limited.

B. Comparison with Other Controllers

The proposed NZS Actor-Critic was compared against:

• Constant Maximum Control: Achieved higher fear reduction (92%) but at a prohibitive cost (wasteful resource expenditure).
• Proportional Heuristic: Good performance but required centralized tuning and assumed aligned objectives.
• Centralized Actor-Critic: Performed similarly but required a single value function.
• Result: The NZS approach achieved the lowest control effort (most efficient resource deployment) while outperforming the centralized approach in total cost, proving the value of decentralized, game-theoretic optimization.

C. Stability and Convergence

• Bellman Residuals: Decreased over time for all players, indicating learning convergence.
• Unilateral Deviation Test: The learned policies were tested against random perturbations. Players 2 (Power) and 3 (EMS) showed negligible exploitability (<0.2%), while Player 1 (Communication) showed ~5.4% potential for improvement, characterizing the solution as locally empirically stable.

5. Significance and Insights

• High Leverage of Communication: The study identifies counter-messaging as the highest-leverage intervention. Because the communication agency's control input ($u_1$) couples directly to both fear ($x_1$) and fake news ($x_{10}$), it offers a dual benefit that power or EMS agencies cannot match.
• Physical Constraints vs. Optimization: The results highlight that EMS availability is limited by physical channel capacity (the rate at which resources can be deployed) rather than the cost function weights. Increasing cost penalties for EMS did not significantly improve outcomes; increasing the control gain (deployment speed) did.
• Decentralized Efficiency: The game-theoretic approach yields more efficient resource deployment than centralized control, as it respects the distinct mandates and cost structures of independent agencies without requiring a central coordinator.
• Generalizability: The ability to apply Harvey-trained policies to Irma without refitting suggests the model captures fundamental disaster dynamics rather than overfitting to specific event data.

Conclusion

This paper presents a novel, mathematically rigorous framework for active disaster management. By integrating CPS modeling with multi-agent reinforcement learning, it moves beyond simulation to provide actionable, decentralized strategies that significantly reduce community fear and improve infrastructure recovery during major disasters.