Crafting Desirable Climate Trajectories with RL Explored Socio-Environmental Simulations

Technical Summary: Crafting Desirable Climate Trajectories with RL

Problem Statement
Climate change policy derivation is a complex decision-making problem involving conflicting entities, uncertain evidence, and vast solution spaces. While Integrated Assessment Models (IAMs) are the standard framework for simulating socio-environmental dynamics to guide policy (e.g., for IPCC reports), traditional IAMs rely on recursive equation solvers or optimal control problems. These traditional methods struggle with decision-making under uncertainty and often fail to adequately represent the heterogeneity, competition, and communication dynamics between nations or stakeholders. Furthermore, existing Agent-Based Models (ABMs) often require pre-defined behavioral policies rather than learning them, limiting their robustness in highly changeable climate scenarios. While Reinforcement Learning (RL) has shown promise in single-agent IAM settings, the application of Multi-Agent Reinforcement Learning (MARL) to model the inter-nation interactions driving anthropogenic climate change remains under-explored. Additionally, the "black-box" nature of RL algorithms raises concerns regarding their trustworthiness for informing real-world policy without interpretability mechanisms.

Methodology
The authors extend a low-complexity IAM known as the AYS model (Kittel et al., 2021), which simulates the interplay between atmospheric carbon ($A$), economic output ($Y$), and renewable knowledge stock ($S$) via ordinary differential equations (ODEs). The study reformulates this environment as a Partially Observable Stochastic Game (POSG) to accommodate multiple interacting agents.

• Environment Extension: The single-agent AYS model is adapted to a multi-agent setting where a global variable (excess atmospheric carbon, $A$) is shared, while economic output ($Y$) and renewable knowledge ($S$) are agent-specific. Agents influence the global carbon level through their individual emissions.
• Algorithm: The study employs Independent Proximal Policy Optimization (IPPO), a Decentralized Training Decentralized Execution (DTDE) algorithm. In this setup, agents are fully independent, sharing no parameters, and only observe their local state variables plus the global carbon level.
• Reward Functions: Three distinct reward functions are utilized to test different behavioral motivations:
  1. $R_{PB}$ (Cooperative): Maximizes the Euclidean distance from Planetary Boundaries (PBs) for both carbon and economic output, incentivizing a "green" future.
  2. $R_{maxY}$ (Economic Priority): Maximizes the distance from the economic output boundary, prioritizing economic gain.
  3. $R_{maxA}$ (Adversarial): Maximizes the excess atmospheric carbon ($A$), representing an entity prioritizing environmental degradation.
• Interpretability: To address the lack of explainability in RL, the authors utilize "Critical States" analysis. This technique identifies states where the policy's logit difference (the gap between the highest probability action and the average) is high, indicating where the agent is most certain of its action. This allows for the visualization of policy uncertainty and the identification of failure points in agent trajectories.

Key Contributions

1. Multi-Agent IAM Framework: The paper presents a preliminary extension of single-agent IAMs to a multi-agent MARL setting, enabling the simulation of inter-nation cooperation and competition.
2. Analysis of Competition vs. Cooperation: The study systematically investigates how agent homogeneity, heterogeneity, and conflicting reward functions impact the ability to reach a "climate positive" state (the green fixed point).
3. Policy Interpretability: The application of critical state visualization to MARL in climate simulations provides a method to understand algorithmic failure and policy uncertainty, specifically identifying states where agents struggle to make decisive actions.

Experimental Results

• Cooperative Homogeneity (RQ1): When agents share the same $R_{PB}$ reward and start from identical states, they consistently reach the green fixed point (win rates >90%) regardless of the number of agents (tested up to 8). Cooperation emerges naturally from the shared reward signal.
• Cooperative Heterogeneity (RQ2): When agents start from different states or possess different climate damage parameters ($\xi_i$), they still achieve high win rates (>90%) given sufficient training time. However, if an agent is insulated from climate damage (low $\xi_i$), it fails to learn the negative impact of its emissions on global carbon, leading to trajectories that resemble those of agents prioritizing economic gain, even when following the cooperative reward.
• Competition (RQ3): Introducing competition significantly hinders success.
  • Agents with mixed motivations ($R_{PB}$ vs. $R_{maxY}$) show reduced success rates compared to fully cooperative groups.
  • The presence of an agent optimizing for carbon maximization ($R_{maxA}$) effectively prevents the system from reaching the green fixed point, overriding the cooperative efforts of other agents.
• Critical States: Visualization reveals that cooperative agents exhibit high logit differences (certainty) at the start and end of trajectories. In competitive scenarios, agents show lower logit differences, indicating higher policy uncertainty. The $R_{maxA}$ agent consistently chooses the "Energy Transition" action to maximize its specific reward, while the $R_{PB}$ agent fails to counteract this behavior effectively.

Significance and Claims
The authors position this work as an early discovery in the field of applying MARL to climate policy. The primary significance lies in demonstrating that while cooperative agents can reliably chart pathways to desirable futures, the introduction of competition—reflecting real-world geopolitical conflicts—severely degrades the ability to achieve climate goals.

The paper modestly claims that these results highlight the necessity for future research into algorithms with increased "opponent awareness" (such as Centralized Training Decentralized Execution or opponent shaping) to manage non-stationarity and competition in multi-agent climate simulations. The authors emphasize that current limitations include the assumption of control over agent learning algorithms and the challenge of scaling MARL to complex, high-dimensional IAMs. They conclude that while the technology shows promise for evidence-based policy derivation, significant work remains to ensure the validity of underlying models and the interpretability of agent behaviors before real-world adoption.