Are we Doomed to an AI Race? Why Self-Interest Could Drive Countries Towards a Moratorium on Superintelligence

Technical Summary: "Are we Doomed to an AI Race? Why Self-Interest Could Drive Countries Towards a Moratorium on Superintelligence"

Problem Statement
The prevailing view in the discourse surrounding Artificial Superintelligence (ASI) is that a global moratorium on its development is unfeasible due to strategic incentives. Critics argue that even if mutual restraint is collectively optimal, individual states possess a dominant strategy to defect and continue racing for geopolitical supremacy. This "race to the bottom" dynamic suggests that the fear of falling behind (the "sucker's payoff") outweighs the shared risk of catastrophic loss of control, rendering a moratorium contrary to a state's self-interest. This paper challenges that assumption, questioning whether a moratorium can ever be a rational, self-interested strategy for geopolitical superpowers.

Methodology
The authors employ a formal game-theoretic framework to model the strategic interaction between two geopolitical superpowers (a "Frontrunner" and a "Laggard"). The model operationalizes the decision to either "Race" (continue development with standard safety measures) or "Pause" (impose a moratorium to maximize safety).

The model is defined by four key variables:

1. Capability Gap ($\Delta$): The relative advantage one player holds in developing ASI, aggregating factors like computational power and research progress.
2. Winner's Advantage ($W$): The value of the first-mover advantage, ranging from 0 (cooperative) to 1 (winner-takes-all), capturing the geopolitical subordination of the loser.
3. Cost of Uncontrolled ASI ($C$): The disutility arising from a loss of control, normalized relative to the benefit of a safe win (normalized to 1). Crucially, the model allows $C > 1$, reflecting scenarios where civilizational collapse outweighs any strategic gain.
4. Technological Uncertainty ($\sigma$): The epistemic uncertainty regarding the possibility and timeline of ASI development.

The policy choice is binary: Pause ($s=1$, maximum safety) or Race ($s=0.85$, representing a 15% catastrophic risk based on expert surveys). The analysis identifies Nash equilibria by mapping the "Catastrophic Cost" ($C$) against the "Capability Gap" ($\Delta$) to determine under which conditions a pause becomes the rational best response for a state.

Key Contributions

1. Formalization of the Moratorium as a Self-Interested Strategy: Unlike previous models that often treat the race as a Prisoner's Dilemma where defection is dominant, this paper demonstrates that under specific parameter conditions, a moratorium is not merely a cooperative ideal but a rational strategy driven by self-interest.
2. Identification of Four Strategic Worlds: By varying the cost of catastrophe relative to the capability gap and winner's advantage, the authors delineate four distinct strategic environments:
   • Safe Harmony: Both states find it rational to pause because the cost of catastrophe is so high that racing offers no marginal gain.
   • Preemption: The unique stable outcome is a race. The fear of the rival's first-mover advantage outweighs the fear of catastrophe, particularly when states are near parity.
   • Trust: A coordination game where both prefer a mutual pause but fear being exploited if the rival races.
   • Subversion: An asymmetric world where the Frontrunner rationally chooses to pause (due to high probability of winning and fear of catastrophe), while the Laggard races to close the gap.
3. Quantification of Thresholds: The model quantifies the specific "cost-thresholds" where the strategic space shifts from racing to pausing. It shows that as the perceived cost of loss of control ($C$) increases sufficiently relative to the winner's advantage ($W$) and uncertainty ($\sigma$), the rational strategy shifts toward a moratorium.

Results
The analysis reveals that the feasibility of a moratorium is not static but depends on the interplay of variables:

• The Role of Catastrophic Cost: When the perceived cost of an uncontrolled ASI exceeds the expected benefits of geopolitical supremacy, a stable moratorium becomes a self-interested choice.
• Impact of Uncertainty: Moderate technological uncertainty ($\sigma$) expands the strategic space for a moratorium. When states are uncertain about the possibility of winning the race, the incentive to cut corners diminishes, making a pause more rational.
• Impact of Winner's Advantage: A reduction in the winner's advantage ($W$) lowers the threshold for the cost of loss of control required to make a moratorium rational.
• Empirical Trajectory: The paper presents empirical evidence suggesting that the global perception of ASI risk is rising. Citing expert statements (e.g., the 2023 Open Letter, the Bletchley Declaration), public opinion polls (e.g., Reuters/Ipsos), and the establishment of AI Safety Institutes, the authors argue that the perceived cost of loss of control ($C$) is increasing. This shift suggests that the conditions required for a moratorium to be in a state's self-interest are becoming more plausible in the current geopolitical landscape.

Significance and Claims
The paper claims to overturn the assumption that a moratorium is inherently contrary to a state's self-interest. It argues that the "race" is not an inevitable outcome of rational state behavior but a contingent result of specific parameter values—specifically, a low perceived cost of catastrophe relative to the benefits of supremacy.

The authors modestly conclude that while the current geopolitical reality is still dominated by competitive dynamics, the strategic space for a stable, rational moratorium is expanding. They posit that as the scientific consensus on control problems grows and the perceived severity of catastrophic outcomes rises, the conditions under which a moratorium becomes the dominant strategy for self-interested states are "closer than commonly assumed." The paper does not claim a moratorium is guaranteed, but rather that it is a viable, rational option under evolving conditions, challenging the fatalistic view that an AI race is inevitable.