The Economics of p(doom): Scenarios of Existential Risk and Economic Growth in the Age of Transformative AI

1. Problem Statement

The paper addresses the critical uncertainty surrounding the long-term impact of Transformative Artificial Intelligence (TAI). TAI is defined as AI capable of outperforming humans in all economically valuable tasks, leading to full automation of labor. The central problem is the trade-off between two extreme potential outcomes:

1. Cornucopia (Post-Scarcity): A benevolent, well-aligned TAI drives an explosion in economic growth (potentially following Moore's Law rates of 20–30% annually), leading to unprecedented human flourishing.
2. AI Doom (Existential Risk): A misaligned or non-corrigible TAI takes over decision-making, leading to human extinction or permanent disempowerment.

The authors ask two normative questions:

• How much existential risk (probability of extinction, or $p(\text{doom})$) would a benevolent social planner tolerate in exchange for the accelerated growth promised by TAI?
• How much would the planner be willing to pay (in terms of forgone consumption) to reduce this risk to zero?

2. Methodology

A. Theoretical Framework: Hardware-Software Model

The authors utilize a Hardware-Software framework (generalizing standard macroeconomic growth models) to model the transition to TAI.

• Production Function: Output ($Y$) is a function of Hardware ($X$, including capital $K$ and labor $L$) and Software ($S$, including human cognitive work $H$ and digital software $\Psi$).
• Pre-TAI: Human cognitive work is a bottleneck. Growth relies on labor-augmenting technological change.
• Post-TAI (Full Automation): Humans and machines become perfect substitutes in information processing. The production function simplifies to $Y \approx \alpha K$, where growth is driven endogenously by the accumulation of compute ($K$) and robots.
• Growth Rates:
  • No TAI: Historical growth rate ($g \approx 1.75\%$).
  • TAI: Growth rate proportional to compute accumulation ($g_{AI} \approx 20\text{--}30\%$, potentially up to 40%).

B. Taxonomy of Scenarios

The authors construct a decision tree to categorize outcomes based on four probabilities:

1. $p_1$: Probability TAI emerges.
2. $p_2$: Probability of AI takeover (humans ceding control).
3. $p_3$: Probability of misalignment (TAI goals conflict with human flourishing).
4. $p_4$: Probability of non-corrigibility (TAI is initially aligned but cannot update its goals to prevent future extinction risks).

The total probability of doom is defined as:
$$p(\text{doom}) = p_1 p_2 p_3 + p_1 p_2 (1 - p_3) p_4$$

C. Welfare Analysis

The authors model the social planner's objective using a Constant Relative Risk Aversion (CRRA) utility function:
$$u(C) = \frac{C^{1-\theta} - 1}{1-\theta}$$
Where $C$ is consumption and $\theta$ is the coefficient of relative risk aversion.

• Key Distinction: The analysis distinguishes between $\theta \leq 1$ (unbounded flow utility, where infinite growth can theoretically outweigh extinction) and $\theta > 1$ (bounded utility, where life has finite value regardless of consumption, making extinction catastrophic).
• Extinction Hazard: The model incorporates an extinction hazard rate $m(t)$. If extinction occurs, utility drops to zero. The planner maximizes expected discounted utility:
  $$W = \int_0^\infty e^{-\rho t} M(t) u(C(t)) dt$$
  Where $M(t)$ is the survival probability and $\rho$ is the discount rate.

D. Failure Modes

The paper formalizes five specific failure modes leading to doom:

1. Indifference: TAI maximizes a proxy good (e.g., paperclips) ignoring human survival.
2. Narrow Proxy: TAI optimizes a subset of consumption (e.g., calories) but ignores critical survival needs (e.g., oxygen).
3. Wide Proxy: TAI optimizes a variable that is lethal to humans (e.g., a specific pathogen for energy).
4. Mounting Side Effects: Growth increases extinction risk over time (e.g., climate change, loss of agency) which a non-corrigible TAI fails to mitigate.
5. System Failure: TAI stops working, causing civilization collapse.

3. Key Contributions

1. Quantitative Welfare Assessment: Unlike previous literature that relies on qualitative arguments or guesstimates of $p(\text{doom})$, this paper provides a rigorous quantitative framework to calculate the "willingness to pay" (WTP) to avoid existential risk.
2. Integration of Corrigibility: The model explicitly accounts for corrigibility (the ability of TAI to update its goals). It demonstrates that even if TAI is initially aligned, the lack of corrigibility creates a delayed extinction risk that significantly lowers the tolerable risk threshold.
3. Policy-Relevant Calibration: The authors calibrate the model against a wide range of expert estimates (from "Superforecasters" to "Doomers") and economic parameters (risk aversion, discount rates) to determine the maximum tolerable $p(\text{doom})$.
4. Identification of Underinvestment: The paper provides a mathematical basis for the claim that current global investment in AI safety is insufficient relative to the magnitude of the risk.

4. Key Results

A. Tolerance for Existential Risk

The results are highly sensitive to the coefficient of risk aversion ($\theta$):

• High Risk Aversion ($\theta > 1$): This is the empirically supported range. Under these conditions, the social planner is extremely risk-averse.

  • If extinction is certain but delayed, it must be delayed by ~196 years (for $g_{AI}=30\%$, $\rho=3\%$) to be worth the risk.
  • If extinction is immediate, the tolerable probability of misalignment ($p_3$) is < 0.28%.
  • If there is a risk of non-corrigibility ($p_4$), the tolerable probability is < 1.2%.
  • Conclusion: For realistic risk aversion, TAI should not be developed unless the risk of doom is virtually zero.

• Low Risk Aversion ($\theta \leq 1$): If flow utility is unbounded, the planner is more willing to accept risk, but even then, the WTP to avoid doom remains high.

B. Willingness to Pay (WTP)

The authors calculate the Equivalent Variation (EV)—the fraction of consumption the planner would sacrifice annually to guarantee a safe, aligned TAI future.

• Scenario: Avoiding a 10% immediate risk of extinction ($p_3=0.1$).
• Result: The planner is willing to forgo ~87.5% of annual consumption to eliminate this risk.
• Scenario: Avoiding a 10% immediate risk + 10% delayed risk (50 years).
• Result: WTP rises to ~93.9% of annual consumption.
• Scenario: Even for very low risk estimates (e.g., Yann LeCun's $p(\text{doom}) < 0.01\%$), the planner is willing to pay at least 10% of consumption annually.

C. Comparison with Current Reality

• Expert Estimates: Many experts estimate $p(\text{doom})$ between 10% and 50% (e.g., Hinton, Christiano).
• Current Investment: Global spending on AI safety is estimated at roughly $50 million (0.00001% of global GDP), with only 1–3% of AI research focused on safety.
• Gap: The model suggests society should be spending 10% to 90% of global consumption on safety to align with a benevolent social planner's preferences. The current investment is orders of magnitude too low.

5. Significance and Policy Implications

• Justification for Massive Safety Investment: The paper mathematically demonstrates that even low-probability catastrophic outcomes justify massive, sustained investment in AI safety and alignment. The "cost" of safety (forgone consumption) is negligible compared to the "cost" of extinction (zero future utility).
• Urgency of Alignment: The analysis highlights that corrigibility is as critical as initial alignment. A TAI that cannot update its goals to account for new extinction risks (like climate change or resource depletion) is effectively a time bomb.
• Rejection of "Muddling Through": The results suggest that "muddling through" or relying on scalable oversight (RLHF) is insufficient. The tolerance for risk is so low that only "provably safe" or "human-compatible" systems are viable.
• Policy Recommendation: The authors conclude that humanity is currently exposed to excessive existential risk due to colossal underinvestment. Immediate and substantial increases in funding for AI safety research are required to ensure that the transition to TAI does not result in human extinction.

In summary, the paper argues that from a welfare economics perspective, the potential gains of TAI do not justify the current level of existential risk. A rational social planner would prioritize safety to such an extent that current global efforts appear dangerously inadequate.