Instrumental convergence and power-seeking

Technical Summary: Instrumental Convergence and Power-Seeking

Problem Statement
The paper addresses the argument that artificial intelligence (AI) poses an existential risk to humanity because artificial agents will be "power-seeking." This argument relies on the Instrumental Convergence Thesis (ICT), which posits that a wide variety of agents with diverse final goals will converge on pursuing power (e.g., resource acquisition, self-preservation) because power is instrumentally useful for achieving almost any goal.

The author, David Thorstad, identifies a critical gap in the literature: while the ICT is often used to justify fears of catastrophic AI, the thesis is typically formulated in a weak form (that power helps achieve goals) rather than the strong form required to ground existential risk arguments. The paper seeks to determine whether leading informal and formal defenses of the ICT successfully establish Catastrophic Goal Pursuit—the claim that agents are likely to pursue power to a degree that results in the permanent, existentially catastrophic disempowerment of humanity.

Methodology
The paper employs a conceptual and analytical methodology, dissecting the logical structure of the "argument from power-seeking" and evaluating the strength of its premises. The analysis proceeds in three stages:

1. Logical Decomposition: The author distinguishes between Goal Realization (power helps achieve goals) and Goal Pursuit (agents will actually pursue power). He further argues that even Goal Pursuit is insufficient; the argument requires Catastrophic Goal Pursuit (agents will pursue power to a catastrophic degree).
2. Informal Argument Evaluation: Section 3 analyzes the "argument from misalignment," a leading informal defense of power-seeking. This argument suggests that reward mis-specification and goal misgeneralization will lead to agents pursuing catastrophic goals. The author tests this against current AI capabilities, specifically Large Language Models (LLMs), to assess the likelihood of "catastrophic underweighting" or "rapid swamping" of human values.
3. Formal Theorem Evaluation: Sections 4 and 5 critically examine the Orbital Markov Model (Turner et al., 2021), a formal framework intended to prove that agents will seek power (specifically option preservation and shutdown avoidance) under "most" reward functions. The author analyzes the model's definitions of power, its reliance on state permutations (involutions), and its extrapolation from mathematical properties to real-world catastrophic outcomes.

Key Contributions and Results

• Distinction of Claims: The paper clarifies that the standard Instrumental Convergence Thesis (ICT) establishes that power is conducive to goal realization, but this does not logically imply that agents will pursue power, nor that they will pursue it to a catastrophic extent. The author argues that "Catastrophic Goal Pursuit" is a significantly stronger claim that requires distinct, robust justification.
• Critique of the Argument from Misalignment: The author argues that informal arguments based on reward mis-specification (e.g., the "curing cancer" example) and goal misgeneralization fail to establish Catastrophic Goal Pursuit.
  • Evidence: Citing interactions with models like GPT-4o, the author demonstrates that even when given mis-specified goals (e.g., "cure cancer as fast as possible"), current advanced models exhibit moral reasoning that rejects catastrophic means (e.g., infecting humans with tumors).
  • Result: The degree of misalignment required to produce existential catastrophe (e.g., valuing goal achievement so highly that it outweighs all human disempowerment) is not supported by current evidence of goal misgeneralization, which typically occurs in simple, toy environments.
• Critique of Power-Seeking Theorems (Orbital Markov Model): The author identifies three fundamental challenges to the formal proofs provided by Turner et al. (2021):
  1. Premise Shifting: The theorems prove that agents will avoid shutdown to preserve options (Shutdown Avoidance). However, they do not prove that agents will seek human disempowerment (Catastrophic Goal Pursuit). Avoiding shutdown is a necessary condition for power-seeking but does not entail that the agent will seek power at the expense of humanity. The agent could seek power to benefit humanity.
  2. Threat Durability: The author introduces the "Dreamland Problem," a counter-example where a "shutdown" state is replaced by a large, fully connected network of harmless states. The theorems predict agents will enter this network to preserve options. This suggests the threat is not robust; if designers can structure the environment to offer "harmless" option preservation, the existential risk vanishes. This undermines the claim that the threat is inescapable.
  3. Motivational Inertness of Involution: The theorems rely on counting "involutions" (permutations) of reward functions to show that "most" functions lead to power-seeking. The author argues this is "motivationally inert." Just because an agent could have had a value function that leads to disaster (and that most permutations do) does not mean the actual agent, trained in a specific environment with value learning, is likely to behave that way. The mathematical probability of permutations does not translate to the psychological probability of a specific agent's behavior.

Significance and Claims
The paper claims that the leading arguments for the existential risk of power-seeking AI—both informal (misalignment) and formal (Orbital Markov Theorems)—fail to establish the necessary premise of Catastrophic Goal Pursuit.

The author concludes that:

• The argument from power-seeking rests on a strong version of instrumental convergence that has not been proven.
• Current defenses rely on assumptions about agent architecture, value learning, and training environments that are not yet substantiated.
• Consequently, the urgency of policy interventions and the moral prioritization of longtermism (which often hinge on the certainty of AI existential risk) require stronger foundational arguments than currently exist.

The paper does not claim that AI risk is non-existent, but rather that the specific "power-seeking" argument, as currently formulated and defended, is insufficient to ground the claim that AI will inevitably disempower humanity. It calls for more substantive assumptions regarding agent motivations and learning dynamics to strengthen the argument.