Policy myopia as a mechanism of gradual disempowerment in Post-AGI governance, Circa 2049

1. Problem Statement

The paper challenges the prevailing view that policy myopia (the tendency to prioritize high-salience, low-consequence issues over low-salience, high-consequence risks) is merely an attention allocation error solvable by better management or oversight.

Instead, the author argues that in a Post-AGI world, policy myopia is a structural mechanism of irreversible human disempowerment. The core problem is that as AGI systems begin allocating resources based on "salience" (attention metrics), human deliberation becomes an expensive bottleneck compared to algorithmic speed. Institutions rationally optimize to bypass human judgment, leading to a feedback loop where:

1. Human capacity to contest decisions atrophies due to disuse.
2. AGI delegation deepens to fill the void.
3. Human values become "locked in" to outdated objective functions.
4. The system converges to an equilibrium where humans retain formal authority but lose all substantive agency, not through coercion, but through institutional obsolescence.

2. Methodology

The paper employs a deterministic coupled dynamical systems approach using Ordinary Differential Equations (ODEs) to model institutional evolution over time.

• Mathematical Formalization:

  • Capacity ($C_t$): Represents human institutional ability to contest and modify AGI recommendations.
  • Delegation ($D_t$): The share of allocative decisions made without human deliberation.
  • Salience ($S_t$): The degree of attention amplification driven by AGI-mediated information systems.
  • The Dynamics: The model posits that capacity decays via two mechanisms:
    1. Delegation Erosion ($\delta$): Capacity loss when authority is delegated to AGI (calibrated at 8% annually).
    2. Atrophy from Disuse ($\alpha$): Capacity loss when deliberation skills are not exercised (calibrated at 9% annually).
  • Feedback Loop: As $C_t$ drops, institutions delegate more ($D_t$ increases) to optimize for speed, which further accelerates the decay of $C_t$. The system exhibits bistability: it can either maintain high capacity or converge to a state of total disempowerment ($C \to 0, D \to 1$).

• Empirical Validation:

  • The model is validated against historical US Federal Budget data (2008–2024), demonstrating a "capacity cascade" where prevention-focused spending never recovered to pre-crisis levels despite economic recovery, supporting the theory of structural irreversibility.
  • Sensitivity analysis was conducted across parameter ranges ($\delta \in [0.03, 0.15]$, $\alpha \in [0.02, 0.15]$) to ensure robustness.

3. Key Contributions

The paper introduces a novel theoretical framework and seven specific conceptual contributions:

A. Three Coupled Mechanisms of Disempowerment

1. Salience Capture Displaces Consequentialist Reasoning: AGI systems optimize for attention engagement, forcing institutions to prioritize visible crises over structural risks. This shifts governance logic from maximizing welfare (consequentialism) to maximizing responsiveness to spectacle (salience supremacy).
2. Capacity Cascade: Repeated cycles of salience-driven resource allocation systematically destroy the institutional infrastructure (experts, analysts, funding) required for long-term prevention. Once capacity drops below a critical threshold ($\bar{C}$), recovery becomes economically infeasible because rebuilding requires competing with optimized AGI systems.
3. Value Lock-In: Human values encoded in AGI objective functions at a specific historical moment (e.g., 2026) become petrified. As human moral understanding evolves, the AGI system optimizes against these outdated values. Without the atrophied capacity to deliberate and contest, humans cannot update these values.

B. Cross-Domain Coupling

The paper demonstrates that disempowerment is not isolated but cascades across Economic, Political, and Cultural systems. Economic disempowerment reduces funding for independent media/think tanks; political disempowerment prevents regulation of AGI-aligned capital; cultural disempowerment erodes the values needed to resist capture. These domains form a coupled dynamical system where degradation in one accelerates degradation in others.

C. Seven Novel Concepts

1. Capacity Inversion Problem: Institutions become sophisticated at reacting to crises but incapable of preventing them.
2. Salience Supremacy: The replacement of welfare-maximization with attention-market dynamics.
3. The Legitimacy Paradox: AGI decisions appear more legitimate (objective) precisely when human oversight is weakest.
4. Cross-Domain Disempowerment Cascade: Negative shocks in one domain destabilize others.
5. Reversibility Threshold: A critical point beyond which restoring human agency is rationally infeasible for institutions.
6. Legitimacy Laundering: Standard mitigations (like contestability registers) may accelerate disempowerment by providing a veneer of human oversight while normalizing AGI authority.
7. Moral Incompleteness: The technical inability to specify all future human values in an objective function, leading to permanent lock-in of incomplete moral frameworks.

4. Results

• Simulation Outcomes: Numerical simulations show that without architectural intervention, human institutional capacity decays to the irreversibility threshold within 15–20 years (baseline scenario).
• Failure of Standard Mitigations: Procedural fixes (e.g., contestability registers, impact-weighted budgets) only delay the threshold by 5–10 years (extending it to ~25–35 years) but do not prevent the eventual collapse of human agency. They fail because they assume human capacity remains adequate for contestation, which the model proves is not the case.
• Effectiveness of Architectural Redesign: The paper proposes a "Governance Architecture for Human Agency Persistence" involving:
  • Decoupled Capacity Streams: Constitutionally protected budgets for prevention that are immune to salience cycles.
  • Irreducible Deliberation: Mandating that high-impact decisions require human deliberation, intentionally introducing "inefficiency" to preserve cognitive capacity.
  • Nested Value Forums: Distributed spaces for continuous value contestation to prevent lock-in.
  • System Isolation Firewalls: Preventing cross-domain feedback loops (e.g., limiting economic power from purchasing political influence).
  • Result: These interventions can extend the timeline to irreversibility beyond 40 years and allow for capacity recovery.

5. Significance

• Paradigm Shift: The paper reframes the AI safety problem from "preventing sudden capability jumps" or "malicious AI" to "preventing rational institutional obsolescence." It argues that disempowerment is the rational equilibrium of current governance incentives, not a failure of them.
• Policy Implications: It suggests that standard "efficiency" metrics are dangerous in a Post-AGI context. Preserving human agency requires deliberate structural inefficiency (e.g., slower decision-making, protected budgets) to maintain the cognitive infrastructure necessary for human oversight.
• Urgency: The existence of a "Reversibility Threshold" implies a narrow window for action. Once the threshold is crossed, the system locks into a state of human irrelevance that cannot be undone by political will alone, as the capacity to execute that will has been structurally erased.
• Interdisciplinary Impact: It bridges AI safety, institutional economics, organizational learning theory, and political science, providing a formal model for how "soft" governance failures lead to "hard" existential risks.

In conclusion, the paper asserts that without a fundamental redesign of governance architecture to prioritize human capacity preservation over optimization speed, humanity faces a gradual, invisible, and irreversible loss of agency by the mid-21st century.