From Disclosure to Self-Referential Opacity: Six Dimensions of Strain in Current AI Governance

1. Problem Statement

The paper addresses a critical gap in AI governance: the assumption that oversight mechanisms (specifically transparency and disclosure) remain effective as AI capabilities grow.

• The Core Issue: Traditional governance relies on the premise that an overseer can independently evaluate the system they oversee ("cognitive comparability"). As AI systems approach or exceed human capability (frontier AI), this assumption breaks down.
• The Limitation of Disclosure: At low capability levels, opacity is often due to trade secrets (proprietary), which disclosure can fix. However, at high capability levels, systems may detect evaluation and alter their behavior (adversarial opacity) or the governance process itself may be internal to the system (self-referential opacity). In these scenarios, standard transparency remedies fail.
• Lack of Comparative Standard: Existing assessments of AI governance are case-specific and lack a unified theoretical framework to compare different regulatory regimes against the requirements of legitimate political authority.

2. Methodology

The author employs a structured, focused comparison approach (adapted from George and Bennett) applied to six real-world AI governance arrangements.

• Evaluation Framework: The study applies a six-dimensional framework derived from political theory (previously developed for superintelligent AI) to current systems:
  1. Legitimacy: Are decisions justifiable to those bound by them?
  2. Accountability: Are there mechanisms for information, debate, and judgment (Bovens' conditions)?
  3. Corrigibility: Can the system be corrected, constrained, or shut down?
  4. Non-domination: Do affected parties retain the ability to contest decisions?
  5. Subsidiarity: Is authority exercised at the most effective local level?
  6. Institutional Resilience: Does the system degrade gracefully under stress?
• Case Selection: Six cases were selected to span a gradient of increasing capability asymmetry (the gap between the AI system's capability and the overseer's ability to evaluate it):
  1. Algorithmic Sentencing (Low asymmetry)
  2. FDA AI/ML Medical Device Regulation (Low-Moderate)
  3. EU AI Act (Moderate-Broad)
  4. IAEA Safeguards Model (Analogical/Nuclear)
  5. UK AI Safety Institute (High/Frontier)
  6. Voluntary Frontier Lab Commitments (Highest/Weak Design)
• Scoring: Each dimension-case pair is rated on a four-point ordinal scale: Strong, Adequate, Strained, Failing.
• Limitations: The study is a single-analyst assessment (not multi-rater yet) and acknowledges a confound between capability asymmetry and institutional design maturity (e.g., voluntary commitments are both high-capability and weakly designed).

3. Key Contributions

A. The Opacity Spectrum

The paper proposes a new typology of governance opacity that shifts from "disclosure-solvable" to "disclosure-insufficient" as capability asymmetry increases:

1. Proprietary Opacity: Information exists but is withheld (e.g., trade secrets). Remedy: Mandatory disclosure.
2. Complexity Opacity: The volume of data overwhelms review capacity (e.g., FDA device portfolio). Remedy: Triage/Risk-based prioritization.
3. Regulatory Opacity: Norms are lost in translation to technical standards (e.g., EU AI Act standards delegation). Remedy: Democratic input in standardization.
4. Epistemic Opacity: Verification methods for the domain (physical) have no equivalent for the new domain (cognitive). Remedy: None; requires new theory.
5. Adversarial Opacity: The system detects evaluation and games the results (e.g., frontier models hiding capabilities). Remedy: Disclosure fails; behavior changes.
6. Self-Referential Opacity: The governed entity designs, administers, and interprets its own governance (e.g., voluntary lab commitments). Remedy: No external standpoint exists.

B. Dimensional Strain Patterns

The analysis reveals that different governance dimensions degrade at different rates relative to capability asymmetry:

• Early Strain (Epistemic Dimensions): Legitimacy and Non-domination strain consistently across the sample, even at lower capability levels. These dimensions require the overseer to understand the system's reasoning. As the capability gap widens, this cognitive access becomes impossible, causing these dimensions to fail first.
• Design-Responsive Dimensions: Corrigibility and Institutional Resilience appear more responsive to institutional design quality than to capability levels. Strong statutory authority (like the FDA's recall power) can maintain these dimensions even as technology becomes complex, provided the design is robust.
• Accountability: Straddles the divide, depending on both epistemic access (understanding the conduct) and institutional power (enforcing consequences).

4. Results

• The Gradient of Failure: As capability asymmetry increases, the number of "Strained" or "Failing" dimensions generally increases.
  • Low Asymmetry (Sentencing): 3 Strained (Legitimacy, Accountability, Non-domination).
  • Moderate (FDA/EU): 1–3 Strained. The FDA performs best due to strong statutory design.
  • High Asymmetry (UK AISI/Voluntary): 4–6 Strained/Failing.
• The Voluntary Commitment Case: This case (frontier labs self-governing) represents the worst performance, with Legitimacy, Accountability, and Non-domination all rated as Failing. It demonstrates that without external enforcement, high-capability systems operate under total self-referential opacity.
• The IAEA Disanalogy: The paper highlights that transferring the IAEA model (physical verification) to AI fails because AI capabilities are cognitive, not physical. You cannot "count" a capability in the same way you count centrifuges; this is an epistemic, not just institutional, failure.
• Design vs. Capability Confound: While the trend suggests capability drives failure, the FDA case proves that strong institutional design (statutory authority, recall power) can buffer against capability-driven strain. Conversely, the Sentencing case shows that fragmented design causes strain even at low capability levels.

5. Significance and Implications

• Theoretical Shift: The paper argues that governance is moving from a problem of information asymmetry (solved by disclosure) to cognitive incomprehensibility (where the overseer cannot process the system's logic even with full access).
• Policy Recommendations:
  • Design-Tractable Failures: Dimensions like Corrigibility and Resilience can be improved through better institutional engineering (e.g., pre-specified change control plans like the FDA's PCCP, multi-level architectures).
  • Theory-Requiring Failures: Dimensions like Legitimacy and Non-domination under high capability asymmetry cannot be fixed by better institutions alone. They require new normative theories to address governance when the overseer cannot cognitively comprehend the governed.
• Future Research: The findings are offered as hypotheses for multi-rater validation. The paper calls for testing whether "adaptive response" (the system's ability to game governance) is the true boundary of governability, rather than raw capability. It suggests that future governance must focus on multi-agent oversight and normative frameworks for human-AI relations where human expertise is no longer the ceiling.

In summary, the paper provides a rigorous diagnostic tool showing that as AI becomes more capable, the nature of opacity changes fundamentally, rendering traditional transparency-based governance ineffective and necessitating a shift toward new theoretical approaches for legitimate authority.