Technical Summary: The AI Evaluability Gap

Problem Statement

Organizations deploying AI at scale face a dual challenge: managing AI risk (determining if a system is safe, fair, reliable, and compliant enough to operate) and sustaining AI value (determining if a system justifies continued resource allocation). Current governance practices often fail because they treat these as separate problems with distinct owners and vocabularies, or they focus on system properties rather than the evidence required to make decisions about them.

The paper identifies a fundamental underlying condition termed the AI Evaluability Gap: the state in which organizations lack sufficient evidence to support high-confidence governance decisions regarding either risk or value. This gap leads to two failure modes:

1. Over-deployment: Accepting unmeasured risks due to insufficient evidence of safety.
2. Under-investment: Forgoing unmeasured value due to insufficient evidence of benefit.

The authors argue that current frameworks suffer from a "category error," treating governance as synonymous with risk management and assuming evidence sufficiency as a given rather than a primary object of governance. Furthermore, existing practices often fail to account for the non-stationarity of AI systems, where evidence decays rapidly due to user adaptation, model retraining, and shifting environments.

Methodology and Framework

The paper proposes Evaluability as a new theoretical framework to address this gap. Rather than prescribing specific metrics or statistical methods, Evaluability defines the evidentiary conditions under which governance decisions become defensible.

Core Definition

Evaluability is defined as the capability of a system to generate, maintain, and renew evidence sufficient to support high-confidence governance decisions regarding its operation and resource allocation over time.

Formally, Evaluability is a function of six properties of an AI system's evidence stream:
$$\text{Evaluability} = f(O, A, I, V, C, T)$$
Where:

• Observability (O): Can outcomes relevant to the decision be observed? (e.g., logs, telemetry, business outcomes).
• Attributability (A): Can the AI's contribution be isolated from confounders? (Requires causal or quasi-experimental design).
• Intervenability (I): Can the system be modified to test claims and respond to failures? (e.g., rollback, A/B testing).
• Verifiability (V): Can an independent party audit and reproduce the analysis?
• Calibration (C): Do stated confidence levels correspond to observed frequencies of correctness?
• Temporal Validity (T): Does the evidence remain relevant as the environment and system evolve?

Decision-Centric Formulation

The framework shifts the focus from measuring system properties to assessing the confidence in a decision ($D$) given a body of evidence ($E$).

• Confidence ($Conf(D|E)$): An epistemic state regarding the sufficiency of evidence, not a probability of a world state.
• Decision Rule: An action $D$ is warranted if and only if $Conf(D|E) > \tau(D)$, where $\tau$ is a threshold determined by the decision's materiality and error asymmetry.
• The Gap: The Evaluability Gap exists when $Conf(D|E) < \tau(D)$.

Two Classes of Certification

The framework distinguishes between two types of governance decisions, each requiring different evidentiary structures:

1. Operational Certification: Determines if a system may operate. It relies on structural evidence (verification, stress testing, adversarial evaluation) to establish the absence of failure modes. The evidentiary standard is typically high.
2. Investment Certification: Determines if a system merits continued resources. It relies on causal evidence (counterfactual reasoning, randomized rollouts) to establish value generation. The evidentiary standard scales with the materiality of the spend.

The Evaluability Cycle

The six properties are not a static checklist but a continuous cycle:

1. Observability captures what happened.
2. Attributability links outcomes to the AI.
3. Verifiability allows independent trust.
4. Calibration ensures confidence statements are trustworthy.
5. Temporal Validity assesses if evidence is still current.
6. Intervenability enables action (testing, retraining) when uncertainty arises, generating new observations to restart the cycle.

The framework emphasizes that evidence is perishable. In the absence of renewal, confidence decays over time ($Conf(D|E(t)) \to Conf(D|\emptyset)$), necessitating continuous re-certification rather than one-time approval.

Institutional Layer

To avoid epistemic capture, the framework proposes a distributed institutional model:

• Evidence Producers: Engineering and data teams generating logs and results.
• Evidence Verifiers: Risk, audit, and finance functions validating integrity and calibration.
• Evidence Adjudicators: Boards or committees setting thresholds and resolving conflicts between certifications.

Key Contributions

The paper makes three primary contributions:

1. Unification of Governance Objects: It identifies evidence sufficiency as the unifying object across governance domains, formalizing it through the function $Conf(D|E)$. It argues that governance is not about measuring properties (safety, value) directly, but about determining if sufficient evidence exists to warrant decisions about those properties.
2. Operational vs. Investment Certification: It introduces a structural distinction between operational and investment decisions. This clarifies the confusion regarding "value" in governance taxonomies, allowing organizations to coherently assert that a system is operationally safe but not investment-worthy (or vice versa), thereby resolving governance paralysis.
3. Elevation of Temporal Validity and Calibration: It elevates temporal validity and calibration from peripheral concerns to first-class properties. The paper argues that current AI governance is rich in monitoring but poor in calibration, leading to "confident decisions whose confidence is itself unwarranted."

Results and Claims

The paper does not present empirical experimental results, as it is a theoretical framework. Its "results" are logical propositions and structural arguments:

• Proposition 3.1 (Governance Non-Identifiability): Two AI systems with indistinguishable observed outcomes may require different governance decisions because the confidence warranted by their evidence streams differs. Governance cannot be inferred from outcomes alone; it depends on the structure and provenance of the evidence.
• The Necessity of the Six Properties: A system may be accurate or profitable but still fail the Evaluability test if the supporting evidence is unverifiable, uncalibrated, or obsolete.
• The Decay of Evidence: The paper claims that in AI, evidence decays rapidly due to non-stationarity, capability emergence, and human adaptation. Therefore, governance must be a continuous process of re-certification tied to empirically estimated decay rates, not calendar conventions.

Significance and Scope

The authors position Evaluability as a substrate for existing governance frameworks (such as NIST RMF, ISO/IEC 42001, and assurance cases) rather than a replacement.

• What it is: A framework for determining if sufficient evidence exists to support high-confidence decisions. It provides the architectural standards for evidence generation and validation.
• What it is not: It does not determine whether a system is actually safe, fair, reliable, or valuable. It does not replace domain-specific risk models, fairness metrics, or ROI calculators. It does not prescribe specific statistical methods for causal inference or confidence representation (allowing for Bayesian, frequentist, or Dempster-Shafer approaches).

The paper concludes that as AI systems become more dynamic and autonomous, the ability to generate, maintain, and renew evidence (Evaluability) may become as critical as the underlying models themselves. The framework aims to close the gap between the complexity of modern AI systems and the sufficiency of the evidence required to govern them responsibly.