Governing What You Cannot Observe: Adaptive Runtime Governance for Autonomous AI Agents

1. Problem Statement

The paper addresses a critical gap in the governance of autonomous AI agents (particularly Large Language Model agents). While existing platforms (e.g., AWS, Microsoft, Google) provide static, identity-centric authorization (e.g., "Is this action allowed?"), they fail to detect behavioral drift and emergent risks that evolve over time without code changes.

The authors identify three specific failure modes that static policies miss:

1. Behavioral Non-Stationarity: Agents shift distributions silently due to model updates or prompt drift.
2. Compositional Risk: Individual benign actions (e.g., small transfers) combine to form fraud (e.g., structuring).
3. Emergent Discrimination: Adaptive learning amplifies initial data imbalances into systemic bias.

Current solutions (rule-based guardrails, LLM evaluators, static policy engines) are either evadable, introduce latency, or lack memory of prior behavior. The core problem is that agents can remain fully authorized yet become unsafe as unobserved risks accumulate.

2. Core Methodology: The Informational Viability Principle

The paper proposes a theoretical reframing of governance based on Aubin's Viability Theory. The central insight is the Informational Viability Principle:

Governing an autonomous agent reduces to continuously estimating a bound on unobserved risk ($\hat{B}(x)$) and allowing an action only when the agent's observed capacity ($S(x)$) exceeds this bound by a safety margin ($\epsilon$).

Decision Rule:
$$\text{Allow } x \iff S(x) \ge \hat{B}(x) + \epsilon$$

Decomposition of Unobserved Risk ($\hat{B}(x)$)

The paper decomposes $\hat{B}(x)$ into three statistically distinct terms, each requiring specific estimators:

1. $U(x)$ (Uncertainty/Distributional Drift): Risk from the agent's behavior shifting away from its reference distribution. Estimated via KL Divergence on sliding windows of transaction outcomes and tool calls.
2. $SB(x)$ (Structural Bias): Risk from systematic disparities in how different population segments are treated. Estimated via Segment-vs-Rest Z-tests (two-proportion tests).
3. $RG(x)$ (Reality Gap/Sequential Context): Risk from patterns that only emerge in sequences (e.g., structuring fraud). Estimated via PlanGovernor (pattern matching on action histories).

The Agent Viability Framework (AVF)

To operationalize this principle, the authors define three necessary and sufficient properties for a governance system:

• P1 (Monitoring): Continuous estimation of $\hat{B}(x)$ across all constraints (requires stateful history).
• P2 (Anticipation): Predicting the time to boundary crossing ($t^*$) by monitoring the rate of change ($d\hat{B}/dt$) before a violation occurs.
• P3 (Monotonic Restriction): Interventions can only tighten restrictions, never relax them. This preserves the integrity of risk estimates against adversarial manipulation.

3. System Implementation: RiskGate

The authors present RiskGate, a reference implementation of the AVF.

• Architecture: A three-layer Directed Acyclic Graph (DAG) consisting of a Governance Layer (statistical estimators), a Domain Layer (business rules), and an Abstraction Layer (policy API).
• Statistical Estimators:
  • Dual-Channel KL Divergence: Monitors transaction outcomes and tool calls separately with Bonferroni correction for joint False Positive Rate (FPR) control.
  • Adaptive Thresholds (D-UCB1): Uses a Discounted Upper Confidence Bound algorithm to dynamically optimize detection thresholds, balancing false positives vs. missed anomalies.
  • Bias Detection: Implements a segment-vs-rest Z-test with self-inclusion bias correction.
• The Autopilot (Regulation Map):
  • Formalized as an instance of Aubin's Regulation Map ($R_A$).
  • Operates on a "tighten-only" basis. If the system approaches a viability boundary, it automatically tightens thresholds.
  • Includes a Kill Switch as a "last-resort" admissible control to ensure the system never gets stuck without a valid action.
• Predictive Governance:
  • Calculates a Viability Index (VI) $VI(t) \in [-1, +1]$.
  • Uses OLS regression on recent $VI$ values to predict $t^*$ (time to boundary crossing), transforming governance from reactive to predictive.

4. Key Contributions

1. Theoretical Framework: The Informational Viability Principle unifies statistical estimators and governance properties under a single mathematical structure, explaining why monitoring, anticipation, and monotonicity are necessary.
2. Agent Viability Framework (AVF): A formal definition of agent degradation (senescence) and the three properties (P1-P3) required to prevent it.
3. RiskGate Implementation: A complete Python library featuring:
   • Multi-channel drift detection with Monte Carlo calibration.
   • Adaptive threshold optimization via Multi-Armed Bandits.
   • A monotonic pipeline where decisions are strictly cumulative ($D_{final} = \bigwedge D_i$).
   • A closed-loop Autopilot with a formal guarantee of non-empty viability kernels.
4. Analytical Validation:
   • Coverage Mapping: The framework is mapped against the "Agents of Chaos" failure taxonomy (11 case studies), showing how the three terms of $\hat{B}(x)$ cover various failure modes.
   • Worked Demonstrations:
     • Structuring Attack: Shows how $RG(x)$ (PlanGovernor) blocks fraud that $U(x)$ and static rules miss.
     • Emergent Bias: Shows how P2 (Anticipation) detects bias drift 108 operations before a reactive Z-test would trigger.

5. Results and Evaluation

• Note: The paper explicitly states that quantitative empirical evaluation (measured detection rates, latency, ROI) is scoped as future work. The current results are analytical and theoretical.
• Analytical Coverage: The framework successfully maps to 11 documented failure cases from Shapira et al. [23], demonstrating that no single estimator covers all risks; the composition of $U(x)$, $SB(x)$, and $RG(x)$ is required.
• Convergence: The drift reduction strategies in RiskGate align with findings from Rath [21], suggesting that combined strategies (memory, routing, anchoring) can reduce drift by up to 81.5%.
• Predictive Capability: The $t^*$ prediction mechanism is shown to provide a "lead time" for intervention, allowing the system to act before a constraint is violated.

6. Significance and Impact

• Paradigm Shift: Moves AI governance from static, stateless authorization (checking if an action is allowed now) to dynamic, stateful viability (checking if the agent's trajectory is safe).
• Regulatory Alignment: The framework directly addresses requirements of the EU AI Act (Risk Management, Accuracy, Human Oversight) by providing quantifiable, continuous risk estimation and audit trails.
• Formal Rigor: By grounding the system in Viability Theory, the paper provides a mathematical guarantee that the system will not get "stuck" (Theorem 3.1) and that interventions are structurally safe (P3).
• Adversarial Resilience: The Monotonic Restriction (P3) property is highlighted as a critical defense against "adversarial erosion," where attackers try to trick the system into relaxing its own defenses.

In summary, this paper provides a rigorous theoretical foundation and a concrete reference implementation for governing autonomous agents that can adapt to behavioral drift, compositional risks, and emergent bias, ensuring safety even when the agent's behavior is not explicitly observable in real-time.