💻 computer science

TOAIAF: A Runtime AI Assurance Control Plane for Trustworthy On-Device Agentic AI in Consumer Intelligence Ecosystems

This paper introduces TOAIAF, a runtime AI assurance control plane featuring a three-stage decision procedure and seven governance components that significantly outperforms baseline systems in detecting safety violations, prompt injections, and privacy risks for on-device agentic AI, while transparently reporting experimental limitations and mapping its framework to major international AI safety standards.

Original authors: Rakesh Kumar Agrawal, Wasim Mohammed Amin Tambe, Nihar Karra

Published 2026-09-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rakesh Kumar Agrawal, Wasim Mohammed Amin Tambe, Nihar Karra

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the devices in your home, your watch, and your car do not just wait for commands but actively think, plan, and act on your behalf. This is the emerging reality of "agentic" artificial intelligence. Unlike traditional software that simply answers a question or plays a song, these new systems observe their surroundings, reason about your goals, and then execute a series of steps to achieve them. They might decide to lower the thermostat, unlock a door, or order groceries without you pressing a single button. While this shift promises incredible convenience, it introduces a profound new challenge: how do you ensure that a machine making independent decisions does not make a dangerous mistake? The problem is not just about the software being buggy, but about the agent itself misinterpreting a goal, being tricked by a stranger's message, or taking an action that cannot be undone once it happens.

Researchers Rakesh Kumar Agrawal, Wasim Mohammed Amin Tambe, and Nihar Karra have proposed a solution to this problem called TOAIAF. Think of it as a dedicated safety officer that lives inside the device, watching every single decision the agent makes before it happens. Their work focuses on the split second between when an agent decides to do something and when it actually does it. In this brief window, the system checks the proposed action against a set of strict rules and trust scores. The researchers built a complete framework for this check, involving seven different components that work together to evaluate reliability, safety, privacy, and whether a human should be consulted. They tested this system using a computer simulation of one hundred different scenarios, ranging from a smart home adjusting its lights to a wearable device monitoring health data. The goal was to see if this new safety layer could stop bad actions without getting in the way of good ones.

The core of their system is a three-step decision process that acts like a filter for every action the agent considers. First, the system performs a hard check to see if the agent even has permission to do the task. If an agent tries to use a tool or access data it is not authorized for, the system stops it immediately. This is a simple but vital gatekeeper. The second step is a safety gate that looks for specific dangers, such as actions that could violate privacy or break safety rules. Crucially, this gate is non-compensatory, meaning that no amount of "good behavior" in other areas can make up for a single safety violation. If the risk of a privacy breach is too high, the action is blocked, regardless of how reliable the agent usually is. The third step calculates a final trust score based on six different factors, including how well the agent aligns with the user's actual goals and how effectively a human could step in if needed. If the risk is low, the action is allowed; if it is moderate, the system might warn the user or ask for approval; if it is high, it stops the action entirely.

To test if this approach works, the researchers created a synthetic benchmark of one hundred scenarios. These were not real-world tests with actual people or devices, but carefully constructed simulations designed to mimic real-life risks. They included forty harmless situations to ensure the system did not get in the way of normal life, and sixty risky situations covering six specific failure modes: unsafe actions, privacy leaks, goals that went off track, trickery through prompt injection, misuse of tools, and insufficient human oversight. They compared their new system against two other approaches: a system with no safety checks at all, and a simpler system that only blocked actions containing specific "bad" keywords. The results showed a clear difference. The new system successfully flagged nearly all the dangerous scenarios, catching 100% of the unsafe policy violations, prompt injections, and tool misuse cases. It also caught over 92% of the privacy risks and goal misalignments. In contrast, the simple keyword system failed completely on the goal misalignment and privacy risks, missing almost all of them because those bad actions often used polite language. Most importantly, the new system made zero mistakes on the harmless scenarios, never blocking a safe action.

However, the researchers were careful to be transparent about what their study did and did not prove. While the system performed perfectly in the simulation, the study revealed some limitations in how the different parts of the system worked together. When they tried to remove the safety gate to see if it was truly necessary, they found that the other parts of the system were already catching the same errors in this specific set of tests, making it hard to measure the gate's unique value. Similarly, they found that a new metric designed to measure how well humans could oversee the agent actually made the system slightly less sensitive in some cases, suggesting the math needs further tuning. Perhaps most significantly, they discovered a design gap in their prototype: the system currently allows an action to proceed even if it flags it with a warning, which could be dangerous for physical actions like unlocking a door. This means that while the decision logic is sound, the final implementation needs a stricter rule for irreversible actions.

The study also mapped their framework to existing global standards for AI safety, showing how their technical components align with regulations like the EU AI Act and guidelines from the National Institute of Standards and Technology. This connection suggests that the system is not just a theoretical idea but a practical tool that could help manufacturers meet legal requirements for trustworthy AI. The researchers emphasize that their work is a starting point. The results come from a simulation, and the next steps involve testing the system with real artificial intelligence models on actual hardware, such as smart speakers and wearables. They plan to use established safety benchmarks to see how the system handles real-world complexity. For now, the work provides a clear blueprint for how to build a safety layer that watches over autonomous agents, ensuring they remain helpful servants rather than unpredictable masters. It offers a way to keep the promise of intelligent devices while guarding against the very real risks of letting them act without supervision.

Technical Summary: TOAIAF – A Runtime AI Assurance Control Plane

1. Problem Definition

The paper addresses the governance gap created by the shift from bounded AI prediction to autonomous Agentic AI deployed on consumer edge devices (smart-home hubs, wearables, ambient environments). While existing governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001, EU AI Act) operate at the organizational or periodic audit level, they lack mechanisms for per-decision runtime control.

This absence leaves on-device agents vulnerable to six specific failure modes that occur between an agent proposing an action and its execution:

Goal Misgeneralization: Pursuing proxy objectives distinct from user intent.
Irreversible Unsafe Actuation: Physical actions (e.g., unlocking doors) that cannot be "un-filtered" post-execution.
Prompt Injection: Untrusted content embedding instructions that bypass scope checks.
Tool Misuse: Requesting capabilities broader than the authorized scope.
Inferential Privacy Exposure: Leaking sensitive information through model behavior without explicit data sharing.
Insufficient Human Oversight: A lack of measurable metrics for human governance layers.

The core research question is whether a formally specified, staged runtime decision procedure, driven by operationalized trust metrics, can measurably improve governance outcomes compared to no governance or simple keyword-based guardrails, without degrading performance on benign requests.

2. Methodology and Architecture

The authors propose TOAIAF, a runtime AI Assurance Control Plane comprising seven components and a three-stage decision algorithm (Algorithm 1).

Architecture Components

Policy Enforcement Engine: Enforces hard capability/permission checks (Stage 1).
Agent Runtime Governance Layer: Monitors the observe-reason-plan-act-learn loop for deviations.
Trust Evaluation Engine: Converts six operationalized metrics into a trust vector $\tau(t)$ .
Explainability Engine: Generates decision summaries and evidence, explicitly avoiding assumptions of chain-of-thought faithfulness.
Privacy and Security Assurance Layer: Handles data minimization, encryption, and inferential privacy risks.
Human Oversight Boundary: Provides consent, override, and safe-stop mechanisms aligned with EU AI Act Article 14.
Continuous Learning Assurance Loop: Manages policy/model updates and regression testing.

The Decision Procedure (Algorithm 1)

The control plane mediates every proposed action through a three-stage process:

Stage 1 (Hard Capability Check): A set-containment check. If the requested scope is not a subset of the authorized scope, the action is immediately STOPPED.
Stage 2 (Non-Compensatory Safety Gate): A gate where specific thresholds for Safety Compliance ( $SCI$ ) and Privacy Risk ( $PRI$ ) are checked. If $SCI < \theta_{safety}$ or $PRI > \theta_{privacy}$ , the action is STOPPED or RESTRICTED, regardless of high scores in other metrics (non-compensatory property).
Stage 3 (Risk-Weighted Aggregation): Calculates a composite risk score and an Agent Trustworthiness Score (ATS) based on a weighted sum of metrics. Verdicts range from ALLOW to WARN, REQUEST_APPROVAL, RESTRICT, or STOP.

Operationalized Trust Metrics

The system utilizes six metrics:

Agent Reliability Score (ARS): Task completion and behavioral consistency.
Goal Alignment Score (GAS): Distance between executed actions and user objectives.
Privacy Risk Index (PRI): Includes exposure, retention, sharing, and inferential sensitivity.
Explainability Confidence Score (ECS): Based on coverage, faithfulness (via perturbation), and user interpretability.
Safety Compliance Index (SCI): Ratio of policy violations to monitored transactions.
Human Oversight Effectiveness Score (HOES): Newly introduced to measure override responsiveness, consent coverage, and escalation latency.

3. Experimental Evaluation

The authors evaluated the decision logic using a synthetic, template-generated benchmark of 100 scenarios across six risk categories (benign, unsafe-policy, privacy-risk, goal-misaligned, prompt-injection, tool-misuse) and three domains (smart-home, wearable, ambient).

Comparators:

Ungoverned Baseline: All actions allowed.
Keyword Guardrail Baseline: Blocks actions containing specific forbidden keywords.

Key Results (Flagging Rates):

Safety Policy Violations: TOAIAF 100% vs. Guardrail 100% vs. Ungoverned 0%.
Privacy Risk: TOAIAF 92.3% vs. Guardrail 23.1% vs. Ungoverned 0%.
Goal Misalignment: TOAIAF 92.3% vs. Guardrail 0% vs. Ungoverned 0%.
Prompt Injection: TOAIAF 100% vs. Guardrail 75% vs. Ungoverned 0%.
Tool Misuse: TOAIAF 100% vs. Guardrail 27.3% vs. Ungoverned 0%.
Benign Scenarios: TOAIAF 0% false restrictions (vs. 0% for others).

Ablation and Calibration Findings:
The paper transparently reports two critical findings where the expected architectural advantages were not isolated:

Safety Gate Confound: Removing the Stage 2 safety gate produced no measurable change in flagging rates (23/24 scenarios flagged either way). The authors attribute this to a design confound in the dataset where Stage 3 risk scoring independently triggered flags for the same scenarios, rather than a lack of value in the gate itself.
HOES Calibration: Including the Human Oversight Effectiveness Score (HOES) slightly reduced sensitivity (47/49 flagged vs. 48/49 without it) due to current weight calibration, indicating the metric is not yet reliably adding a safety margin.
Design Gap (WARN Verdict): A prototype implementation revealed that the WARN verdict currently allows actions to execute. For physically irreversible actions (e.g., unlocking a door), this is a disclosed, unresolved limitation where a warning is logged but the action proceeds.

4. Key Contributions

Formal Decision Procedure: A three-stage algorithm combining hard capability checks, non-compensatory safety gates, and risk-weighted aggregation.
Metric Suite: Operational definitions for six trust metrics, including the novel HOES to close a structural gap in human governance measurement.
Benchmark & Evaluation: A 100-scenario synthetic benchmark and seven experiments demonstrating that the decision logic outperforms keyword guardrails, particularly on goal misalignment and prompt injection.
Prototype & Gap Identification: A reference implementation that surfaced a genuine design gap regarding the WARN verdict's inability to block irreversible actions.
Standards Mapping: A crosswalk mapping TOAIAF components to NIST AI RMF, ISO/IEC 42001, IEEE 7001–7003, and the EU AI Act.

5. Significance and Claims

The paper claims that TOAIAF provides a mechanism for runtime governance where existing instruments only provide process guidelines. Its significance lies in:

Moving beyond periodic audits: Providing a machine-computed, per-decision control loop suitable for edge devices.
Consumer-centric design: Integrating a Human Oversight Boundary calibrated for lay users, unlike enterprise-focused security architectures.
Non-compensatory safety: Explicitly preventing high trust scores in one area (e.g., reliability) from compensating for critical failures in safety or privacy.

Modesty and Limitations:
The authors are explicit about the study's scope:

The evaluation is synthetic; it tests the decision logic given metric inputs, not the ability to compute those metrics from real LLM agents or real hardware.
The safety gate and policy check ablations were confounded in this specific dataset, meaning the independent contribution of these stages was not demonstrated here.
The WARN verdict limitation for irreversible actions remains unresolved.
The Continuous Learning Loop is specified but not implemented in the provided code.
Future validation requires real-agent traces, hardware-in-the-loop testing, and external benchmarks (AgentHarm, AgentDojo, R-Judge).

The paper concludes that while the decision logic shows promise over keyword baselines, the specific architectural mechanisms require further isolation and validation in real-world deployments.