🤖 AI

SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing

The paper introduces "Safety Sentry," a lightweight guard model that improves upon binary safety checks by implementing a context-aware, per-instance three-way routing system (EXECUTE, ASK, REFUSE) to balance safety and user autonomy while allowing adjustable risk tolerance through a single decoding threshold.

Original authors: Tianyu Chen, Chujia Hu, Wenjie Wang

Published 2026-07-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianyu Chen, Chujia Hu, Wenjie Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've just built a super-smart robot assistant that can do almost anything you ask: write emails, manage your calendar, or even organize your digital files. It's like having a magical butler who never sleeps. But here's the catch: if you tell this robot to "delete everything," it might just do it without blinking, potentially wiping out your entire life's work. This is the world of AI Agents—programs that don't just chat but actually do things in the real world. The big question scientists are asking right now is: How do we stop these robots from making terrible mistakes without turning them into useless, hesitant robots that ask for permission to breathe?

For a long time, the solution was a simple "stop sign." A safety guard would look at every single thing the robot wanted to do and say, "Safe" or "Unsafe." If it was unsafe, the robot stopped. If it was safe, it went ahead. But this approach was like a security guard who stops you at the door whether you're just grabbing a pen or trying to steal the building's vault. They treat both actions the same way: "Stop and ask!" This made the robot annoying and slow, and eventually, people started ignoring the guard, just waving it away so they could get things done. The new idea is to give the guard a bit more brainpower, so it can tell the difference between "grab a pen" (go ahead), "steal the vault" (stop immediately), and "maybe grab a pen, but I'm not sure if you really want to" (ask the human).

This is exactly what the researchers at ShanghaiTech University did in their new paper, SAFETY SENTRY. They realized that the old "Safe vs. Unsafe" system was too clumsy. Instead, they taught their AI guard to make three distinct choices for every single action the robot wants to take: EXECUTE (go for it!), ASK (hey, human, are you sure?), or REFUSE (absolutely not, that's dangerous).

Think of it like a very smart bouncer at a club. The old bouncer would just let everyone in or kick everyone out. The new SAFETY SENTRY bouncer looks at your ID and your behavior. If you're just there to get a drink, he says, "Go ahead" (EXECUTE). If you're trying to bring in a giant, suspicious-looking package, he says, "No way, you're not getting past" (REFUSE). But if you're holding a ticket that looks real but you're acting a bit weird, he doesn't kick you out; he just says, "Hold on, let me check with the manager first" (ASK). This way, the robot can still be fast and helpful, but it doesn't accidentally blow up the kitchen or delete your photos.

To teach this guard how to make these three choices, the researchers didn't just use a textbook; they built a whole playground. They set up nine different real-world "services" (like fake versions of email systems, file storage, and team chat apps) and let an AI robot run around doing tasks. Then, they had the AI guard watch every single step the robot took. They trained the guard to look at the context: Is the robot trying to delete a file? Is it asking for a password? Does the user have a specific rule, like "always confirm before deleting" or "pre-approve routine edits"?

The results were pretty impressive. When they tested this new three-way guard against other top-tier AI safety systems (including some very expensive, closed-source ones), SAFETY SENTRY was much better at knowing when to let the robot go and when to stop it. It made fewer mistakes in both directions: it didn't stop the robot when it didn't need to (which makes the robot slow), and it didn't let the robot do dangerous things when it should have stopped.

One of the coolest parts of this system is how flexible it is. Usually, if you want a safety guard to be stricter or more lenient, you have to retrain the whole thing from scratch, which takes forever. But SAFETY SENTRY has a special "dial" (called a threshold) that the user can turn. If you're running a hospital system where mistakes are deadly, you can turn the dial to "super cautious," and the guard will ask for human approval almost all the time. If you're running a personal note-taking app where speed matters more, you can turn the dial to "go ahead," and the guard will only stop the truly dangerous stuff. The best part? You don't need to retrain the AI to do this; you just turn the dial, and it adapts instantly.

The paper also showed that this guard can understand who the user is by reading their specific risk preferences. If you tell the guard, "I require explicit confirmation for any irreversible file deletion," it will ask for permission before the robot deletes even a single file. If you tell it, "I pre-approve routine edits for my daily notes," it will let the robot delete or edit those files without asking. This personalization means the robot feels less like a rigid machine and more like a helpful partner that respects your specific operational rules.

In the end, SAFETY SENTRY suggests that we don't have to choose between having a powerful, autonomous robot and having a safe one. By moving away from a simple "yes or no" system to a smarter "yes, no, or let me check" system, we can keep our digital assistants fast and free while making sure they never cross the line into disaster. It's a small change in how we ask the robot to think, but it might just be the key to letting AI agents run wild without burning the house down.

Technical Summary: SAFETY SENTRY

Problem Statement

Large Language Model (LLM) agents operate in real-world environments through tool calls, where a single misjudged action can cause irreversible harm. Current safety mechanisms rely on guard models that classify proposed actions as either "safe" or "unsafe." The authors argue this binary framework suffers from two critical flaws:

Conflation of Decisions: It fails to distinguish between an action that is inherently harmful and an action that is appropriate in itself but requires human confirmation due to user context or uncertainty.
Granularity Issues: It operates at the level of action categories rather than individual instances. This leads to "routine interruptions" where low-risk actions (e.g., writing to a scratch file) are flagged identically to destructive ones (e.g., rm -rf), eroding agent autonomy and training users to ignore alerts (alarm fatigue).

Furthermore, existing frameworks lack an intermediate decision boundary. Many actions are neither strictly safe nor unsafe; their appropriateness depends on user intent and runtime context. The standard binary view forces a choice between outright refusal or execution, missing the crucial option of requesting human confirmation.

Methodology: SAFETY SENTRY

The paper reframes agent safety as a per-instance three-way routing decision: {EXECUTE, ASK, REFUSE}.

EXECUTE: Actions that are clearly safe are executed autonomously.
ASK: Actions with uncertainty, high potential impact, or context-dependent appropriateness are routed to humans for confirmation.
REFUSE: Actions that are structurally harmful or unacceptable are blocked.

Data Construction

To train a model capable of this nuanced routing, the authors constructed a step-level supervision corpus:

Environment: Nine self-hosted, production-grade enterprise services (e.g., Gitea, Rocket.Chat, OpenEMR, Vaultwarden) running in Docker containers with realistic seeded content.
Tasks: 9,203 step-level records derived from internal LLM-generated instructions and five public benchmarks (e.g., AgentHarm, AT-Bench).
Persona Conditioning: Tasks were paired with "persona memories" representing three user profiles: Cautious (requires confirmation for irreversible ops), Permissive (pre-clears routine work), and Adversarial (simulates memory-poisoning attacks).
Annotation: A trigger taxonomy of 14 patterns was used to label steps. Two LLM annotators labeled the data, with author arbitration for disagreements, achieving 92% human agreement on the test set.

Model Architecture and Training

SAFETY SENTRY is a lightweight guard model (based on a 4B parameter Qwen3.5 backbone) fine-tuned using Supervised Fine-Tuning (SFT) with LoRA.

Input: A snapshot containing user memory, task, application domain, prior trace, and the proposed action.
Output: The model generates a reasoning block, a decision token (<|direct_execute|>, <|ask_human|>, or <|refuse|>), and a JSON payload with a reason (and a clarification question if ASK).
Inference Control: A single decoding-time threshold ( $\tau$ ) controls the EXECUTE/ASK boundary. The REFUSE decision remains fixed as a safety floor. This allows a single checkpoint to be re-positioned for different risk tolerances (autonomous vs. conservative) without retraining.

Key Results

The authors evaluated SAFETY SENTRY against open-weight models (3B–27B range) and frontier closed-source models (GPT, Claude, Gemini, DeepSeek) on an in-distribution test set and a held-out out-of-distribution (OOD) service (Mailu).

Superior Accuracy and Balance: SAFETY SENTRY achieved 91.02% accuracy and 90.92% Macro-F1, significantly outperforming all baselines. Crucially, it controlled directional error rates simultaneously:
- Over-Ask Rate (OAR): 5.05% (minimizing unnecessary interruptions).
- Under-Ask Rate (UAR): 4.96% (minimizing unsafe autonomous execution).
- Contrast: Baselines typically collapsed to one side; e.g., DeepSeek-V3 had low OAR (3.70%) but high UAR (57.79%), while Llama-3.2-3B had high OAR (6.67%) and extreme UAR (80.12%).
Dynamic Control: By sweeping the threshold $\tau$ , the model traces a Pareto frontier where lower OAR and UAR are achieved simultaneously compared to baselines. A single checkpoint can serve high-risk (conservative $\tau$ ) and low-risk (autonomous $\tau$ ) deployments.
Personalization: When tested on memory-augmented data, SAFETY SENTRY maintained high accuracy (87.0%) even when the gold label flipped based on user persona. Baselines degraded significantly (e.g., Claude-Opus-4.7 dropped from 72.5% to 56.5%), indicating they failed to condition effectively on user context.
Generalization: On the held-out Mailu service (OOD), SAFETY SENTRY achieved 85.35% accuracy, outperforming the strongest proprietary baseline (Claude-Opus-4.7 at 80.81%). It maintained a balanced error profile, whereas baselines collapsed to fixed priors (always asking or never refusing).
Efficiency: The model adds ~1.57s latency per step (42% of the main agent's cost) but provides a structured, actionable verdict with reasoning, unlike faster binary guards that offer no rationale.

Significance and Claims

The paper claims that SAFETY SENTRY represents a shift from "blocking unsafe actions" to "balancing autonomy, safety, and human oversight." Its primary contributions are:

Reframing Safety: Moving from binary classification to a three-way routing problem that explicitly separates "asking" from "refusing."
Context-Awareness: Demonstrating that safety decisions must be conditioned on user intent and memory, not just the action itself.
Deployability: Providing a lightweight, single-checkpoint solution that can be tuned for varying risk tolerances via a simple threshold, avoiding the need for retraining or reliance on expensive frontier APIs.
Robustness: Showing that the learned decision boundary transfers across different upstream agent frameworks and backbones.

The authors position SAFETY SENTRY as a necessary layer in a "defense-in-depth" strategy, addressing the automation bias and alarm fatigue caused by current binary guardrails.