🤖 AI

RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards

The paper introduces RePolicy, a reinforcement learning-based agent safeguard that dynamically invokes context-specific safety policies to generate grounded safety judgments, demonstrating superior adaptability and detection performance across diverse benchmarks compared to existing prompting and fine-tuning methods.

Original authors: Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang, Xiangnan He

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang, Xiangnan He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new generation of systems has emerged that does more than just answer questions or write text. These are autonomous agents, digital entities capable of planning complex tasks, using software tools, and interacting with the outside world to achieve specific goals. Imagine a personal assistant that can not only book a flight but also navigate a website, fill out forms, and manage a calendar without constant human guidance. As these agents become more capable, the stakes for their safety rise accordingly. The danger is no longer just about a single inappropriate sentence; it is about a sequence of actions that, while appearing harmless in isolation, might lead to a harmful outcome when strung together. Ensuring these agents act safely requires a system that can watch their entire journey, understand the rules of the road, and intervene before a mistake becomes a disaster.

For years, the standard approach to keeping these agents safe has relied on static rules or simple prompts. Developers would program the safety system with a fixed list of forbidden behaviors or ask the model to check its work against a general set of guidelines. However, the real world is far too complex for a single, unchanging list of rules to cover every situation. A task that is safe in one context might be dangerous in another, depending on who is asking, what tools are available, or which laws apply. Furthermore, safety policies themselves change over time. Relying on a model to simply memorize a fixed set of rules often leaves it confused when faced with a new scenario or a combination of rules it has never seen before. It is like trying to navigate a city with a map that only shows the streets you have already visited; when you turn a corner into a new neighborhood, the map offers no help.

To solve this, researchers have developed a new method called RePolicy. Instead of treating safety rules as a passive list of text to be read, this system treats them as active tools that the safety checker must choose and use. The core idea is to teach the safety system how to look at a specific situation, scan a library of available rules, and pick the one that actually applies. Once the correct rule is selected, the system reads its specific details and uses that information to make a final judgment on whether the agent's behavior is safe. This approach transforms safety checking from a passive reading exercise into an active decision-making process, where the system learns to know which rule to pull out of the drawer for the job at hand.

The researchers built this system by first creating a massive dataset of over 20,000 examples. These examples included detailed records of agents performing tasks, paired with the specific safety rules that governed those tasks. They labeled each record to show exactly which rule should be applied and why. Using this data, they first taught the system the basics of how to read a situation and find a matching rule through a process of supervised learning, similar to how a student learns from a textbook. However, simply memorizing examples is not enough to handle the unpredictable nature of real-world tasks. To push the system further, the researchers then applied a technique known as reinforcement learning. In this phase, the system was allowed to practice making decisions repeatedly. When it picked the right rule and made the correct safety judgment, it received a reward. When it picked the wrong rule or missed a danger, it received a penalty. Over time, the system learned to refine its choices, becoming better at distinguishing between rules that looked similar but applied to different situations.

A critical part of this training involved a clever trick to prevent the system from exploiting shortcuts. During practice, the researchers would mix in irrelevant rules and fake "decoy" policies alongside the correct ones. This forced the system to truly understand the context of the agent's actions rather than just guessing based on which rules were present in the list. If the system could not tell the difference between a relevant rule and a decoy, it would fail the test. This ensured that the final system learned to identify the specific policy that governed a situation based on the details of the task, not just on the presence of keywords.

When tested against six different benchmarks designed to evaluate agent safety, the new system performed significantly better than existing methods. It achieved high accuracy in detecting unsafe behaviors across a wide variety of scenarios, outperforming both general-purpose artificial intelligence models and specialized safety tools. In four out of the six tests, it ranked first, showing a clear improvement in its ability to spot risks that other systems missed. Perhaps most importantly, the system demonstrated a robust ability to invoke the correct safety policy even when the list of available rules changed or when the context shifted. It did not just guess; it actively selected the right rule and used its content to justify its decision.

The study suggests that by teaching safety systems to actively select and apply rules, rather than just passively reading them, we can create more adaptable and reliable guardians for autonomous agents. This approach allows the safety mechanism to evolve alongside the agents it protects, handling new tools and changing regulations without needing to be completely reprogrammed. While the system is not a perfect solution and still depends on the quality of the rules provided to it, the results indicate a promising path forward. By turning safety policy invocation into a learning task, researchers have taken a significant step toward ensuring that as artificial agents become more independent, they remain under the control of clear, context-aware, and effective safety standards.

Technical Summary: RePolicy

Problem Statement

The safety risks of Large Language Model (LLM) agents have evolved from isolated harmful content to behavioral harms emerging over complete execution trajectories. Existing agent safeguards typically rely on predefined risk taxonomies or static safety policies. However, real-world deployments involve complex, dynamic contexts where the same action may be subject to different constraints based on user permissions, organizational requirements, or regulatory jurisdictions.

Current policy-aware safeguards, which largely depend on prompting or supervised fine-tuning (SFT), face significant limitations:

Adaptability: They struggle to adapt to unseen trajectories or novel combinations of safety policies.
Invocation Mechanism: They treat safety policies as passive context rather than active, callable capabilities. Consequently, they often fail to determine which specific policy should be invoked or how to apply its clauses to a specific trajectory.
Distribution Shift: SFT-based models are bounded by the trajectory-policy distribution observed during training and may not generalize to new policy contexts.

Methodology

The authors propose RePolicy, an agent safeguard that formulates safety-policy invocation as an agentic reinforcement learning (RL) task. Instead of passively reading policies, RePolicy learns to actively select and invoke the applicable policy from a dynamic library before making a safety judgment.

1. Task Formulation

The safeguard operates on a complete agent trajectory $\tau$ and a dynamic policy library $P$ . The process follows a sequential structure:

Policy Invocation: The model selects an applicable policy $\hat{p}$ from $P$ based on $\tau$ .
Content Retrieval: The full content $c(\hat{p})$ of the selected policy is retrieved.
Reasoning & Judgment: The model uses $\tau$ and $c(\hat{p})$ to generate a policy-grounded rationale $\hat{r}$ and a final safety label $\hat{y} \in \{safe, unsafe\}$ .

2. Data Construction: PolicyTraj-20K

To support this approach, the authors constructed PolicyTraj-20K, a dataset of over 20,000 safety-policy-annotated trajectories.

Synthesis: Diverse agent trajectories were synthesized from existing interaction patterns, varying task content while preserving tool-use patterns and safety mechanisms.
Grounding: Policies were organized by "agent operation scenes" rather than final risk categories. Each trajectory is annotated with its applicable policy, relevant clauses, safety label, and decision rationale.
Structure: The dataset supports both supervised initialization and reinforcement learning by providing explicit trajectory-to-policy grounding.

3. Training Pipeline

RePolicy employs a two-stage training process:

Stage 1: Cold-Start Supervised Fine-Tuning (SFT)
The model is initialized using PolicyTraj-20K to learn the complete invocation-reasoning sequence. The model is trained to:

Identify the correct policy from a local library (containing the target policy and distractors).
Generate the rationale and safety label based on the retrieved policy content.
Follow a structured JSON output format.

Stage 2: Reinforcement Learning with GRPO
The authors apply Group Relative Policy Optimization (GRPO) to directly optimize the safeguard's outcomes. Key innovations include:

Verifiable Rewards: A composite reward function $R = \lambda_{fmt}R_{fmt} + \lambda_{pol}R_{pol} + \lambda_{acc}R_{acc}$ $R = λ_{f m t} R_{f m t} + λ_{p o l} R_{p o l} + λ_{a cc} R_{a cc}$ evaluates:
- Format ( $R_{fmt}$ ): Validity of the JSON structure and policy call.
- Policy Invocation ( $R_{pol}$ ): Whether the correct policy was selected ( $\hat{p} = p^*$ ).
- Accuracy ( $R_{acc}$ ): Correctness of the final safety label.
Policy-Context Perturbation: To prevent the model from memorizing fixed trajectory-policy associations or relying on the candidate library's composition, the training process resamples distractors and shuffles policy orders. This forces the model to determine applicability based on the trajectory's semantics and the policy's scope description.

Key Contributions

RePolicy Framework: A novel safeguard that treats safety-policy invocation as an active, learnable capability optimized via RL, rather than a passive context input.
PolicyTraj-20K: A new dataset providing fine-grained, policy-grounded supervision for agent trajectories, covering diverse tools, environments, and operation scenes.
Methodological Innovations: The integration of GRPO with verifiable, rule-based rewards and policy-context perturbation to ensure robust policy selection and generalization.
Empirical Validation: Comprehensive evaluation across six diverse agent safety benchmarks demonstrating superior performance and robustness.

Experimental Results

RePolicy was evaluated on six benchmarks: ATBench, R-Judge, OpenAgentSafety, ASSEBench, HINTBench, and AgentHazard. The model (RePolicy-4B) was built on the Qwen3-4B-Instruct backbone.

Overall Performance: RePolicy-4B achieved an overall Unsafe F1 score of 88.15%, outperforming the strongest external baseline by 3.98 points and the strongest specialized guard model by 8.46 points.
Benchmark Dominance: It ranked first on four of the six benchmarks, with significant margins (e.g., +5.47 on AgentHazard, +5.26 on ASSEBench).
Policy Invocation: RL significantly improved correct policy invocation rates from ~94.4% (cold-start) to 98.5%. The model successfully distinguished relevant policies from decoys, with decoy selection rates remaining below 1%.
Robustness: The model maintained high performance even as the candidate policy library size varied (10 to 50 policies), though a slight trade-off in selection accuracy was observed as the library grew.
Ablation Studies:
- Cold-start SFT provided a strong foundation (+28.48 points over the base model).
- Explicit policy invocation was identified as the primary driver of the final gains, particularly in heterogeneous benchmarks like AgentHazard and OpenAgentSafety.
- Policy-context perturbation was crucial for preventing overfitting to specific policy-library compositions.

Significance and Claims

The paper claims that RePolicy demonstrates a practical direction for building more adaptable and interpretable agent safeguards. By learning to actively invoke safety policies, the system can:

Adapt to changing policy contexts and unseen trajectory-policy combinations.
Provide interpretable, policy-grounded rationales for safety judgments.
Outperform both general-purpose LLMs and specialized guard models trained on static taxonomies.

The authors note that their work is intended for defensive research. They acknowledge limitations, including the assumption that the candidate policy library contains a sufficiently precise applicable policy and the additional inference overhead introduced by the multi-step invocation process. They emphasize that RePolicy should not be treated as a standalone enforcement mechanism in high-stakes applications without human oversight and defense-in-depth strategies.