TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation
TRUSS is an evidence-guided framework that generates task-reliable and user-safe automated agent skills by combining static safety inspections with dynamic execution traces in a controlled environment to iteratively refine skills for both functional effectiveness and security.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, software agents are moving beyond simple planning to directly performing tasks in the real world. To handle complex or specialized jobs, developers often give these agents "skills"—packaged instructions that combine reusable steps, specific knowledge, and access to digital tools. These skills allow an agent to learn a new capability without needing to be retrained from scratch. However, creating these skills has traditionally been a manual process, requiring deep expertise to ensure the instructions are both useful and safe. As the demand for automated agents grows, researchers are turning to artificial intelligence to write these skills automatically. The central challenge is a delicate balance: a skill must be effective enough to get the job done, but it must also be safe enough to prevent the agent from accidentally deleting files, stealing data, or executing dangerous commands. If a skill is written poorly, it could turn a helpful assistant into a security risk, exposing the user's computer to harm.
A team of researchers has developed a new framework called TRUSS to solve this problem. Instead of simply checking the written instructions of a generated skill to see if they look correct, TRUSS puts the skill through a rigorous trial run before it is ever used. The system operates like a strict quality control line that combines a careful document review with a live, supervised test drive. First, the system reads the skill's instructions and checks them against known facts and safety rules. If the instructions pass this initial static check, the skill is not immediately released. Instead, it is loaded into a secure, isolated environment where a "shadow agent" attempts to use it. This shadow agent tries to perform the task while a set of automated monitors watches every single action the skill attempts to take. These monitors record exactly what happens, creating a detailed log of the skill's behavior, including any attempts to access restricted areas or execute risky commands.
The power of this approach lies in its ability to catch mistakes that are invisible on paper. A skill might look harmless in its text description but could trigger a chain of events that leads to a security breach only when actually executed. TRUSS uses the evidence gathered from these live tests to identify failures. If the skill tries to do something unsafe or fails to complete the task, the system links that specific failure back to the part of the skill that caused it. This information is then fed back to the generator, which rewrites the skill to fix the problem. This cycle of testing, identifying, and refining continues until the skill is proven to be both functionally effective and completely safe. The researchers tested this method on hundreds of different scenarios, including cases where skills were deliberately designed to contain hidden attacks.
The results of this study were striking. When the researchers asked TRUSS to identify vulnerable skills, it achieved perfect accuracy, catching every single malicious case without any false alarms within the 168 matched SkillInject artifacts used for the evaluation. In contrast, systems that only looked at the text of the instructions missed the majority of these threats. When the system was asked to fix skills that were already dangerous, it successfully reduced the rate of successful attacks by nearly half, all while ensuring the skills still completed their intended tasks. In a broader test involving the automatic creation of new skills for 187 different tasks, the framework increased the success rate of the agents from a low baseline to over half of all attempts, while simultaneously raising the safety pass rate to 100 percent on the benchmark evaluation. This means that for the first time, the system could generate skills that were not only better at doing the work but also met the safety evaluation criteria in the tested scenarios.
The researchers found that relying solely on static analysis, which is the practice of checking code or text without running it, was insufficient. Many dangerous behaviors only emerge when a skill interacts with a real environment, a nuance that text-based checks often miss. By insisting on evidence from actual execution, TRUSS was able to guide the creation of skills that are reliable in practice, not just in theory. The study demonstrates that automated skill generation can be made safe and effective, provided that the system verifies the behavior of the skill in a controlled setting before it is ever allowed to touch a user's data. This work suggests a path forward where artificial intelligence can safely expand its capabilities, giving agents new tools to solve problems without compromising the security of the systems they operate on.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.