💬 NLP

RedAct: Redacting Agent Capability Traces for Procedural Skill Protection

This paper introduces RedAct, a framework that protects procedural skills in AI agent execution traces by selectively redacting sensitive information while preserving audit evidence, effectively preventing unauthorized skill transfer without compromising accountability.

Original authors: Shuwen Xu (May), Zhitao He (May), Yi R. (May), Fung

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuwen Xu (May), Zhitao He (May), Yi R. (May), Fung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a master chef to cook a complex, secret family recipe for a dinner party. The chef leaves behind a video recording of the entire cooking process: the exact measurements they used, the specific brand of spices, the precise temperature of the oven, and the unique tricks they used to fix a burnt sauce.

If you post this video online so people can see how the chef did it (for transparency), a rival chef could watch it, copy the secret tricks, and start selling the same dish without ever having learned the recipe themselves. They didn't steal the physical recipe book; they just reverse-engineered the "how-to" from the video.

This paper, REDACT, tackles this exact problem for AI agents (smart computer programs that use tools like calculators, code editors, or search engines).

The Problem: The "Video Leak"

When AI agents solve hard problems (like analyzing medical data or fixing financial models), they leave behind a "trace"—a detailed log of every thought, tool click, and decision they made.

The Good: These logs help humans check if the AI is working correctly and fix bugs.
The Bad: These logs often contain the AI's "secret sauce"—proprietary formulas, specific thresholds, and clever strategies that the company owns. If released publicly, other AI systems can watch these logs, learn the secrets, and replicate the skills without paying for the original training.

The authors call this "Black-Box Trace Disclosure." It's like giving someone a map of your treasure hunt that shows exactly where the gold is buried, even if you don't give them the map's original key.

The Solution: REDACT

The team created a system called REDACT (Redacting Agent Capability Traces) to protect these secrets while still letting people see the work. Think of it as a smart editor that watches the cooking video and edits it before it goes online.

REDACT does two main things:

1. The "Smart Blur" (Selective Rewriting)

Instead of just blacking out the whole video (which makes it useless for checking if the chef actually cooked), REDACT uses a "Smart Blur."

What it keeps: It keeps the steps that prove the work was done. For example, "The chef checked the oven temperature" or "The sauce was verified as safe." This allows auditors to confirm the process was valid.
What it hides: It replaces the specific secrets with generic descriptions. Instead of "Add 3.42g of salt and simmer at 185°F," it says "Add the correct amount of seasoning and simmer at the appropriate temperature."
The Result: The video still looks like a real cooking show, but a rival chef can't copy the exact recipe anymore.

2. The "Invisible Ink" (Behavioral Watermarks)

To make sure people don't steal the video and pretend they made it themselves, REDACT adds "invisible ink" to the logs.

These aren't hidden words, but rather behavioral habits. For example, the AI might be programmed to always check the room temperature before starting a task, or to say a specific phrase like "Let's double-check the tools" after an error.
If someone tries to use the AI's skills, they might accidentally copy these habits. The original owner can then scan new AI behaviors to see if they contain these specific "ticks," proving the new AI learned from the original video.

The Test Kitchen (CAPTRACEBENCH)

To prove this works, the authors built a massive test kitchen called CAPTRACEBENCH.

They created 75 different complex tasks (like analyzing DNA sequences or fixing code) across 7 fields (like biology, finance, and engineering).
They included 154 specific "skills" (the secret recipes).
They let other AI systems try to learn from the "raw" videos versus the "REDACT-ed" videos.

The Results

The experiments showed that:

Raw videos are dangerous: When AI systems watched the unedited logs, they were able to copy about 45% to 67% of the secret skills. They basically became clones of the original expert.
REDACT stops the theft: After using REDACT, the ability of other AIs to steal the skills dropped to zero (or even below zero, meaning they performed worse than if they had no help at all). The "Smart Blur" successfully removed the reusable secrets.
The audit still works: Even with the secrets hidden, the logs still contained enough information for humans to verify that the work was done correctly.
The invisible ink works: The behavioral watermarks were detected 93% to 100% of the time when the skills were stolen, with almost no false alarms.

In Summary

The paper argues that releasing AI logs is like releasing a cooking video: it's great for transparency, but dangerous for intellectual property. REDACT is a tool that edits these logs to hide the "secret recipes" while keeping the "proof of cooking," and adds invisible markers to catch anyone who tries to copy the work. It allows companies to share their AI's work safely without giving away their competitive edge.

Technical Summary: REDACT – Redacting Agent Capability Traces for Procedural Skill Protection

1. Problem Motivation and Definition

As tool-using agents transition from research prototypes to deployed systems in domains like biomedicine, finance, and engineering, they increasingly rely on proprietary "skills"—structured packages of procedural knowledge including workflows, code templates, and domain heuristics. To ensure transparency and accountability, these agents often release execution traces (trajectories of reasoning, actions, and observations). However, this creates a fundamental tension: while traces support auditing, they also expose the underlying procedural skills.

Recent work on skill learning and evolution has demonstrated that agent experience can be distilled into reusable skills, significantly improving downstream task performance. This transforms trace release into a procedural asset-protection problem. A downstream party with access only to released traces may recover proprietary formulas, threshold configurations, tool-call sequences, and validation routines without needing access to model weights or private skill files.

The authors formalize this risk as Black-Box Trace Disclosure. In this setting, downstream reuse methods observe only released artifacts ( $z = \phi(x, \tau)$ ) and cannot access private skill files, model weights, hidden states, or server prompts. The central risk is not the copying of final answers, but the recovery of reusable protected procedures applicable across repeated deployments.

2. Methodology: REDACT Framework

To address this, the authors propose REDACT (Redacting Agent Capability Traces), a protected trace release framework operating in two complementary layers:

2.1 Key-Information-Guided Rewriting

This layer localizes and abstracts protected procedural information while preserving verifier-critical evidence. The process assumes the skill owner has access to the protected skill package ( $s$ ) before release.

Key-Item Localization: An LLM-based extractor ( $g_\eta$ ) identifies task-level protected items ( $K_{prot}$ ) from the task instruction and skill package. These items include formulas, constants, thresholds, specialized tool choices, and validation routines.
Human Review: A lightweight step deduplicates and canonicalizes the identified items (e.g., merging synonymous phrases).
Trace Rewriting: An LLM-based rewriter ( $\rho_\theta$ $ρ_{θ}$ ) constructs the protected trace ( $z_{RA}$ $z_{R A}$ ). It abstracts intermediate turns revealing protected key items into task-level descriptions while strictly preserving:
- Final outputs.
- Tool-use evidence (names, execution order).
- Verifier-critical fields (schema keys, required columns).
- The execution order necessary for auditing.

The result is a trace more informative than an "answer-only" release but less useful as a reusable procedure than a raw trajectory.

2.2 Behavioral Watermarking

To enable provenance tracking without affecting task semantics, the framework injects behavioral watermarks into selected traces.

Hook Families: The system defines classes of functionally neutral behaviors (hooks) that can be inserted and later detected.
- Standalone Hooks: Inserted at fixed positions (e.g., "Ritual Marker" at start/end, "Env Check" for environment probing).
- Contextual Hooks: Inserted only after specific conditions, such as tool results ("Cross Check") or error feedback ("Error Anchoring").
Injection Scheme: For a trace corpus $C$ and ratio $r$ , the system selects eligible traces and injects a harmless action-observation pattern along with an activation phrase ( $\kappa_h$ ).
Detection: At detection time, the skill owner queries a suspected student model using the activation phrase and applies a detector ( $W_{detect}$ ) to the generated trajectory to determine if the behavioral pattern was reproduced.

3. Benchmark: CAPTRACEBENCH

To evaluate this problem at scale, the authors constructed CAPTRACEBENCH, the first benchmark for procedural skill protection evaluation.

Scope: 75 specialized long-horizon tasks across 7 domains (Life Sciences, Chemistry, Earth & Climate, Engineering, Finance & Economics, Media & Web, Software & Security).
Skills: 154 curated skills covering diverse reuse scenarios.
Design: Tasks require multi-step tool use and include human-authored instructions, executable environments, and automatic verifiers. The benchmark is stratified into Easy, Medium, and Hard difficulty levels based on procedural complexity.
Evaluation Protocol: A "teacher" agent generates a trajectory. A disclosure policy releases a public artifact. A "student" agent attempts to solve the same task using only the released artifact. Success is measured by the student's ability to pass the automatic verifier, indicating whether protected procedural knowledge was successfully recovered.

4. Experimental Results

The authors evaluated REDACT against four downstream reuse methods: Single-Agent Skill Extraction, Multi-Agent Skill Evolution, Agent Workflow Retrieval Reuse, and Trajectory Fine-tuning.

4.1 Vulnerability of Raw Traces

Raw execution traces were found to be vulnerable attack surfaces. Across various models and reuse methods, raw traces allowed downstream agents to recover substantial procedural value:

Normalized Skill Transfer (NST): Raw traces yielded an NST of 44.7–67.1%, meaning downstream agents recovered a significant portion of the skill advantage provided by oracle access to private files.
Skill Recovery: Extracted skills from raw traces nearly closed the gap to oracle skills, achieving an average Step Success Rate (SSR) of 73.5% compared to the 76.5% of oracle skills.

4.2 Effectiveness of REDACT

REDACT significantly reduced the ability of downstream methods to reuse protected skills:

Skill Suppression: After rewriting, the average SSR for skill extraction, evolution, and retrieval dropped to 67.5%, 66.3%, and 65.5%, respectively. Crucially, these values fell to or below the No-Skill baseline (68.0%).
NST Reduction: The normalized skill transfer dropped from 44.7–67.1% (raw) to at most -5.9% (protected), indicating that the protected traces provided no procedural advantage over having no skills at all.
Information Leakage: Recovered Protected Information (RPI) decreased by 37–48% across channels.
Hard Tasks: The protection was most critical for hard tasks, where raw traces provided a 6.5–10.5% SSR gain over the baseline, which REDACT compressed to at most 4.8% (or below baseline for some methods).

4.3 Provenance Detection

Behavioral watermarks provided strong evidence of reuse:

True Detection (TD): Standalone hooks (e.g., "Env Check") achieved 93.6–100.0% true detection rates on student models.
False Alarm (FA): The false alarm rate remained low, at most 1.9%.
Contextual Hooks: While harder to detect (16.4–32.2% TD), they introduced zero false alarms, suggesting they are reproduced only when the student preserves specific execution contexts.

4.4 Audit Integrity

REDACT preserved the utility of traces for auditing:

Evidence Preservation: Final answers, tool names, verifier paths, and schema fields were retained at 91.0–96.6% of raw levels.
Operational Stability: Tool-call counts and trace lengths remained within the 90–110% range of raw traces.
Human Rating: Protected traces received high scores for naturalness (7.14/10) and audit usability (6.48/10) with high inter-annotator agreement.

5. Significance and Claims

The paper frames public agent traces as security interfaces. Its primary significance lies in demonstrating that selective redaction can effectively reduce procedural capability leakage without removing the audit evidence required for transparency.

Key claims include:

New Evaluation Problem: The formalization of reusable skill extraction from agent traces as a "black-box trace disclosure" problem.
Benchmark Creation: The introduction of CAPTRACEBENCH to evaluate procedural skill protection across diverse domains and difficulty levels.
Dual-Layer Defense: The demonstration that combining selective trace rewriting (to remove reusable content) with behavioral watermarking (to track provenance) offers a viable strategy for protecting proprietary agent skills.
Balanced Trade-off: The results show that it is possible to release traces that support debugging and reproducibility while stripping away the specific formulas, thresholds, and routines that constitute proprietary intellectual property.

The authors note limitations, acknowledging that their evaluation uses controlled tasks with automatic verifiers, whereas real-world deployments may involve noisier traces and weaker verifier feedback. They also emphasize that behavioral watermarks should be treated as statistical provenance evidence rather than cryptographic proof of ownership.