💬 NLP

Safeguarding LLM Agents from Misalignment through Provenance Analysis

The paper introduces ProvenanceGuard, a provenance-based framework that significantly improves the detection of LLM agent misalignment and reduces false interventions compared to traditional LLM-as-a-judge baselines by verifying tool calls against traceable evidence in the agent's context.

Original authors: Yining She, Yiliang Liang, Eunsuk Kang

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yining She, Yiliang Liang, Eunsuk Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a very smart, eager personal assistant (an LLM Agent) to help you with your daily tasks. You tell them, "Please ask Daniel for $30." Because this assistant is connected to the real world, they can actually send emails, transfer money, or change files.

The problem is that this assistant is sometimes too eager or misunderstands you. Instead of asking for money, they might accidentally send money to Daniel. Or they might decide to pay a bill you didn't mention. This is called misalignment: the assistant does something that technically follows the rules but goes against what you actually wanted.

This paper introduces a new safety system called ProvenanceGuard to stop these mistakes before they happen. Here is how it works, explained simply:

The Core Idea: The "Paper Trail" Detective

The authors realized that existing safety systems are like a boss who just asks a second, equally confused assistant, "Is this a good idea?" This often leads to inconsistent answers.

Instead, ProvenanceGuard acts like a forensic detective. It doesn't just guess; it demands a paper trail (provenance). It asks: "Can you show me exactly where in our conversation or the instructions you found the evidence to do this specific action?"

If the assistant cannot point to a specific sentence in your request or a previous fact that justifies the action, the system blocks it.

The Three Types of Mistakes

The paper breaks down "bad ideas" into three categories, using a detective's checklist:

The Wrong Tool (Tool-Level):
- The Scenario: You say, "Request money." The assistant tries to use a "Send Money" tool.
- The Detective Check: "Does this tool match the job?" The system checks the tool's manual and your request. If the tool doesn't fit the job description, it's blocked.
The Wrong Details (Parameter-Level):
- The Scenario: You say, "Send $30 to Daniel." The assistant uses the right tool but sends $30 to Sarah because they guessed the name.
- The Detective Check: "Where did you get the name 'Sarah'?" If the assistant can't point to a fact in the conversation that says "Sarah," the system says, "No evidence found. Stop."
The Wild Guess (Interpretation-Level):
- The Scenario: You say, "Handle this request," without saying how. The assistant decides to automatically pay a bill.
- The Detective Check: "Is there only one way to 'handle' this?" The system realizes your request was vague. Since there are multiple ways to interpret "handle" (pay it, reject it, or ask you), the assistant is making a wild guess. The system blocks the action and forces the assistant to ask you for clarification instead.

How the System Works (The Three-Stage Pipeline)

ProvenanceGuard isn't a single question; it's a three-step security checkpoint that an action must pass before it can happen:

Checkpoint 1 (The Tool): "Is this the right tool for the job?" If not, STOP.
Checkpoint 2 (The Details): "Do you have proof for the specific numbers or names you are using?" If not, STOP.
Checkpoint 3 (The Interpretation): "Is this the only logical thing to do, or are there other possibilities?" If there are other possibilities, the system assumes the assistant is guessing and STOPS to ask for help.

The Results: Smarter and Less Annoying

The authors tested this system against 10 different "brain" models (LLMs) using two different sets of test scenarios.

Fewer Mistakes: Compared to the old method (just asking a second AI "Is this okay?"), ProvenanceGuard caught almost all the bad actions. On one test, it reduced the error rate from 43% down to 2%.
Less Annoyance: A safety system that blocks everything is useless because it stops the assistant from doing anything helpful. The authors found that ProvenanceGuard was very good at letting the correct actions pass through. It didn't get in the way of successful tasks, unlike some other methods that blocked good actions too often.

The Bottom Line

This paper proposes that instead of guessing whether an AI agent is doing the right thing, we should force it to prove it. By requiring the agent to show a "paper trail" connecting its action to your actual request, we can stop it from making dangerous, irreversible mistakes (like sending money or deleting files) without slowing down the helpful work it's supposed to do.

Note: The paper focuses strictly on preventing these specific types of misunderstandings in software agents. It does not claim to solve all AI safety issues, nor does it discuss medical or clinical applications.

Technical Summary: Safeguarding LLM Agents from Misalignment through Provenance Analysis

Problem Statement

As Large Language Model (LLM) agents evolve from conversational assistants to autonomous systems capable of executing irreversible actions (e.g., transferring money, modifying files), ensuring their actions align with user intent is critical. The paper identifies misalignment as a phenomenon where an agent's proposed tool invocation deviates from the user's intent, not necessarily due to malicious intent, but due to misunderstandings or unjustified assumptions about the user's request.

Existing runtime guardrails often rely on an "LLM-as-a-judge" paradigm, where a single LLM prompt evaluates whether an action should be allowed. The authors argue this approach lacks a systematic framework for reasoning about alignment, leading to inconsistent judgments, subjective rationales, and difficulties in auditing. Furthermore, traditional safety measures like sandboxing limit consequences rather than preventing the initial misaligned action, and post-hoc verification cannot undo irreversible tool invocations.

Methodology

1. Conceptual Framework: Provenance-Based Guardrails

Motivated by provenance analysis (the study of an artifact's origin and derivation history), the authors propose a framework where an action is considered aligned only if it is supported by traceable evidence within the agent's context. The context includes the user query ( $q$ ), tool documentation ( $d$ ), and interaction history ( $h$ ).

The framework formalizes misalignment detection by categorizing it into three distinct types, each corresponding to a failure in establishing specific provenance relations:

Tool-Level Misalignment: The agent selects a tool irrelevant to the task.
- Provenance Check: Does the selected tool ( $t$ ) have documentation ( $d_t$ ) that is RelevantTo a subtask ( $s$ ) that ContributesTo the user query ( $q$ )?
Parameter-Level Misalignment: The agent selects the correct tool but assigns parameters that are not grounded in the context.
- Provenance Check: Can the parameter assignment ( $a$ ) be DerivedFrom factual evidence ( $e$ ) appearing in the query or history? Additionally, does the instantiated tool ( $t$ with $a$ ) CanAddress the subtask ( $s$ )?
- Note: The framework distinguishes between derivable parameters (which require provenance) and generative parameters (e.g., drafting an email body), which do not require direct provenance tracing.
Interpretation-Level Misalignment: The agent resolves an underspecified query by arbitrarily committing to one interpretation when multiple plausible interpretations exist.
- Provenance Check: Is the proposed action the only provenance-admissible candidate given the context? If multiple distinct actions are supported by the context, the query is underspecified, and committing to one without clarification is misaligned.

2. Implementation: ProvenanceGuard

The authors implement this framework as ProvenanceGuard, a multi-stage runtime pipeline that intercepts tool calls before execution. Unlike a single-step "LLM-as-a-judge," ProvenanceGuard decomposes the decision into three sequential stages:

Stage I (Tool-Level): Filters for environment-changing tools and verifies if the selected tool is relevant to the agent's current subtask plan.
Stage II (Parameter-Level): Identifies which parameters are derivable and verifies if their values can be traced back to the context and if they correctly instantiate the tool for the subtask.
Stage III (Interpretation-Level): Asks the LLM (without revealing the agent's chosen action to avoid bias) if there exist other provenance-admissible actions. If multiple exist, the action is blocked as it relies on an unjustified assumption.

The system utilizes an LLM to evaluate the specific provenance relations (e.g., RelevantTo, DerivedFrom) but structures the reasoning process to be more explicit and auditable than a holistic prompt.

Key Contributions

Taxonomy of Misalignment: A formal distinction between tool-level, parameter-level, and interpretation-level misalignment.
Provenance-Based Framework: A conceptual model that frames alignment checking as determining whether tool calls are grounded in traceable evidence, formalized through specific binary relations.
ProvenanceGuard: A multi-stage pipeline implementation of the framework that operates as a runtime guardrail.
Benchmarking: Evaluation across two benchmarks (Agent-SafetyBench and WorkBench) and 10 backbone LLMs, including a manually annotated subset of Agent-SafetyBench traces.

Experimental Results

The authors evaluated ProvenanceGuard against a DirectPrompt Baseline (standard LLM-as-a-judge) and a Single-step Provenance Prompt (ProvePrompt, which uses the framework concepts in a single pass).

Misalignment Detection (Error Rate):
- On Agent-SafetyBench (focused on underspecification), ProvenanceGuard reduced the error rate on misaligned traces from 42.9% (Baseline) to 1.8%.
- On WorkBench, the error rate decreased from 32.1% to 17.3%.
- The multi-stage design (ProvenanceGuard) consistently outperformed the single-step variant (ProvePrompt), particularly on Agent-SafetyBench, suggesting the value of decomposing the reasoning process.
Intervention Cost (False Positives/Unnecessary Interventions):
- On Agent-SafetyBench (aligned traces), ProvenanceGuard did not introduce a statistically significant increase in unnecessary interventions compared to the baseline.
- On WorkBench (task-successful traces), ProvenanceGuard significantly reduced the intervention rate from 30.5% (Baseline) to 12.8%, indicating it is more precise than the baseline in allowing correct actions to proceed.

Significance and Claims

The paper claims that structured, provenance-based reasoning provides an effective and practical foundation for safeguarding LLM agents. The significance lies in:

Systematic Reasoning: Moving away from subjective, holistic judgments to a framework based on explicit, auditable dependency links between actions and context.
Handling Underspecification: The framework specifically addresses the risk of agents making unjustified assumptions on underspecified queries, a common cause of misalignment that standard safety filters often miss.
Balanced Performance: The multi-stage approach achieves high detection accuracy for misaligned actions without incurring a prohibitive cost in terms of blocking valid, successful agent tasks.

The authors conclude that while the framework cannot detect misalignment in purely generative parameters (like free-form text generation) and relies on the agent's own plan (which could theoretically be misaligned), it offers a robust mechanism for preventing irreversible, context-ungrounded actions in agentic systems.