💻 computer science

Does Provenance Add Value Beyond Matched Skill Guidance?

This preregistered study finds that provenance metadata in agent skills does not reliably improve task utility beyond matched ordinary guidance, suggesting its value is context-dependent and primarily realized only when it alters executable actions rather than through bundled skill scores alone.

Original authors: Haocheng Lu

Published 2026-09-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haocheng Lu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new generation of computer programs has emerged that does not just answer questions but takes actions. These systems, often called agents, can browse the web, write code, or manage files by using a set of digital tools. To make these agents smarter and more reliable, developers often feed them "skills." Think of a skill as a pre-written instruction manual that tells the agent how to solve a specific type of problem. These manuals are frequently bundled with extra information called provenance. This metadata acts like a digital receipt or a citation, showing the agent where the instruction came from, what evidence supports it, and under what conditions it applies. While this extra context is undeniably useful for humans who need to audit or debug the system, a lingering question has remained for researchers: does this extra information actually help the computer agent perform the task better, or is it just noise that the agent ignores?

A researcher at New York University set out to answer this question with a rigorous experiment designed to separate the value of the instruction from the value of its history. They wanted to know if an agent could solve a problem more effectively when given a skill that included source details, compared to a version of the same skill where those details were replaced with a blank placeholder. To do this, they created a controlled environment where the agent faced 250 distinct tasks, ranging from managing file conflicts to handling memory errors. For every single task, the agent received the exact same core instructions, the same tools, and the same configuration. The only difference was the content of the skill slot: one version contained the full provenance with source and evidence fields, while the other contained a fixed-width blank space that took up the same amount of space in the message but held no meaningful information.

The results of this direct comparison were surprising and counterintuitive. When the researcher measured the success rate of the agents, the version with the full provenance information actually performed slightly worse than the version with the blank placeholder. Specifically, the agent with the extra history succeeded on 38.8 percent of the tasks, while the agent with the simplified, history-free instruction succeeded on 42.0 percent. This represents a drop of 3.2 percentage points for the version with the provenance. The researcher was careful to account for the fact that the version with provenance was longer in terms of the number of words or tokens the computer had to process. They ran a second, separate experiment with 100 tasks where they matched the length of the messages exactly, ensuring the agent received the same amount of text in both cases. In this length-matched scenario, the version with provenance showed a small improvement of 2.0 percentage points, but the data was not precise enough to confirm this as a real benefit rather than random chance.

The study suggests that the apparent value of adding source history to an agent's instructions is not a straightforward benefit. In the primary test, the extra information did not help the agent succeed; in fact, it seemed to slightly hinder performance, likely because the additional text distracted the model or altered how it processed the instructions. The researcher found that the outcome depended heavily on how the experiment was designed. When they constructed a specific scenario where the correct action was impossible to determine without the source history, the provenance became critical, and the agent with the history succeeded far more often. However, in the general pool of tasks where the correct action was clear from the instructions alone, the extra metadata provided no advantage and sometimes caused a decline in performance.

This work challenges the common assumption that more information is always better for artificial intelligence. The researcher concludes that the utility of provenance cannot be judged by looking at a bundled skill score that mixes instructions with history. Instead, the value of these source details changes depending on the specific context and how the information is presented. For developers building these systems, the finding implies that simply adding citations or source logs to every instruction may not improve the agent's ability to complete a task and could even be detrimental if the agent gets confused by the extra text. The study advocates for a more careful approach, where the necessity of provenance is tested against the specific needs of the task, rather than assuming it is a universal upgrade for all AI agents.

Technical Summary: Does Provenance Improve Terminal Task Success Beyond Matched Skill Guidance?

Problem Statement
Tool-using Large Language Model (LLM) agents increasingly rely on external skills derived from traces, source files, and prior runs. These "skills" often bundle ordinary task guidance with metadata fields such as source events, applicability conditions, and evidence references (provenance). While provenance aids auditing, its specific contribution to terminal task success is difficult to isolate. Current evaluation methods typically measure the utility of the entire skill package, creating an attribution problem where improvements cannot be distinguished from the guidance itself, context length changes, or serialization effects. The paper asks: once ordinary guidance is held fixed, does the inclusion of provenance metadata improve an agent's ability to complete a task?

Methodology
The study employs a controlled, paired intervention design to isolate the utility of provenance fields.

Experimental Design: For each task, two arms receive identical ordinary guidance, tools, student configuration, and execution policies. The only variable is the content of the "evidence-bearing slot":
- Arm P (BESD): Contains source/evidence and applicability fields (provenance).
- Arm M (SkillGenMatched): Contains a fixed-width placeholder (no provenance semantics).
Benchmarks and Splits: The study utilizes a source-local benchmark with 834 tasks across ten scenario families (e.g., workspace writes, retries, permissions). The primary analysis focuses on 250 task-ID-held-out tasks. A separate exploratory amendment uses 100 tasks matched on provider-reported input token length to control for context length.
Intervention Contract: The system uses a Harness Evidence Graph to ensure observable evidence is closed. The compiler admits clauses only when evidence is visible, preventing inference of inaccessible provider context.
Statistical Analysis: The primary estimand is the finite-population paired contrast ( $\Delta_{P-M}$ ). The study reports absolute success rates, paired differences, and exact two-sided McNemar tests. Confidence intervals are generated via task-level bootstrapping (10,000 draws) and cluster-aware resampling to account for dependence in task pairs.
Control Variants:
- Provider-Token-Matched: A placebo payload matched to the input length of the provenance arm to isolate token-count effects.
- Mechanism Controls: Variants including correct, wrong, shuffled, and irrelevant provenance to test semantic specificity.
- Opportunity Cell: A constructed subset where provenance explicitly determines the oracle action, testing the boundary of decision relevance.

Key Results

Primary Payload-Matched Comparison: In the primary 250-task cell, BESD (provenance) achieved a 38.8% success rate compared to 42.0% for SkillGenMatched (no provenance). The paired estimate was -3.2 percentage points (95% task-bootstrap interval: [-6.0, -0.4]). This indicates that, under payload matching, adding provenance did not improve terminal success and may have slightly degraded it, potentially due to context length or serialization overhead.
Provider-Token-Matched Amendment: In a separate 100-task cell where input lengths were matched, the estimate shifted to +2.0 percentage points (interval: [-2.0, +6.0]). The interval spans zero, meaning this result is imprecise and does not establish a statistically significant positive benefit.
Mechanism Controls: In the 100-task mechanism cell, "CorrectProvenance" performed similarly to "Wrong," "Shuffled," and "Irrelevant" provenance payloads (all showing negative or neutral contrasts relative to the baseline). This suggests that the observed effects in the primary cell were not driven by the semantic correctness of the provenance but likely by nuisance factors like context length.
Family Heterogeneity: The negative contrast was heavily concentrated in the retry_wrapper family (-36.4 points), while other families showed zero difference. This suggests the effect is scenario-dependent rather than a universal mechanism.
Opportunity Boundary: In a constructed cell where the oracle action depended on the provenance label, "CorrectProvenance" succeeded on 50/50 tasks, compared to 25/50 for the "SkillGenMatched" baseline. This demonstrates that provenance can be decision-relevant, but only when the task design explicitly requires it to select between otherwise indistinguishable actions.

Key Contributions

Evaluation Protocol: The paper introduces a protocol for separating "whole-skill utility" from "provenance utility" by treating context matching as part of the estimand.
Attribution Clarity: It demonstrates that bundled skill scores conflate guidance utility with metadata utility. The study shows that provenance utility is not a fixed property but changes based on the matching regime (payload vs. token) and task design.
Controlled Comparisons: The work provides a rigorous framework for auditing agent skills, emphasizing the need for guidance-matched comparators and provider-token balance reporting.

Significance and Claims
The paper concludes that provenance utility cannot be inferred from a bundled skill score. The apparent value of provenance is contingent on how the comparison is controlled:

When context length is uncontrolled (payload matching), provenance fields may introduce noise or overhead, leading to a negative or neutral impact on terminal success.
When context length is controlled (token matching), the effect becomes uncertain (spanning zero).
Provenance only yields a large, positive utility in specific "opportunity" conditions where the metadata is required to determine the correct executable action.

The author argues that for evaluators, the standard practice must shift toward publishing guidance-matched comparators, auditing provider-reported tokens, and explicitly stating whether a task oracle makes provenance decision-relevant. For practitioners, provenance remains valuable for auditability and debugging, even if it does not consistently improve terminal task success in current agent architectures. The study is modest in its claims, noting that results are bounded to the source-local, fixture-conditioned setting and do not generalize to unconstrained applications without further stress testing.

Technical Summary: Does Provenance Improve Terminal Task Success Beyond Matched Skill Guidance?

More like this