💬 NLP

ProvenAI: Provenance-Native Traces of Evidence in Generated Answers

ProvenAI introduces a novel framework that enhances transparency in retrieval-augmented question answering by decomposing it into three independently measurable layers—answer correctness, citation fidelity, and per-document influence—to reveal critical gaps between cited sources and the evidence that actually drives model outputs.

Original authors: Mohammad Faizan, Dalal Alharthi

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad Faizan, Dalal Alharthi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you ask a very smart, well-read librarian to answer a difficult question. To do this, the librarian grabs a stack of ten books from the shelf, reads them, and writes you an answer. At the end of the answer, the librarian writes down a "References" list, but only mentions two of the ten books they actually held.

ProvenAI is a new system designed to audit exactly what happened in that scenario. It asks: Did the librarian actually use those two books to write the answer? Or did the other eight books secretly influence the story without getting a mention?

Here is a simple breakdown of how the paper works, using everyday analogies.

1. The Problem: The "Fake Citation" Illusion

In the world of AI, systems often show you a list of sources (citations) to prove they aren't making things up. But the paper argues that a citation list is like a receipt that doesn't match the shopping cart.

Just because the librarian says they used Book A and Book B doesn't mean those books actually shaped the story.

The Reality: The librarian might have been heavily influenced by Book C, D, and E (which they didn't list), while Book A and B were barely glanced at.
The Gap: The paper calls this the "Citation-Influence Gap." It's the difference between what the AI claims it used (the citation) and what the AI actually relied on to form its answer (the influence).

2. The Solution: ProvenAI's Three-Layer Audit

Instead of just checking if the answer is right, ProvenAI breaks transparency down into three separate layers, like checking a car in three different ways:

Layer 1: Is the Answer Correct? (The Destination)
- Analogy: Did the car actually arrive at the right address?
- What it checks: Does the final answer match the known truth?
Layer 2: Is the Citation Honest? (The Receipt)
- Analogy: Did the driver actually stop at the gas stations they listed on the receipt?
- What it checks: Do the books the AI listed actually contain the facts it claims?
Layer 3: What Actually Moved the Needle? (The Engine)
- Analogy: If we took the car apart and removed one specific engine part, would the car still run?
- What it checks: This is the most unique part. The system takes the AI's answer, removes one specific book from the stack, and asks the AI to write the answer again. If the answer changes, that book was influential. If the answer stays the same, that book was just "decoration."

3. The "Magic Trick" Experiment

The researchers tested this on a famous puzzle dataset called HotpotQA, where questions require connecting dots across multiple documents.

The Setup: They ran the AI on 7,405 questions.
The Surprise: The AI got the right answer about 53% of the time, but it cited the correct sources 71% of the time. This means the AI was often good at finding the right books, but bad at using them to write the final sentence.
The "Citation-Influence Gap" Case Study:
- They looked at a specific question about two movie directors.
- The AI cited two books perfectly. The "Receipt" was 100% clean.
- BUT, when they removed one of those cited books, the answer didn't change.
- AND, when they removed seven other books that the AI didn't even cite, the answer did change.
- Conclusion: The AI was secretly relying on the uncited books to get the answer right, while the cited books were just "window dressing."

4. Why This Matters (Without the Jargon)

The paper argues that we cannot just look at one score (like "Accuracy") to trust an AI. We need to see the whole picture:

Did it get the right answer?
Did it list the right sources?
Did those sources actually do the heavy lifting?

If you only check #1 and #2, you might think the AI is trustworthy. But if you check #3, you might realize the AI is actually being swayed by "ghost" documents it never told you about.

5. The "Black Box" Limitation

The paper admits a small limitation: To measure influence perfectly, you usually need to see the AI's internal "thought process" (math probabilities). Since their system couldn't see inside the AI's brain, they used a clever workaround: they just compared the final sentences before and after removing a book.

Analogy: Instead of listening to the engine's internal combustion, they just checked if the car moved differently when they removed a part. It's not as precise as looking under the hood, but it's a very good guess.

Summary

ProvenAI is a tool that stops us from being fooled by a pretty list of citations. It proves that in AI, what is cited is not always what is used. By testing what happens when you remove a source, it reveals the hidden "ghosts" in the machine that are actually driving the answers.

Technical Summary: ProvenAI

Problem Statement
Retrieval-augmented generation (RAG) systems routinely provide citations alongside generated answers to establish factual grounding. However, the paper argues that the presence of a citation does not confirm that the cited source meaningfully shaped the output. Current evaluation metrics often collapse answer correctness, citation fidelity, and resource influence into single scores, obscuring critical diagnostic information. Specifically, a model may cite correct sources while being driven by uncited retrieved context, or it may generate a correct answer while citing irrelevant sources. This gap between "cited" evidence and "influential" evidence is particularly problematic in security and forensic applications where accountability requires a verifiable record of what actually shaped a conclusion.

Methodology
ProvenAI is not a new generative model but a measurement infrastructure designed to decompose transparency in multi-hop question answering into three independently measurable layers. The system operates on the HotpotQA distractor benchmark using a seven-stage pipeline:

Data Normalization: Converting HotpotQA records into stable local artifacts (JSONL) and building a canonical retrieval corpus of 509,300 passages.
Retrieval & Generation: Using dense passage retrieval (FAISS with all-MiniLM-L6-v2 embeddings) and a local inference path (Qwen2.5-3B-Instruct) to generate citation-aware answers with inline tags.
Citation Auditing: Evaluating whether cited titles align with gold supporting facts using Jaccard similarity.
Influence Estimation: Applying a "leave-one-resource-out" ablation strategy. For each retrieved document $r_i$ , the system regenerates the answer and citation set without $r_i$ to measure the shift in output.
Verdict Assignment: Documents are classified into four categories based on a measurable function $V$ $V$ combining citation status and an influence proxy $\phi(r_i)$ $ϕ (r_{i})$ :
- USED: Cited and influential.
- HALLUCINATED CITATION: Cited but not influential.
- UNCITED INFLUENTIAL: Not cited but influential.
- LOW INFLUENCE: Neither cited nor influential.

Key Technical Components

Citation Fidelity: Measured via token-set Jaccard similarity between cited titles and gold supporting titles, evaluated under both exact ( $\tau=1$ ) and semantic ( $\tau=0.34$ ) matching.
Influence Proxy: Since the local inference backend (MLX) does not expose per-token probabilities required for direct KL-divergence calculation, the system uses a surface proxy $\phi(r_i)$ . This proxy aggregates the Jaccard distance between the original and ablated answer strings ( $\Delta a$ ) and the citation sets ( $\Delta c$ ), weighted 0.8/0.2 respectively.
Theoretical Grounding: The paper formalizes the relationship between this surface proxy and the token-level KL-divergence target. Proposition 1 establishes that under near-deterministic decoding, a non-zero answer change ( $\Delta a > 0$ ) certifies a strictly positive KL shift, making the proxy a sound (though one-sided) indicator of distributional displacement.
Traceability: The system integrates the Model Context Protocol (MCP) to record trace graphs of resource access, linking retrieval artifacts to the generation process.

Results
Evaluated on 7,405 validation examples from the HotpotQA distractor split, ProvenAI yields the following aggregate metrics:

Answer Accuracy: 53.53%
Mean Citation Fidelity: 71.55%

The significant 18-point gap between citation fidelity and answer accuracy demonstrates that models frequently identify and cite correct supporting titles even when the final answer string is incorrect. Conversely, the system identifies cases where the answer is correct but the citation set is degraded.

The paper highlights a specific "citation-influence gap" through a case study (Scott Derrickson / Ed Wood nationality). In this example, the model achieved a perfect citation audit ( $F_\tau = 1.0$ ), yet ablation revealed that one cited source had weak influence while seven uncited sources demonstrably shifted the output. This confirms that the observable citation set and the empirical sensitivity profile can diverge substantially.

Significance and Claims
The paper claims that meaningful transparency in retrieval-grounded QA requires traceable links across three distinct layers: retrieved evidence, cited evidence, and behaviorally influential evidence.

Decomposition of Transparency: The primary contribution is the operationalization of transparency into three independent layers, arguing that collapsing them discards necessary diagnostic information.
The Citation-Influence Gap: The work characterizes a specific failure mode where clean citation audits coexist with complex influence profiles driven by uncited context.
Provenance-Native Reasoning: The framework connects measurement-based auditing to cryptographic provenance architectures. It argues that for autonomous scientific discovery, an audit trail must not only verify that cited documents were retrieved (cryptographic integrity) but also verify that the output was actually sensitive to those documents (behavioral sensitivity).
Limitations: The authors modestly note that the current influence estimate relies on a surface proxy due to the lack of per-token logits in the local backend, and attribution is currently limited to the document-title level rather than sentence-level entailment.

ProvenAI establishes that current citation mechanisms are insufficient proxies for influence and provides a reproducible pipeline to measure the discrepancy between what a model claims to use and what actually drives its output.