Auditing LLM-mediated interpretation through source-to-claim documentation

In the humanities and social sciences, researchers have long relied on a simple but powerful idea: if you want to understand how a conclusion was reached, you must be able to trace the path from the original evidence to the final claim. This path is often called an "audit trail." It is a record of the decisions made, the notes taken, and the steps followed, ensuring that another scholar can look at the same materials and see how the interpretation was built. For decades, this process has been human-to-human, where a researcher reads a text, highlights a passage, and writes an analysis. But a new tool has entered the room: generative artificial intelligence. These systems can read a text and instantly write a fluent paragraph that summarizes or interprets it. The problem is that this speed comes with a hidden cost. The AI produces a smooth, polished sentence that hides the messy work of transformation that happened inside the machine. A reader sees the final claim but cannot see how the machine turned the original words into that specific conclusion. Worse, because these AI systems are hosted on servers owned by companies, the researchers cannot simply "re-run" the exact same calculation later. The software might change, or the same question might get a slightly different answer next time. This creates a gap in the audit trail, leaving the connection between the evidence and the claim fragile and difficult to verify.

To solve this, a team of researchers from the University of Minnesota and the Georgia Institute of Technology proposed a new way of working called "source-to-claim documentation." Instead of just saying, "We used an AI," they suggest keeping a complete, linked chain of records that shows exactly how a piece of evidence became an analytical claim. This framework does not just list the tools used; it preserves the specific journey of every single piece of generated text. The researchers tested this idea in a controlled experiment where they created twelve simple sentences to act as source material. They asked two different versions of an AI model to rewrite these sentences and to write short interpretive notes about them. The goal was not to see which AI was smarter, but to see if their new documentation method could catch errors that would otherwise go unnoticed. They found that the AI often made subtle but significant changes that looked harmless on the surface. For instance, when the source sentence said a woman "lent" her notes to a friend, the AI repeatedly rewrote it as "gave" her notes. In the real world, "lending" implies the item will be returned, while "giving" implies permanent ownership. The AI made this swap every single time, producing identical, fluent sentences that sounded correct but had actually altered the meaning of the original evidence.

The power of the new method became clear when the researchers looked deeper than the final text. They introduced a step where they asked the AI to create a structured map of the sentence's meaning alongside the rewritten text. This map acted like a blueprint, showing the relationships between the words. When the researchers compared the blueprint to the original sentence, the error jumped out: the map still showed the action as "lending," even though the final paragraph said "giving." This revealed that the AI had made a consistent mistake that the fluent prose had successfully concealed. The researchers also found that when the AI wrote interpretive notes, it sometimes added ideas that were not in the original text, such as guessing a person's motivation, even though it was required to quote the source directly. By keeping the original sentence, the AI's raw output, the structured map, and the human review all linked together in a single archive, the researchers could see exactly where the meaning shifted. They discovered that simply checking if two AI runs produced the same answer was not enough, because the AI could agree with itself while both being wrong.

The study involved eight human readers who reviewed the AI's work. These readers confirmed that while the AI's rewritten sentences often looked very similar to the original, they frequently changed the meaning in ways that mattered. In one specific case involving a sentence about a loan, the readers identified that the change from "lend" to "give" was a significant distortion in more than half of the evaluations. The researchers also tested whether their method could handle changes in the AI system itself. They ran the experiment with two different versions of the AI model, separated by months. Even though the models were different, the documentation method allowed them to trace the history of the claim and see how the evidence was transformed in each case. The key finding was that the framework successfully preserved the "evidential path." It allowed a later researcher to look at the final claim, follow the link back to the raw AI output, and then check the structured map to see if the meaning had been preserved. This process did not require the researcher to trust the AI or to believe the final text was perfect; it simply required them to have the full record of how the text was made.

This approach offers a practical way forward for researchers who want to use AI without losing the ability to verify their work. The researchers suggest that the amount of documentation should match the importance of the claim. For a private note or a classroom exercise, a simple record of the source and the prompt might be enough. But for a published article or a dataset meant for others to use, the full chain of evidence—including the raw output, the structured checks, and the human review—should be saved. This does not mean every single sentence needs a full audit, but it does mean that when a claim depends on an AI's interpretation, the path from the original words to that claim must remain visible. The study showed that without this visibility, fluent prose can easily hide errors, and repeated agreement between AI runs can give a false sense of security. By keeping the links between the source, the machine, and the human reviewer intact, researchers can ensure that their conclusions are built on a foundation that can be inspected, questioned, and understood by anyone who follows the trail.