Quantifying Uncertainty in LLM-Extracted Causal Relations for Safety-Critical Decision Support
This research proposes and evaluates a comprehensive framework that quantifies uncertainty in LLM-extracted causal relations from safety-critical narratives using multi-source signals and conformal prediction, demonstrating that calibrated confidence estimates significantly improve extraction precision and enable cost-effective expert triage for trustworthy risk-informed decision-making.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the high-stakes world of nuclear power, aviation safety, and medical device monitoring, experts rely on written reports to understand what went wrong when things fail. These documents, often called event reports, are filled with complex stories about mechanical failures and human errors. For decades, the only way to find the hidden causes within these thousands of pages was to have human experts read every word, a slow and expensive process. Recently, powerful computer programs known as large language models have shown they can read these reports and automatically pull out the cause-and-effect relationships, such as identifying that a specific valve failure led to a pressure spike. This promises to speed up safety improvements dramatically. However, a major problem has held back their use: these computer programs act as if they are always certain. They present a discovered cause as a fact, even when the text is vague or the computer is guessing. In safety-critical fields, acting on a guess can lead to wasted resources or, worse, missed dangers. The core challenge is not just finding the answers, but knowing when the computer is confident enough to be trusted.
A researcher set out to solve this problem by teaching these computer programs to admit when they are unsure. They developed a new system designed specifically for nuclear power plant reports to measure how confident the computer really is about every cause-and-effect pair it finds. Instead of just spitting out a list of facts, the new system attaches a confidence score to each one, telling a human operator whether to accept the finding immediately or to flag it for a second look by a human expert. The researcher tested this approach using a large collection of real nuclear event reports. They found that by combining several different ways of checking the computer's work, they could create a reliable "uncertainty score." This score successfully distinguished between the computer's correct guesses and its mistakes, matching the judgment of human experts almost as well as the experts matched with each other.
The study revealed that the computer's confidence is not a single feeling but a result of checking its own work in multiple ways. The researcher built a pipeline that looks at the computer's internal confidence, asks the computer to read the same report ten times to see if it gives the same answer, and compares the results of three different versions of the computer trained separately. They also used a statistical method to guarantee that the confidence scores are mathematically sound. When they combined all these signals, the system became very good at sorting the results. The computer could identify a group of findings that were almost certainly correct, while isolating the messy, confusing ones that needed human attention. In tests, the system was able to spot 82 percent of the computer's errors within the first 20 percent of the reports it reviewed, whereas a random review would have found only 24 percent of the errors in that same amount of time.
This ability to sort the reliable from the unreliable has a direct impact on how safety models are updated. When the researcher fed the computer's findings into a risk model, they found that using only the high-confidence results led to a much more accurate picture of danger than using all the results or picking them at random. By filtering out the low-confidence guesses, the updated risk model was nearly twice as accurate as one built from a random selection of findings. The study showed that the computer's uncertainty scores were particularly good at flagging difficult cases, such as when the cause was implied rather than stated directly, or when the report described a chain of events spanning many sentences. These are exactly the situations where human experts are needed, and the system successfully identified them for review.
The research demonstrates that we do not have to choose between the speed of automation and the safety of human oversight. By giving these computer programs a way to express their uncertainty, we can use them to handle the bulk of the work while directing human experts to the specific cases where their judgment is most valuable. The findings suggest that this approach can make the process of updating safety models faster and more reliable, ensuring that the lessons learned from past accidents are incorporated correctly. The work does not claim that the computer is now perfect or that it understands the physics of a nuclear plant, but rather that it has learned to recognize the limits of its own understanding. This distinction is what allows the technology to move from a theoretical tool into a practical partner for safety professionals, providing a clear signal of when to trust the machine and when to call in a human.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.