🤖 AI

Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems Using Bayesian Networks

This paper proposes a multi-agent framework for actuarial risk modeling that leverages token-level log-probabilities and Bayesian Networks to quantify and propagate runtime uncertainty, ensuring reliable decision-support in high-stakes LLM-based workflows.

Original authors: Bart Custers, Koorosh Aslansefat

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bart Custers, Koorosh Aslansefat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just follow strict instructions but can actually "chat" and "think" their way through complex jobs. This is the realm of Artificial Intelligence, specifically a branch called Large Language Models (LLMs). Think of these models as incredibly well-read students who have read almost everything on the internet. They are great at writing stories, answering questions, and even solving math problems. However, there's a catch: they are a bit like a creative writer who sometimes makes things up (called "hallucinations") or changes their mind if you ask the same question twice.

Now, imagine you need to make a life-or-death decision, like figuring out how much to charge for car insurance. You can't just ask one of these AI students to guess; you need a whole team working together. This is where Multi-Agent Systems come in. Instead of one robot doing everything, you have a team of specialists: one gathers the data, another builds the math model, a third checks for mistakes, and a fourth explains the results. The big question scientists are asking is: "If one of these AI team members gets confused or makes a mistake, how do we know before the whole project fails?" This paper dives into that exact problem, trying to build a safety net for AI teams working on high-stakes financial tasks.

The AI Team and the "Confidence Meter"

In this study, the researchers set up a digital insurance company run by a team of AI agents. They didn't just let the AI chat randomly; they built a strict workflow with a "Central Hub" acting like a project manager. This manager assigns tasks to four specific AI specialists:

The Data Prep Agent: Cleans up the messy numbers.
The Modelling Agent: Builds the risk prediction math.
The Reviewing Agent: Double-checks the work for errors.
The Explanation Agent: Makes sure the results make sense and are fair.

The tricky part is that these AI agents are probabilistic, meaning they don't give a single "right" answer every time. They generate text word-by-word, and for every word, the computer has a tiny "log probability"—a number that says how likely the model thought that specific word was to come next. Usually, people just look at the final answer. But this paper suggests a clever trick: listen to the whisper of the AI's confidence.

The authors realized that you can't just take the AI's internal confidence numbers and say, "This is 90% correct." That's like assuming a student who speaks fluently is definitely telling the truth. Instead, the team created a system to translate those raw confidence whispers into a Bayesian Network.

The Bayesian Network: A Digital Ripple Effect

Think of the Bayesian Network as a giant, interactive flowchart of dominoes. Each domino represents a step in the insurance workflow. If the first domino (Data Prep) wobbles, it might knock over the second (Modelling), which could topple the third (Reviewing).

The researchers used the AI's internal confidence scores to set the "wobble factor" for each domino. They didn't treat the AI's confidence as a guarantee of truth. Instead, they calibrated it using test runs to see how often the AI was actually right when it felt confident. Then, they fed these calibrated numbers into the network.

The result? The network could show them how uncertainty ripples through the whole system. Even if every single AI agent felt 80% confident in their own work, the network could calculate that the entire workflow only had a 55% chance of being successful because the small doubts added up. It's like a chain of four people passing a bucket of water; if each person is 80% sure they won't spill, the final bucket might still be half-empty by the time it reaches the end.

What They Found: The Temperature Test

To see if their safety net worked, the researchers ran the AI team through different "temperatures." In AI-speak, "temperature" controls how random or creative the AI is.

Low Temperature (0.2): The AI is very conservative and repetitive.
High Temperature (1.2): The AI is wild, creative, and prone to making mistakes.

They tested four scenarios:

Safe Scenario: Everyone was low-temperature. The system was very reliable, with an 87.1% chance of success.
Mixed Scenario: The data and model agents were calm, but the reviewers and explainers were a bit wild. Success dropped to 69.0%.
Moderate Scenario: Everyone was at a standard "medium" setting. Success was 59.2%.
Risky Scenario: Just one agent (the Explanation Agent) got very wild. Even though everyone else was calm, the whole system's success rate plummeted to 42.8%.

This showed that the system is very sensitive. A single shaky link in the chain can drag down the reliability of the whole project.

The Verdict: Who Caught the Mistakes?

The team also tested the AI agents by secretly messing up the data (like hiding numbers or adding fake noise) to see if the agents would notice.

Llama 2 and Llama 3.1: These models were okay at the job but missed a lot of the hidden errors. Llama 2 only caught 45% of the mistakes, and Llama 3.1 caught 65%.
Qwen2.5: This model was the star of the show. It caught 90% of the errors and was much better at adapting its strategy when things went wrong.

The study also found that while the individual agents often felt very confident (scoring around 0.80 to 0.90 on their own), the combined uncertainty of the whole workflow was much lower. For example, with the Qwen model, the individual agents were confident, but the final workflow certainty was only 0.6012.

Why This Matters

This paper doesn't claim to have solved all the problems of AI in insurance. Instead, it suggests a new way to watch the AI team while they work. By using a Bayesian Network to track how small doubts add up, we can spot when a workflow is becoming "unsafe" before it produces a bad result.

The researchers showed that this method can reproduce standard actuarial results (the math used to price insurance) while giving us a clear view of the system's stability. It suggests that in high-stakes jobs, we shouldn't just trust the AI's final answer; we need to listen to the "whispers" of its confidence and watch how those whispers travel through the team. If one agent starts sounding unsure, the whole system should know to pause and check, preventing a small error from becoming a financial disaster.

Technical Summary: Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems Using Bayesian Networks

Problem Statement

The integration of Large Language Model (LLM)-based Multi-Agent Systems (MAS) into high-stakes domains like actuarial risk modelling presents significant challenges regarding reliability and transparency. While LLMs offer the potential to automate complex workflows involving data preparation, model development, and validation, their probabilistic nature introduces risks such as hallucinations, variability across runs, and a lack of explainability. In regulated insurance environments, unreliable outputs can lead to incorrect risk assessments, unfair pricing, and regulatory non-compliance. A critical gap exists in current approaches: existing evaluation metrics often focus on task-specific accuracy but fail to capture how uncertainty arises and propagates through interdependent agent workflows. Furthermore, standard uncertainty estimates (such as token-level log-probabilities) do not directly translate to the probability of task success, nor do they account for the cumulative effect of errors across a multi-step process.

Methodology

The authors propose a runtime safety-monitoring framework designed to quantify and propagate uncertainty within an LLM-based MAS. The methodology consists of three core components:

1. Multi-Agent System Architecture

The system employs a centralized star topology where a Central Hub coordinates four specialized agents via message passing, preventing direct agent-to-agent communication to ensure controlled oversight:

Data Preparation Agent: Handles data ingestion, cleaning, and encoding.
Modelling Agent: Develops predictive models.
Reviewing Agent: Evaluates outputs from the data and modelling stages for consistency and performance.
Explanation Agent: Assesses interpretability, concept alignment (using Testing with Concept Activation Vectors), and fairness (via group-level comparisons).

The framework utilizes three open-source, decoder-only transformer models (Llama 2 7B, Llama 3.1 8B, and Qwen2.5 7B) capable of running on local machines.

2. Uncertainty Quantification via Log-Probabilities

The framework extracts token-level log-probabilities ( $log p_\theta$ ) from LLM outputs. Recognizing that raw log-probabilities are influenced by sequence length and do not directly equate to task success, the authors apply a transformation process:

Length Normalization: Log-probabilities are averaged over the sequence length $L$ to generate a summary score $s_i$ .
Calibration: The normalized score is transformed into a bounded confidence feature $q_i = \exp(s_i)$ .
Calibration Function: A function $f_{cal}$ , estimated via validation runs, maps $q_i$ to a probability of successful task completion $P(X_i = success | q_i)$ .

3. Bayesian Network (BN) for Propagation

To model the dependencies between workflow stages, the calibrated confidence scores are incorporated into a Bayesian Network.

Structure: Each workflow stage (e.g., data preparation, review, modelling) is a node. Directed edges represent conditional dependencies.
Logic: The network uses Conditional Probability Tables (CPTs) to define how the success of a downstream task depends on the success of its parent nodes. For instance, the final workflow success node is modeled as an "AND" gate, requiring all parent nodes to succeed.
Inference: As the workflow executes, the Central Hub updates the BN with new evidence (uncertainty scores), allowing for the calculation of posterior probabilities. This visualizes how uncertainty accumulates from individual agent outputs to the final system state.

Key Contributions

The paper presents a novel approach to runtime monitoring that moves beyond static evaluation:

Uncertainty Propagation Framework: It introduces a method to transform token-level log-probabilities into calibrated task-level confidence estimates and propagate them through a Bayesian Network. This explicitly models the distinction between intrinsic uncertainty (at a single step) and extrinsic uncertainty (propagated from earlier decisions).
Structured Safety Mechanism: The framework provides a structured representation of workflow stability, capable of identifying "unsafe" states where uncertainty in specific components significantly degrades overall system reliability.
Empirical Validation in Actuarial Context: The approach is validated within a realistic actuarial workflow, demonstrating that the system can reproduce baseline actuarial performance while offering granular insights into workflow stability.

Results

The evaluation was conducted across three LLM backends using original and 20 systematically perturbed datasets (introducing errors such as missing values, distribution shifts, and label noise).

Performance: The MAS achieved high task success rates (80–90%) and predictive performance (RMSE) comparable to baseline actuarial models across all backends.
Uncertainty Variance: Significant differences were observed in workflow certainty derived from the Bayesian Network. For example, while individual agents often showed high confidence, the propagation of uncertainty reduced the overall workflow success probability (e.g., from ~80% individual certainty to ~55% overall certainty in a default scenario).
Temperature Sensitivity: Experiments varying LLM temperature confirmed that increased randomness in agent responses directly reduces the probability of overall workflow success. A "risky" scenario with high temperature in a single agent significantly lowered the reliability of the entire system.
Error Detection: The Qwen2.5 7B backend demonstrated superior error detection (90%) and adaptability compared to Llama models (45–65%), particularly in handling distributional shifts.
Limitations: The system showed sensitivity to distributional changes (population shifts) but was less reliable in detecting missing values or sparsity, highlighting specific areas for improvement in the validation process.

Significance and Claims

The paper claims that its primary significance lies in providing a structured, interpretable method for monitoring runtime uncertainty in LLM-based MAS. By combining token-level signals with Bayesian inference, the framework offers a safety mechanism that can identify weak elements in a workflow before they lead to critical failures.

The authors maintain a modest stance, noting that the current implementation represents a proof-of-concept. The Bayesian Network models an abstracted execution trace rather than the full dynamic control logic (e.g., it does not currently model iterative revision loops or temporal ordering beyond direct parent-child dependencies). The authors suggest that extending the model to dynamic Bayesian Networks or influence diagrams is a direction for future work. Ultimately, the work demonstrates that it is possible to support high-stakes decision-making with LLM agents by explicitly quantifying and visualizing how uncertainty propagates through complex, multi-step workflows.