💻 computer science

A Scalability Analysis of Quantitative Confidence Assessment Methods for Assurance Cases

This paper introduces a model to estimate the decision complexity and effort of applying quantitative confidence assessment methods to assurance cases, demonstrating through analysis of Bayesian Belief Network, Dempster-Shafer Theory, and Certus methods that while Certus has the highest worst-case complexity, it requires less average-case effort than the other two approaches.

Original authors: Simon Diemert, Jens H. Weber

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Simon Diemert, Jens H. Weber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, intricate argument to prove that a self-driving car is safe. You have a top-level claim ("The car is safe") supported by layers of evidence, like "The brakes work" and "The sensors are calibrated." This is called an Assurance Case.

The problem is: How do you know if your argument is actually convincing? Just writing it down isn't enough. You need a way to measure your "confidence" in the conclusion. This is where Quantitative Confidence Assessment Methods come in. They are like calculators for trust, turning your written arguments into numbers or scores.

However, there's a catch. Using these calculators is hard work. It takes a lot of time and mental energy to fill them out. The authors of this paper wanted to answer a simple question: "How much work does it actually take to use these different calculators as your argument gets bigger?"

To find out, they built a mathematical model (a simulation) to count the "decisions" a person has to make. Think of it like counting how many buttons you have to press on a remote control to change the channel.

The Three "Calculators" They Tested

The paper compares three different ways to calculate confidence:

The BBN Method (Bayesian Belief Network): Think of this like a flowchart with probabilities. You have to decide how likely each piece of evidence is (0% to 100%) and then decide how much each piece influences the next one. It's like setting the rules for a game of dominoes.
The DST Method (Dempster-Shafer Theory): This is like a detailed survey. For every piece of evidence, you have to give two answers: a "decision" (how acceptable is this?) and a "confidence" (how sure are you?). It requires more buttons to press than the flowchart.
The Certus Method: This is the customizable toolkit. Instead of just numbers, you can use words like "certain," "skeptical," or "reject." It's the most flexible, allowing you to write your own custom rules for how evidence combines. But because it's so flexible, it can get very complicated very quickly.

The Experiment: The "Tree" Analogy

To test these methods, the authors imagined their arguments as trees.

The roots are the final conclusion (Is the car safe?).
The branches are intermediate claims.
The leaves are the raw evidence (documents, test results).

They asked: "If we make this tree taller and wider (more evidence), how many 'button presses' (decisions) does a human have to make?"

They looked at two scenarios:

The "Worst Case" (The Perfectionist): Imagine a user who refuses to use any shortcuts. They manually configure every single rule, weight, and connection for every piece of evidence.
The "Average Case" (The Realist): Imagine a user who uses smart tools. They only manually configure the tricky parts and let the software fill in the boring, standard parts with default settings.

The Surprising Results

The study found some interesting things about how these methods scale (grow) as the argument gets bigger:

In the "Worst Case" (No Shortcuts):
Certus was the hardest. Because it allows for so much customization, if you try to configure every single rule manually, you end up with a massive number of decisions. It's like trying to build a house by hand-crafting every single brick and nail.
BBN and DST were much easier in this scenario because they have stricter, simpler rules.
In the "Average Case" (Using Shortcuts):
Certus actually became the easiest to use! Because it has built-in "macros" (pre-made shortcuts) and smart defaults, a user can skip most of the hard work.
BBN was in the middle.
DST remained the hardest, even with shortcuts, because it still requires you to make two separate decisions (decision + confidence) for every single piece of evidence.

The Bottom Line

The paper concludes that flexibility has a price tag, but only if you pay it in full.

If you try to do everything manually, the most flexible tool (Certus) is the most exhausting. But if you use the tool the way it's intended (with smart defaults and shortcuts), the flexible tool actually saves you the most time.

The authors warn that for very large safety arguments (like those for self-driving cars or medical devices), the "effort" required to use these tools can be a barrier. Their model helps researchers design better tools that don't require humans to press too many buttons, ensuring that safety arguments remain practical and not just theoretical exercises.

Technical Summary: A Scalability Analysis of Quantitative Confidence Assessment Methods for Assurance Cases

Problem Statement

Preparing Assurance Cases (ACs) is a critical activity for safety- and security-critical systems, yet the use of structured notations like Goal Structuring Notation (GSN) does not guarantee that top-level claims are adequately supported. Confidence Assessment Methods (CAMs) provide systematic means to evaluate these arguments. While qualitative, quantitative, and mixed CAMs exist, a recent survey of practitioners identified the additional effort required to apply these methods as a primary barrier to adoption.

Specifically, for quantitative or mixed methods, significant effort is required to assign computation parameters (e.g., edge weights) and provide assessments for every piece of evidence. Previous work by Graydon and Holloway noted a lack of empirical scalability data for these techniques, raising concerns about their feasibility in practice. Consequently, there is a need for a model to estimate the "decision complexity" and effort required to apply quantitative CAMs as argument size increases.

Methodology

This paper proposes a numerical model to estimate the decision complexity (the number of decisions a user must make) and the level of effort (time required) for applying quantitative CAMs. The analysis considers both worst-case and average-case scenarios.

1. Argument Structure Model

To facilitate scalability analysis, the authors model an AC argument as an $n$ -ary tree of claims with height $h$ , where each leaf claim is supported by $m$ evidence nodes. While real-world ACs have diverse structures, this idealized model allows for meaningful comparison. Parameters $n$ (average child claims per step) and $m$ (evidence nodes per line of reasoning) are derived from published AC case studies, yielding $n \approx 1.72$ and $m \approx 1$ .

2. Decision Complexity and Effort Models

The model defines:

$p$ : Decisions per parent claim (configuring belief propagation).
$v$ : Decisions per evidence node (leaf valuation).
$\alpha$ : The proportion of cases requiring non-trivial, custom decisions (set to 0.20 for average-case analysis, based on prior expressivity studies).
$t_p$ and $t_v$ : Time estimates per decision (0.5 minutes for propagation, 5 minutes for evidence valuation).

The worst-case decision complexity ( $O_d$ ) assumes every parameter must be manually specified. The average-case complexity ( $\Omega_d$ ) assumes that for a proportion $(1-\alpha)$ of steps, tooling or sensible defaults reduce the decision load to $p'$ and $v'$ .

3. Methods Analyzed

The model is applied to three representative quantitative CAMs:

Bayesian Belief Network (BBN): Uses probability theory with parameters for combinator (NoisyAND/OR), leakage, and link weights.
Dempster-Shafer Theory (DST): Uses decision and confidence measures with forward/reverse parameters and rules.
Certus: A mixed method using linguistic belief levels and a domain-specific language for propagation expressions (e.g., macros, conditional logic).

Key Results

The scalability analysis was conducted for argument heights $h$ ranging from 1 to 12, with $h=10$ representing a typical size based on published case studies (approx. 350 total nodes).

Worst-Case Complexity: Certus exhibits the highest decision complexity, requiring over 10,000 decisions at $h=10$ . This is significantly higher than BBN (~~971 decisions) and DST (~~1,716 decisions). This is attributed to Certus's flexibility, which allows for custom belief propagation expressions over all child nodes.
Average-Case Complexity: When assuming tool support and defaults for 80% of steps ( $\alpha=0.2$ ), Certus demonstrates the lowest decision complexity among the three methods.
Effort Estimates: For an argument of ~350 nodes ( $h=10$ $h = 10$ ), the model predicts the following average-case effort:
- Certus: ~13 hours
- BBN: ~14 hours
- DST: ~26 hours

Significance and Contributions

The paper claims to be the first to address the scalability of quantitative CAMs through a formal numerical model. Its primary contributions are:

Conceptual Framework: It introduces two specific measures to quantify the adoption barrier: decision complexity (number of decisions) and level of effort (time).
Scalability Model: It proposes a parameterized model to estimate these quantities, allowing researchers developing new or improved CAMs to evaluate the effort required for application.
Empirical Validation: By applying the model to BBN, DST, and Certus, the paper demonstrates that while high-flexibility methods like Certus may have extreme worst-case complexity, their average-case effort can be lower than established probabilistic methods due to the availability of macros and defaults.

Limitations

The authors explicitly note several limitations:

Argument Model: The $n$ -ary tree is an idealization; real ACs may have deep chains or wide steps that deviate from this structure.
Decision Difficulty: The model counts decisions but does not account for the relative cognitive difficulty (e.g., assigning a value on a 5-point scale vs. a continuous [0,1] scale).
Timing Parameters: The time estimates ( $t_p, t_v$ ) are based on intuition rather than empirical calibration and are context-dependent.

The paper concludes that while Certus has the highest worst-case complexity, its average-case efficiency makes it a viable candidate, provided that tooling supports the reduction of manual decisions.