💻 computer science

A Subjective Logic-based method for runtime confidence updates in safety arguments

This paper presents a Subjective Logic-based method that enhances static safety cases by dynamically updating confidence levels at runtime through the continuous evaluation of Safety Performance Indicators, prioritizing safety-relevant responsiveness over exact Bayesian inference.

Original authors: Benjamin Herd, Jessica Kelly, Clarissa Heinemann, João-Vitor Zacchi

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Benjamin Herd, Jessica Kelly, Clarissa Heinemann, João-Vitor Zacchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a very strong, logical argument for why a self-driving car is safe. You've checked the code, tested the sensors, and written a report saying, "We are 95% sure this car won't hit a construction cone." This is your Safety Argument.

The problem is, the real world is messy. The car might encounter a weirdly placed cone, bad lighting, or a new type of road sign that wasn't in your original tests. Your original report is like a snapshot taken yesterday; it doesn't know what's happening right now.

This paper proposes a way to keep that safety argument "alive" and up-to-date while the car is actually driving. Here is how they do it, using simple analogies:

1. The "Safety Scorecard" (SPIs)

Instead of just waiting for a crash to happen (which is too late), the authors suggest using Safety Performance Indicators (SPIs). Think of these as a safety scorecard attached to specific parts of your argument.

The Analogy: Imagine you are a teacher grading a student's essay. The "Design-Time Argument" is the initial grade you give based on the draft. The "SPI" is a live monitor that watches the student while they are writing the final version.
How it works: If the student starts making a specific type of mistake (like forgetting to capitalize "I"), the monitor flags it immediately. In the car example, if the camera misses a cone too often, the SPI flags it.

2. The "Confidence Meter" (Subjective Logic)

The authors use a mathematical tool called Subjective Logic to measure confidence. Instead of just saying "95% safe," they break confidence down into three buckets:

Belief: "I think this is true."
Disbelief: "I think this is false."
Uncertainty: "I don't have enough info to decide."

The Analogy: Imagine a jar of marbles.

Blue marbles represent belief (it's safe).
Red marbles represent disbelief (it's unsafe).
Clear marbles represent uncertainty (I'm not sure).
A "safe" argument has mostly blue marbles and very few clear ones.

3. The "Two-Rule Update System"

This is the core innovation. The authors created a special rule for how the confidence jar changes when the car is driving. They wanted a system that is responsive (quick to react to danger) rather than just statistically perfect.

They use two rules simultaneously:

Rule A: The Slow Accumulator (No Violations)
- What happens: If the car drives for a while and doesn't miss any cones, the SPI monitor adds a few Blue marbles to the jar.
- The Effect: Your confidence slowly grows. The Clear marbles (uncertainty) shrink because you have more data. It's like a slow, steady drip filling a bucket.
Rule B: The Instant Penalty (Violations)
- What happens: If the SPI monitor sees a violation (the car misses a cone), it doesn't just add a red marble. It performs a sudden, targeted swap.
- The Effect: It immediately takes Blue marbles (belief) and turns them into Red marbles (disbelief). It's like a "safety alarm" that instantly lowers your confidence the moment a problem is detected.
- Why this matters: The authors say, "We don't care about perfect math here; we care about safety." If a safety rule is broken, you need to drop your confidence fast, not wait for a slow statistical average to catch up.

4. The Real-World Test (The Construction Zone)

To prove this works, the authors simulated a self-driving car in a construction zone.

The Setup: They had a camera designed to spot orange traffic cones.
The Test: They ran a simulation where the car sometimes missed cones (due to shadows or other cars blocking the view).
The Result:
- When the car drove smoothly, the "Blue" confidence grew, and the "Clear" uncertainty shrank.
- The moment the car missed a cone, the confidence dropped sharply.
- As the car drove well again, the confidence slowly climbed back up, but it never fully returned to the "perfect" level because the system remembered the mistake.

Summary

The paper presents a method to turn a static safety report into a living, breathing safety argument.

Old Way: "We tested this, and it's safe." (Static, doesn't change).
New Way: "We tested this, and it's safe. AND, we are watching it right now. If it does well, we get more confident. If it messes up even once, we instantly lose confidence and know we need to fix it."

This allows engineers and regulators to see exactly how much they can trust the system at any given moment, based on real-time performance, not just past tests.

Technical Summary: A Subjective Logic-based Method for Runtime Confidence Updates in Safety Arguments

Problem Statement
Safety assurance arguments for machine learning (ML) components, particularly in dynamic environments like autonomous driving, face a critical challenge: static arguments based on design-time evidence may become invalid over time due to data distribution shifts, model drift, or evolving operational contexts. While standards like UL 4600 mandate the use of Safety Performance Indicators (SPIs) to monitor system validity, existing methods for incorporating runtime data often lack the granularity to quantify confidence updates dynamically. Qualitative approaches typically offer only binary outcomes (refutation or no change), while classical Bayesian updates may not adequately prioritize immediate safety responsiveness over exact posterior equivalence. There is a need for a formal mechanism that integrates continuous, windowed runtime evidence into quantitative assurance arguments, allowing for both the gradual accumulation of confidence and immediate penalties upon safety violations.

Methodology
The authors propose a method that extends static, quantitative assurance arguments using Subjective Logic (SL) to incorporate runtime SPIs. The core of the methodology involves a specific update rule that combines two SL operators: Cumulative Belief Fusion (CBF) and the Refuting Challenger (RC).

Argument Structure: The safety case is modeled as a structured argument where each claim $c$ is associated with a binomial opinion $\omega_c = (b, d, u, a)$ , representing belief, disbelief, uncertainty, and base rate.
SPI Modeling: Runtime SPIs are treated as binary predicates derived from thresholded measurements over a window of $k$ observations. An SPI opinion $\omega_{SPI}$ is constructed by counting successes ( $r$ , no violations) and failures ( $s$ , violations) within the window, mapped to an SL opinion using standard equations.
The Update Rule: The central innovation is the update equation for a claim opinion $\omega_c$ $ω_{c}$ given a new SPI opinion $\omega_{SPI}$ $ω_{S P I}$ :
$\omega'_c = \neg\omega_{SPI} \boxtimes (\omega_c \oplus \omega_{SPI})$
- Inner Operation ( $\oplus$ ): Cumulative Belief Fusion accumulates the evidence from the SPI window. If no violations occur, this increases belief and reduces uncertainty in the claim.
- Outer Operation ( $\boxtimes$ ): The Refuting Challenger applies an immediate penalty. The SPI opinion is negated ( $\neg\omega_{SPI}$ ) so that belief in an SPI violation maps to disbelief in the claim. This operator reallocates mass from belief to disbelief proportional to the severity of the violation, ensuring that a single violation can immediately degrade confidence, regardless of prior accumulation.

This design explicitly departs from standard conjugate Beta-Binomial Bayesian updates. Instead of merely adding counts, the method prioritizes safety-relevant responsiveness, ensuring that violations trigger prompt corrective confidence drops while non-violations allow for gradual confidence recovery.

Key Contributions

SL-Native Update Mechanism: The paper introduces an explicitly non-Bayesian update rule that integrates windowed runtime SPIs into quantitative assurance arguments. It leverages SL's ability to distinguish between first-order confidence (commitment to belief/disbelief) and second-order confidence (precision of that commitment).
Dual-Property Update Rule: The proposed operator combination satisfies two critical safety properties: (1) gradual confidence increase in the absence of violations, and (2) immediate, targeted penalties upon violations.
Formal Integration of SPIs: The method provides a formal way to link leading SPIs (proactive indicators) directly to specific claims within the assurance argument, enabling continuous reassessment of argument validity.

Experimental Results
The method is demonstrated using a simulation of an ML-based construction cone detection component within an autonomous vehicle system (APIKS platform).

Scenario: A YOLOv8-based detector is monitored for false negatives relative to ground truth. An SPI is defined with a threshold on the ratio of missed cones.
Observations:
- When SPI violations occurred (e.g., due to occlusion), the claim's belief dropped immediately, and disbelief increased.
- In periods without violations, uncertainty decreased, and belief gradually recovered.
- The resulting Beta distributions showed that the method successfully reduced overall uncertainty over time while maintaining sensitivity to safety-critical events.
- Different scenarios illustrated how the method handles varying evidence: shifting mass toward disbelief when violations accumulate, and toward belief when evidence is consistent, while retaining uncertainty when data is insufficient (e.g., in short nighttime segments).

Significance and Claims
The authors claim that this work offers a practical, auditable approach for continuous safety assurance in AI-based systems. By aligning with standards like UL 4600, the method bridges the gap between static design-time arguments and dynamic operational realities.

Governance: The approach provides a transparent "confidence narrative" that maps raw runtime signals to explicit claim-level opinions. This supports fleet operators and safety managers in triggering mitigations (e.g., parameter adjustments, human oversight) based on quantitative thresholds.
Auditability: It creates a formal link between runtime evidence and the safety case, allowing regulators to review how specific violations impacted confidence.
MLOps Integration: The method connects assurance confidence degradation to engineering actions, such as targeted retraining or data acquisition, effectively linking safety arguments to MLOps workflows.

The paper acknowledges limitations, noting that effectiveness depends on accurate SPI thresholding and that the current model assumes independent, non-overlapping windows. Future work is suggested to address temporal correlations, time-based discounting for stale evidence, and the integration of both leading and lagging SPIs within a unified argument structure.