🤖 AI

Update Opacity: Epistemic Accessibility and Governance Under AI System Change

This paper addresses the governance challenge of "update opacity" in AI systems by proposing a framework that combines the EU AI Act and Machine Learning Operations to implement threshold-based disclosure of materially relevant changes, thereby ensuring epistemic accessibility for users without causing information overload.

Original authors: Andrea Ferrario, Joshua Hatherley

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrea Ferrario, Joshua Hatherley

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Silent Shift"

Imagine you have a very smart, helpful GPS app. You've used it for years. You know exactly how it behaves: if you type in "Coffee Shop," it usually shows you the one on Main Street because that's where the traffic is lightest. You have built up a "gut feeling" or a calibration about how the app works.

Now, imagine the app developers quietly update the code overnight to make it faster. They don't tell you. The next morning, you type in "Coffee Shop," and suddenly, it sends you to a different shop on a different street. It's still a valid coffee shop, and the app is still working "correctly" by its own internal standards. But you are confused. You don't know why the answer changed. You don't know if you should trust the new route.

The paper calls this Update Opacity. It's not that the AI is broken; it's that the AI changed in a way that you, the user, can't see or understand. This is dangerous in high-stakes situations (like a doctor using AI to diagnose a patient or a bank using it to approve loans) because users rely on their "gut feeling" about how the system behaves. If the system changes silently, that gut feeling becomes wrong, leading to bad decisions.

The Two Current Solutions (and why they aren't enough)

The authors look at two existing ways we try to manage AI changes, but they say neither works alone:

The "Rulebook" Approach (The EU AI Act):
- The Analogy: Think of this like a strict building code. If you want to build a house, you must follow specific rules. If you decide to knock down a load-bearing wall or add a whole new floor, you must get a new permit and an inspector to check it.
- The Problem: This rulebook is great for big, dangerous changes. But what if the builder just swapped the paint color or moved a light switch? The rulebook doesn't care. But for the people living in the house, those small changes might still be annoying or confusing. The rulebook is too "coarse" (too big) to catch the small, confusing shifts that happen inside the system.
The "Engineer's Dashboard" Approach (MLOps):
- The Analogy: This is like the dashboard in a race car. It tracks every tiny vibration, temperature change, and fuel fluctuation. The engineers can see everything happening inside the engine.
- The Problem: If you showed that dashboard to the driver (the user), they would be overwhelmed. They don't need to know that "Fuel Injector #3 shifted by 0.04%." They just need to know if the car is safe to drive. MLOps tracks the changes, but it doesn't tell us which changes actually matter to the person using the system.

The Authors' Solution: The "Trustworthiness Plateau"

The authors propose a new way to manage this by combining the Rulebook and the Dashboard. They suggest we stop looking at the AI as a single "model" and start looking at it as a Trustworthy System that has different "levels" of safety.

Here is their three-step plan:

Step 1: The "Safe Zone" (The Plateau)

Imagine the AI's performance is a flat plateau. As long as the AI stays on this plateau, it is considered "safe" and "compliant."

Big Changes: If the AI falls off the cliff (e.g., it stops working or breaks the law), that's a crisis. You stop everything and get a new permit (like the EU AI Act requires).
Small Changes: Most updates happen on the plateau. The AI gets slightly better or slightly different, but it's still safe.

Step 2: The "Trustworthiness Profile"

Instead of just checking if the AI is "right," we track a specific list of things that matter to the user. Let's call this the Trustworthiness Profile.

For a medical AI, this might include: "How accurate is it for elderly patients?" "How fast is it?" "Does it confuse similar-looking diseases?"
We turn these into a score. As long as the score stays within a certain range, the AI is on the "Safe Plateau."

Step 3: The "Materiality Threshold" (The Alarm Bell)

This is the most important part. Even if the AI stays on the "Safe Plateau," it can still drift far away from where it started.

The Analogy: Imagine you are walking on a flat field. You start at point A. If you walk 5 feet to the right, you are still on the field. If you walk 500 feet to the right, you are still on the field, but you are now in a completely different part of the field.
The authors say: We need a Threshold. If the AI changes so much that it crosses a certain distance from where it started (even if it's still "safe"), we must ring a bell.
This bell tells the user: "Hey, the system has changed enough that your old 'gut feeling' might not work anymore. Here is what changed."

How This Works in Real Life (The Medical Example)

The paper uses a Stroke Triage AI (a system that helps doctors decide if a patient needs to go to a big hospital or a local one) to show how this works.

The Situation: The AI is updated to handle new types of CT scanners. The update is "safe" (it doesn't break the rules). The overall accuracy actually gets slightly better.
The Hidden Change: However, the update makes the AI slightly more likely to send elderly patients to the big hospital, even if their symptoms are borderline.
The Old Way: The doctor keeps using the AI, trusting their old habits. They might miss the subtle shift, leading to a patient being sent to the wrong place.
The New Way (The Authors' Framework):
1. The system tracks the change.
2. It sees that the change for "elderly patients" has crossed the Threshold.
3. The Disclosure: Instead of a boring technical report, a small, clear message pops up on the doctor's screen: "Model Updated: Recommendations for patients over 75 may now differ. Tap for details."
4. The doctor sees the warning, adjusts their thinking, and makes a safe decision.

The Bottom Line

The paper argues that we don't need to tell users everything about every update (that would be too much information). We also can't tell them nothing (that would be dangerous).

We need a smart filter. We need to measure the AI's "Trustworthiness," watch how much it drifts from its starting point, and only ring the alarm when the change is big enough to confuse the user. This keeps the AI safe, legal, and understandable, even as it constantly evolves.

Technical Summary: Update Opacity, Epistemic Accessibility, and Governance Under AI System Change

1. Problem Definition: Update Opacity

The paper addresses update opacity, a phenomenon occurring when expert users cannot, upon inspection, explain why a current version of an AI system produces a different output ( $y$ ) for a given input ( $x$ ) compared to a previous version ( $z \neq y$ ). Unlike static model opacity (which concerns the internal mechanics of a single model), update opacity is diachronic and comparative; it arises from the temporal evolution of deployed systems.

The core problem is framed as a failure of epistemic accessibility. In high-stakes domains (e.g., healthcare, finance), users rely on accumulated calibration—learned expectations about system behavior, reliability, and failure modes—to integrate AI outputs into decision-making. When a system undergoes model-affecting updates (retraining, recalibration, pipeline revisions, dependency updates) without transparent communication, this user-side calibration becomes misaligned with the system's actual behavior. This misalignment undermines justified reliance, increases cognitive burden, and weakens accountability.

The authors argue that update opacity is a governance problem characterized by a tension between three unsatisfactory extremes:

Total Transparency: Disclosing every update is infeasible and counterproductive, leading to information overload and workflow disruption.
Silence: Disclosing nothing allows silent behavioral changes to erode justified reliance.
Freezing Updates: Halting all updates is unviable in nonstationary environments, leading to performance deterioration.

The central challenge is determining which changes are materially relevant, when they should be surfaced, and to whom, to maintain epistemic accessibility without overwhelming users.

2. Methodology: Merging Two Governance Cultures

The proposed solution integrates two complementary governance cultures: the EU AI Act (AIA) and Machine Learning Operations (MLOps).

The EU AI Act (Normative Perimeter)

The AIA provides the normative framework for defining the "perimeter" of relevant change.

Substantial Modification: The AIA distinguishes between changes that trigger a new conformity assessment (substantial modifications, e.g., changes to intended purpose or compliance with high-risk requirements) and those that do not.
Limitation: While the AIA identifies when a change is legally significant (identity-breaking), it is too coarse-grained for the epistemic problem. Many permissible changes (within the "conformity envelope") can still materially alter user reliance and workflow expectations without triggering legal reassessment.

MLOps (Operational Machinery)

MLOps provides the operational tools for tracking and managing change.

Drift Detection & Monitoring: MLOps utilizes continuous telemetry, versioning, and drift detection (comparing operational data against baselines) to trigger interventions (retraining, rollback).
Limitation: MLOps is oriented toward engineering control and technical metrics (e.g., accuracy, drift indicators). It does not inherently determine which monitored changes are epistemically material for specific stakeholders (e.g., clinicians vs. auditors) or how to communicate them to preserve calibrated reliance.

The Proposed Framework: Trustworthiness-Based Governance

The authors propose a framework that merges these cultures to govern update opacity through three specific steps:

Step 1: Tolerable vs. Non-Tolerable Changes

The framework distinguishes between:

Non-tolerable changes: Those falling outside the assessed trustworthiness envelope (triggering AIA reassessment).
Tolerable changes: Those remaining within the envelope but potentially affecting user reliance. The framework focuses on governing this second category.

Step 2: Trustworthiness Profiles and Levels

The authors introduce a formal model for system state:

Trustworthiness Profile ( $P_A$ ): A set of predetermined, quantifiable dimensions ( $d_i$ ) and measurement protocols ( $m_i$ ) defined ex ante (e.g., accuracy, robustness, subgroup stability, data pipeline integrity).
Trustworthiness Level Function ( $T_A$ ): A function mapping time-indexed measurement vectors ( $m_A(t)$ $m_{A} (t)$ ) to discrete trustworthiness levels ( $1, \dots, K$ $1, \dots, K$ ).
- $T_A(t) = L_A(m_A(t))$ , where $L_A$ is a level assignment function.
- Step-wise Mapping: The authors advocate for step-wise (plateau) mappings rather than smooth curves. This prevents "governance strain" caused by minor fluctuations near transition points, ensuring that only significant shifts in the system's regime trigger escalation.

Step 3: Threshold-Based Disclosure

To address opacity within a single trustworthiness level (within a plateau), the framework introduces a dissimilarity function ( $\Delta_A$ ) and a context-relative threshold ( $\epsilon_A$ ).

Mechanism: For a system $A$ deployed at $t_0$ with baseline profile $m_A(t_0)$ , an update at time $t_i$ triggers disclosure if:
$\Delta_A(m_A(t_0), m_A(t_i)) > \epsilon_A$
Epistemic Materiality: The threshold $\epsilon_A$ is set strictly inside the admissible plateau. It acts as an "epistemic materiality" filter: even if the system remains in the same regulatory level, if the cumulative change exceeds $\epsilon_A$ , it is deemed large enough to disrupt calibrated reliance and thus requires disclosure.
Implementation: $\Delta_A$ can be instantiated using standard MLOps monitoring summaries (e.g., $L_1$ distance between measurement vectors), potentially weighted to penalize specific degradations.

3. Key Contributions

Epistemic Interpretation of Update Opacity: Reframing update opacity not merely as a technical lack of explainability, but as a diachronic failure of epistemic accessibility where materially relevant changes fail to remain accessible to users under their practical constraints.
Integration of AIA and MLOps: Proposing a hybrid governance model where the AIA defines the scope of relevant change (the envelope) and MLOps provides the mechanism for tracking and triggering disclosure within that envelope.
Trustworthiness Profiles and Levels: Formalizing system state via discrete trustworthiness levels to distinguish between tolerable background noise and governance-relevant events, avoiding the brittleness of continuous monitoring.
Threshold-Based Disclosure Rule: Introducing a proportionality criterion where disclosure is triggered not by the mere existence of an update, but by the magnitude of the change relative to a baseline, specifically when it threatens calibrated reliance.
Role-Sensitive Disclosure Design: Arguing that disclosure must be contextualized. Frontline users (e.g., clinicians) require concise, actionable information embedded in workflows, while auditors require full traceability. The framework supports a layered disclosure approach.

4. Results and Illustration

The paper does not present empirical experimental results but offers a conceptual demonstration via a medical AI example: Stroke Triage Decision Support.

Scenario: A deployed system for stroke triage undergoes a model update due to covariate drift (firmware changes in CT scanners). The update improves global AUROC and remains within the highest trustworthiness level (no AIA reassessment triggered).
The Opacity: Despite global stability, the update shifts feature weights, altering transfer recommendations for specific subgroups (e.g., elderly patients) and changing alert patterns. Clinicians, relying on prior calibration, are unaware of these shifts.
Application of Framework:
- The dissimilarity function detects that the cumulative change in subgroup behavior exceeds the threshold $\epsilon_A$ .
- Disclosure Triggered: Instead of a generic changelog, a workflow-embedded banner alerts the clinician: "Model updated [date]. Transfer recommendations for patients over 75 may differ."
- Accountability: This creates a closed epistemic loop. If a patient is undertriaged post-update, the system can demonstrate that the provider disclosed the change and the user was informed, closing the "accountability lacuna" that would otherwise exist if the update were silent.

5. Significance and Claims

The paper claims that governing AI system updates is not solely a matter of technical maintenance or legal compliance, but a fundamental requirement for preserving the conditions under which changing systems remain intelligible.

Modest Scope: The authors do not claim to solve all transparency issues or provide a universal algorithm for all AI systems. They explicitly state that choices regarding profile dimensions, dissimilarity functions, and thresholds are context-sensitive and must be justified case-by-case.
Practical Utility: The framework offers a principled, operationalizable alternative to the "all-or-nothing" approach to transparency. It allows providers to maintain transparency by issuing warnings only when within-envelope changes become materially relevant for deployers and affected parties.
Future Work: The authors identify the need for future research to refine domain-specific choices for profile dimensions and dissimilarity functions, and to empirically test how different disclosure interfaces affect calibrated use in high-stakes settings.

In summary, the paper argues that update opacity is a governance failure of epistemic accessibility. By combining the normative boundaries of the EU AI Act with the operational monitoring of MLOps, and applying a threshold-based disclosure rule, it is possible to manage system evolution while maintaining the calibrated reliance necessary for safe and accountable AI deployment.