From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

Technical Summary: From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

Problem Statement
Current approaches to analyzing risks in Artificial Intelligence (AI) systems predominantly focus on individual components—such as training data or specific machine learning models—in isolation. This "siloed" perspective overlooks hazards that emerge from the interactions between components, the integration of models into broader software stacks, and the sociotechnical processes of development and operation. While regulatory frameworks (e.g., the EU AI Act) and industry practices (e.g., system cards) are beginning to distinguish between AI models and AI systems, there is a lack of empirical frameworks to facilitate hazard analysis at the system level. Existing methods often fail to account for the emergent nature of safety in complex sociotechnical systems, where harms arise not just from component failure, but from the interplay of technical artifacts, human actors, and institutional processes. Furthermore, AI systems possess unique characteristics—such as model opacity, probabilistic capabilities, and output complexity—that vary significantly across paradigms (e.g., linear regression vs. generative models), complicating the application of traditional safety engineering tools.

Methodology
The authors adapt System Theoretic Process Analysis (STPA), a well-established hazard analysis framework from the field of system safety, for use in AI contexts. STPA treats safety as an emergent property of the entire system, analyzing control structures and feedback loops rather than just component failures.

The research employs a qualitative case study approach involving three distinct AI systems across different domains and paradigms:

1. Case 1 (C1): A linear regression-based early warning system for predicting late-onset sepsis in pre-term infants (Human-in-the-loop).
2. Case 2 (C2): A Reinforcement Learning (RL) based automated insulin injection system for diabetes management (Human-out-of-the-loop).
3. Case 3 (C3): Text-to-image (T2I) generative models used for storyboarding in creative practice (Human-as-tool-user).

The methodology involved gathering data through stakeholder interviews, workshops, and review of technical documentation and academic literature. The researchers applied the four standard steps of STPA to each case:

1. Identifying the purpose of the analysis (losses and system boundaries).
2. Creating control diagrams (mapping controllers, controlled processes, and feedback).
3. Identifying Unsafe Control Actions (UCAs).
4. Identifying loss scenarios (causal chains leading to UCAs).

Through iterative, reflexive analysis of these applications, the authors developed PHASE (Process-oriented Hazard Analysis for AI Systems), a guideline that translates STPA concepts to address AI-specific challenges.

Key Contributions
The primary contribution is the PHASE guideline, which adapts STPA for AI by introducing specific modifications to handle the heterogeneity of AI systems:

• Broadened Loss Taxonomy: PHASE expands the definition of "loss" beyond traditional safety-critical outcomes (e.g., loss of life) to include performance-related losses (e.g., efficiency, reputation) and sociotechnical losses (e.g., privacy violations, loss of diversity, algorithmic bias).
• Process-Oriented System Boundaries: Instead of drawing boundaries around the model itself, PHASE defines boundaries around the development and use processes (the algorithmic supply chain), acknowledging that developers control data and design choices but not necessarily the learned latent representations.
• Dynamic Control Structures: The framework accommodates the fluidity of roles in AI, where a component (e.g., an AI assistant) may act as a controller in one context and a controlled process in another, depending on organizational mandates and usage patterns.
• AI-Specific Loss Scenarios: PHASE explicitly accounts for loss scenarios arising from emerging and evolving system capabilities (e.g., model updates, retraining, or expanded use cases) that outpace existing safeguards, a factor often missed in traditional STPA applied to static systems.

Results and Findings
The application of STPA via PHASE across the three case studies yielded several findings:

• Applicability: The core concepts and steps of STPA are applicable to AI systems but require targeted adaptations to address AI-specific challenges like opacity and autonomy.
• Hazard Detection: The system-level analysis successfully identified hazards that would remain invisible in component-level audits. For instance, a model might pass a fairness audit in isolation, yet the system-level interaction between an imperfect alert and a clinician's over-trust could lead to misdiagnosis.
• Heterogeneity Matters: The analysis revealed that different AI paradigms (linear models, RL, generative models) place different demands on the analysis. For example, defining system boundaries was more complex for open-ended generative platforms than for the more constrained RL insulin system.
• Four Key Affordances: The study identifies four specific affordances provided by PHASE for analysts:
  1. System-Level Detection: Identifying hazards arising from the accumulation of disparate issues across the system lifecycle.
  2. Social Factor Acknowledgment: Explicitly mapping social factors (e.g., organizational culture, time pressures) that contribute to algorithmic harms.
  3. Traceable Accountability: Establishing clear chains of accountability by mapping control actions and feedback loops to specific actors and artifacts, distinguishing whether a harm originated from data collection, model training, or user interaction.
  4. Emergent Hazard Monitoring: Enabling the ongoing monitoring of new hazards as AI systems adapt, update, or change their operational context over time.

Significance and Claims
The paper claims that PHASE offers a critical complement to existing Responsible AI (RAI) efforts by shifting the analytical lens from isolated components to the emergent properties of the whole system. It argues that safety in AI is not merely a technical attribute of a model but a sociotechnical property of the system's control structures.

The authors position PHASE as a tool to:

• Facilitate the transition from "silos" to "systems" in AI safety analysis.
• Provide a structured method for establishing traceability between specific harms and the actors or processes responsible for mitigating them, thereby addressing the "accountability gap" in AI development.
• Support the development of a safety culture within organizations by making implicit risks and social factors explicit.

The authors remain modest regarding the generalizability of their findings, noting that while the three case studies span distinct paradigms, they do not represent the entire AI landscape. They acknowledge that the effectiveness of PHASE depends on the expertise of the analysts and the willingness of organizations to allocate resources to safety. The work is presented as a foundational step toward integrating system safety engineering principles into the broader discourse of AI governance and responsible development.