Co-constructing sociotechnical AI governance: participatory system mapping using algorithm registers

Technical Summary: Co-constructing Sociotechnical AI Governance via Participatory System Mapping

Problem Statement
Algorithm registers are increasingly adopted by public sector organizations to provide transparency regarding the use of algorithms in public services. However, current implementations face significant limitations: they often fail to reconcile the varying information needs of diverse stakeholders (citizens, civil society, oversight bodies), and the information provided is frequently either too sparse to be useful or too technical to be understood. Consequently, these registers struggle to facilitate meaningful accountability or to represent the full sociotechnical systems in which algorithms are embedded. The paper posits that without understanding how registers both constitute and inform algorithm governance, governance practices risk becoming disjointed efforts that cannot anticipate or address actual safety hazards.

Methodology
The study employs a mixed-method approach centered on a case study of "Avola," a decision-support tool used by municipal caseworkers in a Dutch city to assess welfare benefit eligibility via legal automation (business rule engine). The research involved five stages, with a primary focus on Stages IV and V:

1. Stakeholder Identification: The study engaged three distinct groups: Municipal Staff (direct stakeholders, including algorithm governance officers and product owners), Municipal Ombudsmen, and Civil Society Organizations (CSOs) (indirect stakeholders).
2. Participatory System Mapping (Stage IV): Two workshops were conducted with these groups. Participants were presented with a preliminary system map (M0) derived from the municipal algorithm register and public documentation. Using worksheets and large-format maps, participants annotated the system to identify missing actors, processes, and feedback loops. This process integrated the "situated knowledge" of indirect stakeholders with the technical documentation of the register.
3. System-Theoretic Process Analysis (STPA) (Stage V): The authors utilized the System-Theoretic Accident Model and Processes (STAMP) framework to conduct a safety analysis. The participatory maps informed the construction of a Hierarchical Control Structure (HCS). The analysis followed four iterative steps:
   • Identifying potential losses (e.g., wrongful denial of benefits).
   • Modeling the hierarchical control structure (technical, operational, organizational, and institutional layers).
   • Identifying Unsafe Control Actions (UCAs) where control mechanisms fail.
   • Deriving Loss Scenarios that link system hazards to specific losses.

Key Contributions
The paper makes three core contributions to the fields of AI governance and safety science:

1. Empirical Assessment: It provides a critical evaluation of a municipal algorithm register, demonstrating that while it serves as a "disciplinary" tool for internal stock-taking, it currently fails to meet the transparency needs of external stakeholders due to abstraction and lack of actionable detail.
2. Participatory Mapping Schema: It develops and validates a method for mapping algorithmic systems and their governance structures by integrating the perspectives of direct and indirect stakeholders. This approach reveals system components and dynamics that are occluded in standard register disclosures.
3. Normative Reflection: It reflects on the political dimensions of system safety analysis, arguing that the abstractions used to map governance structures are inherently political choices that determine what is visible and what remains hidden.

Results
The participatory mapping and subsequent STPA analysis revealed several safety hazards and governance gaps that were not apparent through the algorithm register alone:

• Human-in-the-Loop Failures: While the register asserts that caseworkers have final discretion, the analysis identified a risk of "automation bias" where caseworkers, under time pressure or trusting the "legal automation" as objective, may fail to detect errors (e.g., outdated citizen data). This leads to the loss scenario of wrongful benefit denial.
• Complaints Procedure Deficiencies: The register indicated that incident reports were shared with the developer, but the participatory process revealed that error metrics were not tracked systematically. Consequently, complaints are likely treated case-by-case rather than aggregated to identify systemic patterns, preventing necessary model updates.
• Oversight and Accountability Gaps: Ombudsmen and CSOs found the register's disclosure of risk assessments (FRAIA and DPIA) insufficient. The register stated assessments were conducted but withheld details on findings or error metrics. This lack of technical documentation prevents external actors from effectively contesting decisions or mounting legal challenges in Civil Court.
• Political Context: The mapping revealed that the algorithm register is embedded within a complex political hierarchy (Aldermen, City Council, Ministry of Interior). The information disclosed is subject to political negotiation and resource constraints, often prioritizing the needs of expert intermediaries over the general public.

Significance and Claims
The paper claims that algorithm registers, in their current form, are insufficient for achieving meaningful public accountability. The authors argue that transparency is a relational act that requires understanding the heterogeneity of "publics." By engaging indirect stakeholders (Ombudsmen, CSOs) through participatory system mapping, the study demonstrates that:

• System Safety Requires Pluralism: Safety analysis cannot be purely technical; it must incorporate the situated knowledge of those affected by or monitoring the system to identify hazards like automation bias and opaque feedback loops.
• Registers as Political Artifacts: The content of an algorithm register is not neutral; it reflects political choices about what to disclose. Therefore, improving these instruments requires engaging with the political processes that shape them.
• Methodological Shift: The authors propose that system-theoretic methods like STPA, when combined with participatory design, offer a robust framework for reconstructing the sociotechnical architecture of algorithmic governance. This allows for a more comprehensive identification of safety hazards than the register alone can provide.

The study concludes that while algorithm registers are a necessary step toward transparency, they must evolve to better align with the information needs of diverse stakeholders and be understood as part of a broader, politically situated sociotechnical system.