Public regulatory data infrastructure for lifecycle oversight of AI medical devices

Technical Summary: Public Regulatory Data Infrastructure for Lifecycle Oversight of AI Medical Devices

Problem Statement
Regulatory science for AI-enabled medical devices has shifted toward a "lifecycle oversight" philosophy, acknowledging that software evolves post-market through mechanisms like Predetermined Change Control Plans (PCCPs). However, the public regulatory data infrastructure—specifically the registries used by hospitals, safety officers, and researchers to identify devices, trace changes, and link authorizations to safety signals—has not evolved in tandem. Current registries in major jurisdictions (USA, EU, China, Japan, Australia) remain "registration-oriented," designed primarily to confirm legal market authorization for static hardware. They lack the structured, queryable fields necessary to govern mutable AI software. Consequently, external stakeholders cannot reliably determine the current software version, the evidence base for specific populations, the history of algorithmic changes, or the linkage between authorization and post-market adverse events. The paper argues that without a public data layer supporting "public governability," safe procurement, deployment, and accountability for AI medical devices are structurally compromised.

Methodology
The study employs a source-grounded audit of public regulatory record environments across five jurisdictions: the USA (FDA), European Union (EUDAMED), China (NMPA), Japan (PMDA), and Australia (TGA).

1. Derivation of Specifications: The authors derived a candidate minimum public-data specification consisting of five governance functions and 20 structural requirements. These were not established via expert consensus but through a transparent "four-anchor source-concordance" procedure, linking each requirement to regulatory, governance, clinical/operational, and empirical/problem anchors.
   • F1 Identity: What is the regulated object? (e.g., software status, stable ID).
   • F2 Version & Change Legibility: Has the object changed, and under what authority? (e.g., version history, regulatory basis for change).
   • F3 Evidence Legibility: What was the basis for regulatory confidence? (e.g., validation setting, population, metrics).
   • F4 Linkage: Can pre-market records connect to post-market reality? (e.g., linkage to recalls/adverse events).
   • F5 AI Specificity: Can the public record distinguish AI attributes? (e.g., model class, training data modality, provenance).
2. Audit Instrument: Two annotators independently scored 100 cells (5 jurisdictions × 20 requirements) on a 0–3 ordinal scale:
   • 0: Not publicly available.
   • 1: Public only in narrative, PDF, or free-text form.
   • 2: Structured field but not queryable/downloadable.
   • 3: Structured, queryable, or downloadable.
3. Gap Typology: Unmet requirements were classified into five distinct gap types to determine appropriate reform instruments: Schema (no field exists), Publication–Access (data held but not published), Structuring (public but unstructured), Linkage (no stable identifiers), and Population (fields exist but are empty/inconsistent).
4. Stress Tests: Three empirical probes were conducted to validate architectural findings:
   • FDA AI Identifiability: Testing if known AI devices can be recovered from primary 510(k)/PMA records without the curated AI list.
   • NMPA UDI Findability: Testing if the bulk-downloadable Chinese UDI infrastructure exposes AI status as structured fields.
   • FDA Product-Family Lineage: Testing if structured predecessor/successor links exist for multi-submission AI families in openFDA.
5. Reliability: Inter-annotator agreement was assessed (pre-adjudication Cohen's κ = 0.724 for availability), with discrepancies resolved via documented adjudication. Sensitivity analyses were run using a 12-item "conservative core" specification and high-confidence subsets.

Key Results
The audit reveals a structural mismatch between regulatory philosophy and public data infrastructure.

• Overall Deficit: Only 6% (6 out of 100) of cells reached the threshold of structured, queryable availability (Score 3). 94% of cells fell below this threshold.
• Functional Concentration: Deficits are not evenly distributed but concentrate heavily in F2 (Version & Change), F3 (Evidence Legibility), and most severely, F5 (AI Specificity).
  • F5 (AI Specificity): No jurisdiction exposed model/method class, training-data modality, or data provenance as structured fields in primary public records. All F5b–d cells scored 0 or 1.
  • F2 (Change): Even the USA, which has the most advanced PCCP framework, exposes version and change history only in narrative or inferential forms, not as structured fields.
• Jurisdictional Nuance: The USA (FDA) is closest to structured exposure, particularly in F1 (Identity) and F5a (AI task category via a supplementary list), but still fails to expose current software versions or structured product-family lineage. The EU (EUDAMED) and China (NMPA) show significant gaps in structured evidence and AI metadata.
• Gap Typology Findings: Of the 94 unmet cells, 41 were structuring gaps (data exists but is unstructured), 28 were schema gaps (no field exists), and only 17 were publication–access gaps (data held but not published). This indicates that simply "publishing more" is insufficient; regulators must also structure existing narrative disclosures and create new data fields.
• Stress Test Outcomes:
  • FDA: Only 2.5% of known AI devices on the FDA list could be identified via narrow AI terms in primary 510(k) fields; 50.4% carried no AI signal at all.
  • China: The bulk UDI dataset is downloadable but "AI-blind"; AI status must be inferred via free-text keyword matching, which is unreliable.
  • Lineage: Zero of 70 sampled AI product-family records in openFDA exposed a structured predecessor/predicate field; lineage is visible only via narrative PDFs or applicant grouping.

Key Contributions

1. Public Governability Framework: The paper shifts the design target from "transparency" (making information visible) to "public governability" (making information usable for specific governance tasks like procurement, monitoring, and accountability).
2. Auditable Specification: It provides a source-grounded specification of 20 structural requirements (with a 12-item conservative core) that defines what a public regulatory record environment must contain to support lifecycle oversight.
3. Five-Way Gap Typology: It refines the understanding of data deficiencies by distinguishing between schema, publication-access, structuring, linkage, and population gaps. This distinction is critical because each gap type requires a different policy instrument (e.g., new data collection vs. restructuring existing PDFs vs. transparency mandates).
4. Candidate Minimum Public Data Extension: The authors propose a concrete set of 20 structured fields (Table 7) that could be added to existing registration records. These fields focus on categorical summary metadata (e.g., model class, update status, validation population) rather than raw data or trade secrets, making them implementable without compromising confidentiality.

Significance and Claims
The paper claims that its primary contribution is not the observation that current registries are incomplete (a recognized problem), but the conversion of that problem into a reproducible, auditable public-data specification.

• Modest Claims: The authors explicitly state they do not claim the specification represents expert consensus or has been externally validated. Instead, they claim the derivation is transparent (via the traceability matrix) and the findings are stable under conservative sensitivity restrictions.
• Structural Mismatch: The core finding is that public registries are organized around "market authorization visibility" rather than "lifecycle governability." They can confirm a device entered the market but cannot support its governance once there.
• Policy Implications: The study argues that reform cannot be a one-size-fits-all transparency mandate. Because the majority of gaps are "structuring" or "schema" gaps rather than "publication-access" gaps, effective reform requires creating new structured fields and converting narrative disclosures into queryable data, rather than simply releasing existing documents.
• AI Specificity: The paper highlights that the lack of AI-specific metadata (F5) is a deficit of kind, not just degree. Without structured fields for model class and training data provenance, public records cannot support essential tasks like assessing distribution shift, portability, or equity.

In summary, the paper provides a diagnostic framework and a concrete roadmap for transforming public regulatory records from static approval lists into dynamic infrastructure capable of supporting the lifecycle governance of AI medical devices.