📄 medicine

Population representativeness cannot be assessed for most FDA-authorized AI medical devices targeting high-disparity conditions

This study reveals that for 76.6% of FDA-authorized AI medical devices targeting high-disparity conditions, population representativeness cannot be assessed from public records due to unreported race data, a gap that persists even when reviewing underlying peer-reviewed literature and highlighting the failure of current disclosure guidance to ensure equity assessability.

Original authors: Saptarshi Purkayastha, Aravind V. Kuruvikkattil, Chaitra Sree Sammeta, Aiswarya N. Perumbilly, Rashmita Kudamala, Judy W. Gichoya

Published 2026-10-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saptarshi Purkayastha, Aravind V. Kuruvikkattil, Chaitra Sree Sammeta, Aiswarya N. Perumbilly, Rashmita Kudamala, Judy W. Gichoya

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Artificial intelligence has quietly become a standard tool in modern medicine, helping doctors read X-rays, spot tumors, and predict heart risks. These tools are not magic; they are computer programs trained on vast amounts of patient data to recognize patterns. For a program to work well for everyone, it must be trained on a mix of people who look like the patients it will eventually treat. If a tool learns only from one type of person, it may fail when used on someone different, potentially leading to missed diagnoses or incorrect treatment. This is a matter of fairness and safety. When a medical device is approved for use, the government agency that regulates it, the Food and Drug Administration, publishes a summary of the evidence. These documents are meant to be the public record, showing who was tested and how the device performed. The critical question is whether these summaries tell the full story about who the device was tested on, especially for diseases that affect some racial and ethnic groups more severely than others.

A team of researchers set out to answer this question by examining nearly every artificial intelligence medical device approved in the United States up to the end of 2025. They looked at 1,428 devices, ranging from tools that scan brain images to those that analyze heart rhythms. Their goal was to see if the public documents allowed anyone to judge whether the people tested on the devices matched the people who actually suffer from the diseases the devices are meant to treat. The researchers found a significant gap in the information available. For about three out of every four devices designed to treat conditions where racial disparities are known to exist, the public documents did not report the race or ethnicity of the people in the test data. Without this basic information, it is impossible to tell if the device was tested on a fair mix of patients or if it was tested only on a group that does not represent the broader population.

The researchers developed a specific checklist to evaluate these documents, looking for details like the race, age, and location of the patients involved in the testing. They found that while some information was present, the most crucial data regarding race and ethnicity was missing in the vast majority of cases. This missing information was not just a matter of the summaries being too short; the researchers dug deeper and looked at the original scientific studies behind the devices. Even when they found the full research papers, the data was still missing in most cases. This suggests that the problem lies not just in how the government summarizes the information, but in the research itself. The studies used to prove these devices work often do not include enough detail about the demographics of the patients involved.

The study also revealed that when the data was present, it often showed that the testing groups did not match the groups most affected by the disease. For example, some devices designed to detect skin cancer, a condition that can be harder to diagnose in people with darker skin, were tested almost entirely on people with light skin. In other cases, devices for heart conditions were tested on populations that did not reflect the higher rates of heart disease found in certain minority communities. The researchers noted that while some devices did provide excellent, detailed information about their testing groups, these were the exception rather than the rule. Most of the time, the public record simply did not contain the numbers needed to make a fair judgment.

One of the most striking findings was that the researchers could not reliably grade the fairness of the devices based on the text alone. They found that when asked to interpret vague descriptions or judge how well a group of patients matched a disease, different experts often disagreed. However, they could reliably agree on simple, binary facts: whether a number was reported or not. This led the team to conclude that the current system of reporting is insufficient for ensuring equity. The documents are too often silent on the very details that matter most for fairness. The researchers propose a new, simple standard for these reports, requiring manufacturers to list specific numbers about the race, age, and background of the people tested, rather than just writing general statements.

The implications of this work are clear for patients, doctors, and hospitals. If a hospital wants to buy a new artificial intelligence tool to help diagnose a disease, they currently cannot look at the public approval documents to see if that tool was tested on people like their own patients. They are left guessing. The study shows that for the most common and deadly diseases, the tools being deployed are often built on data that does not reflect the reality of the people who need them most. The researchers emphasize that this is not a problem that can be solved by simply asking for more text; the industry must start reporting specific, structured data. Until then, the public record remains incomplete, leaving a critical blind spot in the safety and fairness of the medical technology that is increasingly shaping American healthcare.

Technical Summary: Population Representativeness in FDA-Authorized AI Medical Devices

Problem Statement
The study addresses a critical gap in the regulatory oversight of Artificial Intelligence/Machine Learning (AI/ML) medical devices. While previous audits have focused on transparency (whether demographic data is disclosed in FDA decision summaries), they have not adequately addressed equity (whether the disclosed data aligns with the populations bearing the highest disease burden). The authors posit that for devices targeting conditions with documented demographic disparities (e.g., stroke, heart disease, specific cancers), it is currently impossible to assess if the training and validation cohorts are representative of the affected populations. This "population-data alignment" gap prevents clinicians, purchasers, and patients from verifying if a device was tested on the people who need it most, potentially perpetuating algorithmic bias and health inequities.

Methodology
The authors conducted a cross-sectional audit of 1,428 FDA-authorized AI/ML devices with retrievable public decision summaries as of December 30, 2025. The study employed a novel, multi-step methodology:

Equity Readiness Assessment (ERA): A 19-item instrument was developed across four domains: Demographic Reporting Depth, Population-Data Alignment, Equity-Relevant Design Features, and External Evidence Linkage.
Epidemiologic Crosswalk: A pre-registered mapping system linked each device's target condition to documented demographic disparity profiles (using CDC, GBD, and ACS data) to determine if the condition carries a high disparity burden.
Reliability-Driven Tiering: To ensure scientific rigor, the study prioritized inter-rater reliability. Two blinded human reviewers coded a stratified validation set ( $n=153$ $n = 153$ ). Items were classified into:
- Tier A (ERA-9): Items with high inter-rater agreement (Cohen's $\kappa \ge 0.60$ or prevalence-adjusted equivalent). These form the validated core for all inferential claims.
- Tier B: Items with low agreement, reported only as descriptive statistics.
Dual-Source Validation: For a subset of devices ( $n=68$ ) with retrievable pivotal peer-reviewed publications, the authors re-scored the devices using both the FDA summary and the primary literature to determine if the data gap was an artifact of regulatory summarization or a fundamental lack of data in the evidence base.
Model-Assisted Extraction: Large Language Models (LLMs) were used for extraction and classification, validated against human consensus. Inferential claims were restricted to items where model-versus-human agreement matched or exceeded human-vers-human agreement.

Key Results

Primary Finding (Unassessability): For 76.6% (95% CI 73.8–79.3%) of the 899 devices targeting high-disparity conditions, population representativeness cannot be assessed because race or ethnicity data is entirely absent from the public decision summary. This finding is robust to variations in how conditions are classified.
Data Gap Source: Consulting the underlying peer-reviewed literature rescued the ability to assess representativeness for only 5 out of 27 previously unassessable devices. This indicates the missing data is a property of the underlying evidence base, not merely a result of terse regulatory writing.
Under-Representation: Among the 210 devices where assessment was possible, 84.8% showed under-representation of the highest-burden demographic groups.
Reliability Constraints: Graded equity judgments (e.g., scoring the degree of representativeness) proved unreliable between coders ( $\kappa = 0.12$ ). Only binary "presence-of-reporting" items were reproducible. Consequently, the study anchors all statistical inferences on the ERA-9 core (9 binary items) rather than a composite score.
Validation Settings: No device in the cohort had a decision summary documenting validation in a community, safety-net, or lower-resource setting. Validation occurred almost exclusively in well-resourced academic environments.
Regulatory Impact: Reporting improved modestly after the 2021 Good Machine Learning Practice (GMLP) principles, but subsequent guidance (PCCP, lifecycle management) showed no significant association with improved alignment.

Key Contributions

ERA-9 Instrument: The authors distill the 19-item ERA into a validated 9-item core (ERA-9) that reliably measures the presence of equity-relevant data, distinguishing between what can be measured (binary reporting) and what cannot (graded alignment) from current regulatory prose.
Epidemiologic Crosswalk: A publicly released mapping of device target conditions to disparity profiles, enabling the identification of high-disparity conditions.
Device-Level Dataset: An open dataset of 1,428 devices with their equity readiness scores, allowing for longitudinal surveillance.
Proposed Minimum Standard: The paper proposes eight structured fields for FDA decision summaries (Table 5), moving away from narrative prose to structured data fields (e.g., specific cohort race/ethnicity distributions, site types) to make equity auditing computable.

Significance and Claims
The paper claims that current disclosure guidance has failed to produce either population-data alignment or its assessability. The authors argue that transparency (disclosing data) is insufficient without alignment (ensuring data matches the burden).

For Regulators: The study demonstrates that existing frameworks (STANDING Together, HEAL) do not solve the problem of retrospective third-party assessment of regulatory summaries. It calls for a mandatory minimum equity-reporting standard with structured fields to replace narrative statements.
For Clinicians and Purchasers: The findings indicate that for most AI devices targeting high-disparity conditions, it is currently impossible to verify who was tested. This lack of information should be a critical factor in procurement decisions.
For Researchers: The study establishes a methodological floor for equity auditing, demonstrating that graded equity scores cannot be reliably coded from current regulatory documents. It highlights that the "alignment gap" is a structural failure of the evidence generation process, not just a reporting deficiency.

The authors explicitly state that their work is a "diagnosis" of the current state: the 19-item composite is not yet a reliable instrument, and the primary contribution is the validated ERA-9 core and the demonstration that the gap is avoidable (as shown by counter-examples like LUMA and Aidoc) but currently unaddressed by regulatory requirements.

Technical Summary: Population Representativeness in FDA-Authorized AI Medical Devices

More like this