📄 medicine

Validating Acute Coronary Syndrome ICD Codes as Training Labels for Clinical AI

This 10-year retrospective study of nearly 620,000 emergency department encounters validates that while ICD-coded Acute Coronary Syndrome labels exhibit high sensitivity and specificity, their moderate positive predictive value is driven by systematic, interpretable errors—primarily suspected cases later ruled out or coder omissions—making them suitable training labels for AI screening models where missing a diagnosis is more critical than over-identification.

Original authors: Daniel K. Amponsah, Gabrielle Bunney, Kate Miller, Ariadna Garcia, Anna Graber-Naidich, Keejeong Ryu, Himani Darji, Marilyn Ndukwe, Valerie J.F. McDonald, William M. Mehring, Jada Henderson, David Lei

Published 2026-10-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel K. Amponsah, Gabrielle Bunney, Kate Miller, Ariadna Garcia, Anna Graber-Naidich, Keejeong Ryu, Himani Darji, Marilyn Ndukwe, Valerie J.F. McDonald, William M. Mehring, Jada Henderson, David Lei, Marium Rizvi, Adenine Cembellin-Kao, Sylvia D. Kreibig, Celina M. Yong, William F. Fearon, Maame Yaa A.B. Yiadom

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the emergency department, time is the most critical resource. When a patient arrives with chest pain, doctors must quickly decide if they are suffering from acute coronary syndrome, a life-threatening condition where blood flow to the heart is blocked. Missing this diagnosis can be fatal, but ordering a full battery of heart tests on every single person who walks through the door is impossible. To solve this, hospitals are increasingly turning to artificial intelligence. These computer programs learn to spot danger by studying millions of past medical records, looking for patterns that human eyes might miss. However, for a computer to learn correctly, it needs a teacher that knows the truth. In the world of medical data, that teacher is often a billing code—a short, standardized label assigned to a patient's visit to handle insurance and hospital accounting. The big question facing researchers is whether these administrative labels are accurate enough to teach a machine how to save lives, or if they contain too many errors to be trusted.

A team of researchers at Stanford University set out to answer this question by looking at a decade of emergency room visits. They examined nearly 620,000 adult encounters between 2015 and 2024, comparing the billing codes assigned to patients against the actual final diagnosis written by the treating medical team. The researchers wanted to see if the computer-readable codes matched the human doctors' conclusions. They found that the billing codes were remarkably reliable for their intended purpose. When the codes said a patient had a heart condition, the doctors confirmed it in 94 out of 100 cases. Even more importantly, the codes rarely missed a true case; they failed to flag a patient with a confirmed heart condition only 4 times out of every 10,000 visits. This means the system is extremely good at catching the people who need help, which is the most important goal for a screening tool.

The study did reveal that the codes were not perfect, but the errors followed a clear and understandable pattern rather than being random mistakes. When the codes were wrong, it was usually because they flagged a patient who was suspected of having a heart attack but was later cleared by doctors. In these cases, the patient had symptoms that looked very much like a heart attack, such as inflammation of the heart muscle or stress-induced heart issues, and the medical team spent time ruling out the most dangerous possibilities. From the perspective of an artificial intelligence designed to screen for danger, these are not failures. A system that flags these patients ensures that no one with a potential heart crisis is overlooked, even if it means some people who turn out to be safe get a closer look. The few times the system missed a real heart attack, it was because the medical coder simply overlooked the doctor's notes, not because the diagnosis was unclear.

These findings suggest that using billing codes to train artificial intelligence is a sound strategy for emergency departments. The computer learns to recognize the broad, urgent patterns of heart trouble, including cases that turn out to be less severe, because the cost of missing a real heart attack is far higher than the cost of running an extra test on someone who is safe. The researchers confirmed that the errors in the data are systematic and predictable, meaning they can be accounted for when building these models. By relying on these codes, hospitals can develop tools that act as a highly sensitive safety net, ensuring that the most critical patients are identified immediately upon arrival. The study concludes that while no single data point is flawless, the collective accuracy of these codes is sufficient to support the next generation of life-saving medical technology.

Technical Summary: Validating Acute Coronary Syndrome ICD Codes as Training Labels for Clinical AI

Problem Statement
Artificial intelligence (AI) models designed to support emergency department (ED) screening for Acute Coronary Syndrome (ACS) rely heavily on structured Electronic Health Record (EHR) data for training. International Classification of Diseases (ICD) codes are the most accessible, large-scale labeling source for such retrospective datasets. However, ICD codes are primarily administrative tools assigned post-encounter by coders, not clinical diagnoses. Their accuracy as proxies for clinical truth in contemporary ED populations remains uncertain. In the context of ACS screening, where missing a diagnosis (false negative) carries high morbidity and mortality risks, the validity of ICD codes as training labels is critical. Prior validation work has largely focused on inpatient registries or non-US populations, leaving a gap in understanding the performance of ACS ICD codes within US ED settings and the specific nature of discrepancies between coding and clinical documentation.

Methodology
The authors conducted a 10-year retrospective cohort study (January 1, 2015 – December 31, 2024) at a tertiary academic medical center in Stanford, California.

Cohort: The study analyzed 619,599 adult ED encounters, excluding trauma and stroke activations.
Reference Standard: The "ground truth" was defined as the final diagnosis documented by the treating clinical team (reviewing ED notes, progress notes, consults, and discharge summaries), rather than an independent adjudication of pathophysiologic truth. This pragmatic approach aligns with the reality that AI models are trained on documented diagnoses.
Label Definition: ICD-based ACS was defined by the presence of specific ICD-9-CM or ICD-10-CM codes for STEMI, NSTEMI, or Unstable Angina (UA) within the first 10 final hospital diagnoses.
Chart Review Strategy:
- All encounters with an ACS ICD code ( $n=5,961$ ) were reviewed.
- To capture potential false negatives, an enrichment strategy was employed for encounters without an ACS code: those with both a Coronary Artery Disease (CAD) ICD code and abnormal troponin results were also reviewed ( $n=510$ ).
- The remaining 613,128 encounters were treated as non-ACS.
Review Process: Nine clinically experienced reviewers, blinded to ICD classifications, performed the chart review. Discrepant cases (where ICD and chart review disagreed) underwent a secondary, multi-stage review by two independent teams and expert adjudication to characterize the reasons for disagreement.
Metrics: The study calculated sensitivity, specificity, Positive Predictive Value (PPV), and Cohen's kappa, comparing ICD codes against the chart-reviewed reference standard.

Key Results

Agreement Statistics: Against the chart-reviewed reference standard, ICD codes demonstrated:
- Sensitivity: 94.4% (95% CI: 93.7%–95.0%)
- Specificity: 99.73% (95% CI: 99.72%–99.74%)
- PPV: 72.4% (95% CI: 71.2%–73.5%)
- Cohen's Kappa: 0.82
Discrepancy Rates: Overall disagreement occurred in only 0.31% of all encounters.
- False Positives (ICD+ / Chart-): 1,647 cases (0.27% of total).
- False Negatives (ICD- / Chart+): 257 cases (0.04% of total).
Analysis of Errors:
- False Positives: The primary driver (80.3%) was cases where ACS was suspected during the encounter but later ruled out (e.g., clinical mimics like pericarditis or stress cardiomyopathy, or initial ED diagnoses revised upon inpatient evaluation). These were not arbitrary coding errors but represented patients who clinically warranted ACS evaluation.
- False Negatives: The primary driver (86.0%) was coder omission of documented ACS, rather than clinical disagreement.
Demographics: The chart-reviewed cohort was older (mean age 69.3 vs. 50.9) and had a higher prevalence of Medicare insurance compared to the general ED population, consistent with the enrichment criteria.

Significance and Claims
The paper claims that ACS ICD codes are suitable training labels for AI models designed to screen ED patients for ACS, particularly when the clinical goal is to minimize missed diagnoses (high sensitivity) rather than to maximize specificity.

Systematic vs. Random Error: The study concludes that labeling errors are systematic and interpretable rather than random. The dominant error pattern (false positives representing patients who warranted evaluation) reinforces the utility of these labels for screening models, as such models should flag these "ACS-adjacent" cases for further workup (e.g., ECG).
Clinical Context: The low false-negative rate (0.04%) suggests that ICD-based labeling would miss only a small fraction of true cases (approximately 26 per year at this institution), supporting the use of these codes for developing early warning systems.
Limitations and Scope: The authors acknowledge that the study was single-center and that the reference standard was the treating team's documentation rather than pathophysiologic truth. They note that the enrichment strategy may underrepresent Unstable Angina (which lacks abnormal troponins), potentially leading to an underestimation of the false-negative rate. Additionally, the study spans the ICD-9 to ICD-10 transition and the adoption of high-sensitivity troponin assays, meaning the results reflect a mix of coding and diagnostic eras.

In summary, the authors argue that despite the administrative nature of ICD codes, their high sensitivity and the specific, clinically informative nature of their errors make them a valid and practical foundation for training clinical AI models aimed at ACS screening in the emergency department.

Technical Summary: Validating Acute Coronary Syndrome ICD Codes as Training Labels for Clinical AI

More like this