Automated Identification of Complex Percutaneous Coronary Intervention from Cardiac Catheterization Reports Using Large Language Models

Hospitals generate a vast ocean of unstructured text every day. Among the most critical of these documents are the narrative reports written after a heart catheterization, a procedure where doctors thread a thin tube through the arteries to examine the heart's blood vessels. These reports are rich with details about the anatomy of a patient's heart and the specific techniques used to fix blockages, but they are written in free-flowing paragraphs rather than neat, check-box fields. For researchers and quality improvement teams, this creates a significant bottleneck. To study how well these procedures work or to track the complexity of cases across a health system, human experts must manually read thousands of these reports and extract specific data points. This process is slow, expensive, and difficult to scale, often limiting the size and speed of important medical studies.

The rise of large language models, a type of artificial intelligence capable of understanding and generating human-like text, offers a potential solution to this problem. These systems can be trained to read complex documents and pull out specific facts, much like a very fast, tireless research assistant. However, it remained unclear whether these tools could handle the nuanced, specialized language found in heart procedure notes, particularly when asked to identify "complex" cases that involve multiple blockages or difficult techniques. A team of researchers set out to test this question using a large collection of real-world heart catheterization reports.

The researchers gathered 1,412 de-identified procedure notes from three different hospitals within the Yale New Haven Health system. These documents covered procedures performed between 2011 and 2017 and included a mix of diagnostic exams and actual interventions where doctors placed stents or used balloons to open blocked arteries. To create a reliable standard for comparison, a group of expert heart specialists manually read every single note. They first determined whether a report described a procedure where a stent or balloon was actually used to treat a coronary artery. For those that did, they then extracted six specific details that define a "complex" procedure: the number of blood vessels treated, the number of distinct blockages fixed, the total number of stents placed, whether a specific two-stent technique was used for a branching vessel, the total length of all stents combined, and whether a chronic total occlusion—a complete, long-standing blockage—was treated.

With this human-verified "gold standard" in place, the team tested three different artificial intelligence models to see if they could replicate the experts' work. They chose models that were open-source, meaning their code and weights were publicly available, which allowed the researchers to run them on secure, local computers to protect patient privacy. One model was a massive, general-purpose system with 70 billion parameters, a measure of its size and capacity for reasoning. The other two were smaller models specifically pre-trained on medical literature, with 7 billion parameters each, designed to understand clinical terminology. The researchers fed the same text prompts and reports to all three models, asking them to identify if a procedure occurred and to extract the six specific variables.

The results showed a clear distinction in capability. The large, general-purpose model significantly outperformed the two smaller, medical-specific models. When asked simply to identify whether a report described a heart procedure, the large model achieved perfect sensitivity, meaning it missed zero actual procedures, and correctly identified non-procedures with high accuracy. In contrast, the smaller models struggled, with one frequently mistaking non-procedure notes for procedures and the other failing to recognize actual procedures almost entirely.

The gap widened when the task required extracting the six complex details. The large model successfully identified the number of stents and the total length of stents with very high accuracy, often exceeding 90 percent. It also performed well at identifying the number of vessels treated. However, its performance dipped on variables that required more interpretation, such as counting the exact number of distinct blockages or determining if a specific two-stent technique was used for a branching vessel. In these cases, the model sometimes missed details or misinterpreted the context, though it still remained far more accurate than the smaller models. The smaller medical models, despite their specialized training, failed to consistently extract these details, often producing results that were too unreliable for research use.

The study also examined whether the models performed differently across the three hospital sites. While the large model maintained high accuracy at all three locations, there were noticeable variations in how well it performed at each site, likely due to differences in how doctors at each hospital wrote their notes. The smaller models showed even greater inconsistency, with their performance fluctuating wildly depending on the location. This suggests that while the large model is robust, the way information is documented can still influence the results.

The researchers analyzed the errors made by the best-performing model to understand where it struggled. They found that mistakes often occurred when the documentation was ambiguous or fragmented. For example, the model sometimes confused a diagnostic test with a treatment, or it struggled to distinguish between a stent that was successfully placed and one that was attempted but not deployed. It also had difficulty when a report mentioned a blockage that was not actually treated, or when the description of a chronic total occlusion was implied rather than explicitly stated. These errors highlight that while the technology is powerful, it is not yet perfect at navigating the messy, inconsistent reality of human medical writing.

Ultimately, the study demonstrates that a large, open-source artificial intelligence model can accurately extract complex data from unstructured heart procedure reports, a task that has traditionally required hours of manual labor. The findings suggest that for many variables, particularly those that are explicitly stated like the number of stents, these tools can achieve a level of accuracy suitable for research. However, for variables requiring deep contextual interpretation, the technology still faces challenges. The work indicates that the future of scalable cardiovascular research may lie in using these powerful models to handle the bulk of data extraction, potentially combined with human oversight for the most difficult cases, rather than relying solely on smaller, specialized models or manual review.