Fully Automated Abstraction of Longitudinal Breast Oncology Records with Off-The-Shelf Large Language Models

This study demonstrates that an open-source, HIPAA-compliant pipeline utilizing off-the-shelf large language models can accurately abstract complex longitudinal breast oncology variables from unstructured clinical records with performance approaching that of expert oncologists, thereby offering a scalable solution for creating research-grade datasets without the need for fine-tuning.

The Gist

Imagine you are trying to solve a massive, complex mystery: How do breast cancer patients really fare over many years, and what treatments work best?

To solve this, you need to look at the "case files" of thousands of patients. But here's the problem: these case files aren't neat, organized spreadsheets. They are mountains of messy, handwritten (or typed) notes, scattered across thousands of pages of doctor's visits, lab reports, and pharmacy records.

The Old Way: The Human Detective

Traditionally, to get the answers, you have to hire a team of expert medical detectives (oncologists) to sit down and read every single page of every single patient's file. They have to manually write down:

• "When did the cancer come back?"
• "What drugs did they take?"
• "Why did they stop taking them?"

This is like trying to find a specific needle in a haystack by reading every single piece of hay one by one. It takes years, costs a fortune, and eventually, the detectives get tired and make mistakes. Because it's so hard, researchers can only study small groups of patients, which limits what we can learn.

The New Way: The Super-Intelligent Librarian

This paper introduces a new tool: Large Language Models (LLMs). Think of these as super-intelligent, tireless librarians who have read the entire internet and can understand human language perfectly.

The researchers built a "pipeline" (a specific set of instructions) that acts like a smart search engine. Instead of a human reading 3,000 pages, the system:

1. Sniffs out the relevant paragraphs in the massive pile of notes (like a bloodhound finding a scent).
2. Hands those specific paragraphs to the AI librarian.
3. Asks the librarian: "Based only on these pages, tell me the date of diagnosis, the drug names, and when the cancer returned."
4. Writes the answer into a neat, organized spreadsheet.

The Big Experiment

The researchers tested this on 100 patients with very complex breast cancer histories. These weren't simple cases; the average patient had 7 different rounds of treatment over 6.5 years, and their medical records were the size of a small novel (about 3,100 pages each!).

They compared the AI's work against two groups:

1. The Gold Standard: A team of top-tier breast cancer experts (the human detectives).
2. The Control Group: Research assistants (who are good, but not medical experts).

The Results: The AI is a Star Performer

Here is what happened:

• The "Fact-Checking" Tasks: When the task was simple, like finding a specific date or a lab result (e.g., "Is the tumor HER2 positive?"), the AI was almost perfect. It agreed with the human experts 99% of the time. It was like a calculator that never makes a math error.
• The "Complex Reasoning" Tasks: When the task required piecing together a story (e.g., "Reconstruct the exact order of 7 different drug treatments"), the AI was very good, but not perfect. It was about as accurate as a second human expert.
  • The Analogy: If two human experts read the same messy file, they might disagree on the details 10% of the time. The AI disagreed with the human experts about the same amount. It was essentially "on par" with human experts.
• The Research Assistants: The human assistants did significantly worse than the AI. The AI was faster and more consistent.

The Most Important Question: Does it Matter?

The researchers asked: "If we use the AI's messy data instead of the human's perfect data, does it change the big picture?"

They ran the survival statistics (who lived longer? who had cancer come back?) using both datasets.

• The Result: The results were identical.
• The Analogy: Imagine two different maps of a city. One is drawn by a master cartographer (the human), and one is drawn by a GPS algorithm (the AI). They might have slightly different street names or minor errors, but if you ask, "How long does it take to drive from Point A to Point B?" both maps give you the exact same answer.

Why This Changes Everything

This paper proves that we don't need to hire armies of expensive doctors to read medical records anymore. We can use off-the-shelf AI tools (the kind available to anyone today, without special training) to turn mountains of messy text into clean, usable data.

The Takeaway:
We can now unlock the secrets hidden in millions of patient files that were previously too difficult to read. This means we can answer big medical questions much faster, cheaper, and on a scale that was previously impossible. It's like going from reading one book a year to reading the entire library in a day, without losing the story.

In short: The AI isn't replacing the doctors, but it is replacing the tedious paperwork, allowing doctors to focus on the big picture while the machine handles the heavy lifting of data collection.

Technical Summary

1. Problem Statement

Manual chart abstraction is a critical bottleneck in clinical research, particularly in oncology. Key variables such as disease recurrence, treatment history, and biomarker status are often documented only in unstructured free-text clinical notes rather than structured fields. This limitation:

• Restricts the scale and quality of observational and epidemiologic studies.
• Makes it labor-intensive to build disease registries and retrospective cohorts.
• Leads to incomplete data usage, as much clinically meaningful information remains trapped in narrative records.
• Creates inconsistency across human reviewers.

Current solutions often rely on proprietary systems or require fine-tuning models on specific institutional data, which limits reproducibility, portability, and local control over Protected Health Information (PHI). The authors sought to determine if off-the-shelf, general-purpose Large Language Models (LLMs) could reliably abstract complex longitudinal cancer records without fine-tuning or institution-specific retraining.

2. Methodology

Study Design & Cohort:

• Cohort: 100 patients randomly selected from an institutional breast cancer cohort enriched for complex care (FoundationOne testing). The cohort had a median of 7 lines of therapy over 6.5 years of follow-up.
• Data Volume: The median chart contained ~2.3 million tokens (approx. 3,100 pages) of text, including clinical notes, pathology reports, medication administration records, and demographics.
• Data Source: Unnormalized, unlabeled, and unedited text extracted from the institutional research warehouse (Epic-based). No images or scanned PDFs were included.

Pipeline Architecture:

• Retrieval-Augmented Generation (RAG): The pipeline segments documents into paragraph-length chunks. It uses a hybrid retrieval strategy (exact word search, BM25, and semantic embedding queries) to retrieve relevant text chunks based on the specific variable being abstracted.
• LLM Models: Four commercially available, HIPAA-compliant LLMs were tested: GPT-5, GPT-4o, DeepSeek-R1, and Gemini 2.5 Pro.
• Workflow:
  1. Documents are segmented and indexed.
  2. Relevant chunks are retrieved based on variable-specific prompts.
  3. LLMs process the chunks using schema-constrained prompts to generate structured outputs.
  4. The process is stateless (no chat history), though intermediate outputs are carried forward for multi-pass tasks.
• Constraints: No fine-tuning, no labeled training data, and no access to expert annotations during the abstraction process.

Evaluation & Ground Truth:

• Reference Standard: Four expert breast oncologists abstracted the variables. For systemic therapy, a second oncologist served as an independent comparator to establish inter-expert variability as the benchmark (acknowledging that a single "ground truth" is difficult for complex therapy reconstruction).
• Comparators: Research coordinators were also compared for specific variables.
• Metrics:
  • Non-drug variables: Exact concordance for categorical data; concordance within ±90 days for dates.
  • Systemic therapy: Patient-level Jaccard similarity for drug lists and therapy lines; precision, recall, and F1 scores.
  • Downstream Inference: Comparison of Kaplan-Meier survival curves and Hazard Ratios (HR) between expert-derived and LLM-derived datasets.
• External Validation: A separate cohort of 97 young patients with early-stage breast cancer was tested using the unmodified pipeline.

3. Key Contributions

• Off-the-Shelf Viability: Demonstrated that general-purpose LLMs, without any fine-tuning or retraining, can achieve performance approaching inter-expert variability for complex medical abstraction tasks.
• Open-Source Pipeline: Developed a fully automated, HIPAA-compliant pipeline that is portable across healthcare systems, allowing institutions to keep patient data local while sharing a validated abstraction workflow.
• Comprehensive Benchmarking: Provided a rigorous comparison against multiple human experts (oncologists and coordinators) and evaluated the impact of abstraction errors on downstream statistical inference (survival analysis).
• Scalability: Showed that the pipeline can process massive longitudinal records (median 3,100 pages) in minutes (1–11 minutes per patient), a task impossible for manual review at scale.

4. Key Results

Variable Abstraction Performance:

• High Concordance: The best-performing LLMs achieved:
  • 99% concordance for recurrence status.
  • 100% for germline BRCA1/2 pathogenic variant detection.
  • 99% for hormone receptor status.
  • 96% for HER2 status.
  • 91% for clinical stage and PIK3CA mutation status.
• Systemic Therapy:
  • LLMs approached inter-oncologist variability. The mean patient-level Jaccard similarity for anti-cancer drugs was 0.91–0.92 for LLMs vs. 0.95 for the second oncologist (overlapping confidence intervals).
  • For exact therapy-line reconstruction, LLMs scored 0.73–0.77, while the second oncologist scored 0.86.
  • LLMs outperformed research coordinators significantly on all systemic therapy metrics.
• Error Analysis: Performance was highest for variables anchored in pathology reports and lowest for variables requiring longitudinal clinical reasoning (e.g., reason for discontinuation) or those dependent on data not present in the research warehouse (e.g., external notes).

Downstream Inference:

• Survival Analysis: Survival estimates were nearly identical between expert and LLM datasets.
  • Median Overall Survival: 78.2 months (Expert) vs. 78.2 months (LLMs).
  • Median Recurrence-Free Survival: 34.9 months (Expert) vs. 34.9 months (LLMs).
• Hazard Ratios: Effect estimates for stage and hormone receptor status on survival were statistically similar, with no significant heterogeneity detected via Cochran's Q test.

External Validation:

• In the external cohort of 97 young patients, the unmodified pipeline showed similar performance for recurrence detection (13 vs. 11 by experts) and adjuvant endocrine therapy use (F1 score >0.96).

5. Significance and Implications

• Scaling Research: This approach offers a practical path to creating large, research-grade retrospective datasets from narrative medical records, enabling studies that were previously too labor-intensive to conduct.
• Multi-Institutional Collaboration: Institutions can share a validated, common abstraction pipeline while maintaining data sovereignty (keeping data local), facilitating large-scale harmonized cohort studies.
• Limitations & Future Work:
  • Performance depends on data availability; errors often stem from missing data (e.g., external notes not in the warehouse) rather than model failure.
  • A gap remains between LLMs and experts on the most complex reasoning tasks (e.g., determining intent of therapy).
  • The study highlights that "scaling abstraction" does not automatically equate to "truth"; rigorous study design and transparent definitions remain essential.
• Conclusion: Off-the-shelf LLMs in a fixed retrieval pipeline can abstract clinically meaningful variables with high fidelity, preserving cohort-level statistical inferences and offering a transformative tool for oncology research.