Leveraging Large Language Models to Extract Prognostic Pathology Features in Ewing Sarcoma

This study demonstrates that Large Language Models can accurately extract prognostic histologic features from unstructured pathology reports in Ewing sarcoma, revealing that Neuron-Specific Enolase (NSE) positivity and S100 negativity are significant independent predictors of inferior survival, particularly in non-metastatic patients.

The Gist

Imagine you have a massive library filled with millions of old, dusty books. These books contain the medical histories of thousands of children with a rare bone cancer called Ewing Sarcoma. The problem? The stories inside aren't written in neat, organized lists. They are scribbled in messy, handwritten-style paragraphs on pages that have been scanned into blurry, black-and-white images.

For decades, doctors couldn't read these stories quickly enough to find patterns. They knew that if a child's cancer had spread (metastasized), it was dangerous. But they were missing hidden clues buried in those messy paragraphs that could explain why some children did better than others, even when their cancer hadn't spread.

This paper is about using a super-smart digital detective (an Artificial Intelligence called a Large Language Model, or LLM) to read those messy books, find the hidden clues, and organize them into a neat spreadsheet.

Here is the story of how they did it, broken down simply:

1. The Problem: The "Dark Data" Library

Think of the pathology reports (the medical descriptions of the tumor) as "Dark Data." They exist, but they are locked away in unstructured text.

• The Analogy: Imagine trying to find out which flavor of ice cream is most popular by reading 900 different handwritten letters from people, where some letters are smudged, some are in French, and some have coffee stains. Doing this by hand would take a human team years.
• The Goal: The researchers wanted to unlock this data to see if specific "flavors" (biological markers) in the cancer cells predicted who would survive longer.

2. The Solution: The AI Detective

The team hired a digital detective (specifically, an AI model called OpenAI o3) to do the heavy lifting.

• The Process: First, they used a tool called OCR (Optical Character Recognition) to turn the blurry scanned images into text. It was like trying to read a photocopy of a photocopy—full of typos and weird symbols.
• The Magic: The AI detective was then given a list of 17 specific "clues" to look for (like NSE and S100, which are proteins found in the cancer cells). The AI read through the messy text and said, "Ah, I see this patient had NSE," or "This one had S100."
• The Result: The AI was incredibly fast and surprisingly accurate. In fact, when tested against human doctors, the AI got 98.1% of the answers right, beating the human experts who got 91.4% and 95.9%. The AI didn't get tired, didn't get distracted by coffee stains, and didn't miss a word.

3. The Big Discovery: Two New "Traffic Lights"

Once the AI organized all the data, the researchers looked at the results and found two major "traffic lights" that change how we understand the disease:

• The Red Light (NSE): They found that if a child's tumor was positive for a marker called NSE, it was a bad sign.

  • The Analogy: Imagine a car with a broken engine. Even if the car hasn't crashed into a wall yet (no metastasis), the broken engine means it's likely to break down soon.
  • The Stat: Children with NSE-positive tumors had more than double the risk of dying compared to those without it. This was especially true for children whose cancer hadn't spread yet. This is huge because current treatment plans often treat all "non-spread" cases the same, but this suggests some "non-spread" cases are actually much more dangerous than we thought.

• The Green Light (S100): On the flip side, they found that if a tumor was positive for S100, it was a good sign.

  • The Analogy: This is like a car with a super-efficient engine. Even if the road is bumpy, this car is built to last.
  • The Stat: Children with S100-positive tumors had a much better chance of survival.

4. Why This Matters

Before this study, doctors were driving blind, relying mostly on whether the cancer had spread to decide treatment. They were ignoring the "engine details" written in the messy reports.

• The Takeaway: This study proves that we can use AI to rescue "lost" information from the past. It's like finding a treasure map in an old attic.
• The Future: Now that we know NSE is a "Red Light" and S100 is a "Green Light," doctors might be able to design better clinical trials. They could give stronger, more aggressive treatments to the kids with the "broken engines" (NSE+) and perhaps less toxic treatments to those with the "super engines" (S100+), sparing them from unnecessary side effects.

In a Nutshell

This paper is a victory for technology meeting medicine. It shows that by using a smart AI to clean up and read thousands of old, messy medical reports, we can discover life-saving secrets that were hiding in plain sight for decades. It turns a library of unreadable scribbles into a clear guide for saving more children's lives.

Technical Summary

1. Problem Statement

Current risk stratification for Ewing sarcoma (ES) relies heavily on clinical factors, specifically metastatic status and tumor size, failing to incorporate histologic heterogeneity as a prognostic indicator. While pathology reports contain rich biological data (e.g., immunohistochemical markers), this information is locked in unstructured narrative text within scanned PDF documents.

• The Bottleneck: Manually extracting structured data from these legacy documents across multi-institutional cohorts is labor-intensive, error-prone, and historically prohibitive for large-scale retrospective analysis.
• The Opportunity: "Dark data" from over 20 years of clinical trials remains inaccessible to conventional statistical methods, potentially hiding novel biomarkers that could refine risk stratification.

2. Methodology

The study employed a retrospective cohort design utilizing a novel LLM-based data abstraction pipeline.

• Data Source:
  • Cohort: 931 unique patients from 185 institutions across six Children's Oncology Group (COG) clinical trials (AEWS0031, AEWS1031, AEWS1221, AEWS02B1, APEC14B1, AEWS07B1) spanning 21 years.
  • Input: Digitized diagnostic pathology reports (scanned PDFs) in English and French, varying in quality and clarity.
• Pre-processing (OCR):
  • Raw PDF images were converted to machine-readable text using Tesseract, an open-source Optical Character Recognition (OCR) engine.
  • The pipeline handled significant noise, including typographical errors, non-English characters, and scanning artifacts.
• LLM Extraction Pipeline:
  • Model: OpenAI o3 model.
  • Task: Extract structured variables from the noisy OCR text.
  • Target Variables:
    1. 17 Immunohistochemical (IHC) Markers: Including CD99, NSE, S100, NKX2.2, FLI-1, ERG, Myogenin, etc. (Classified as Positive, Negative, or Not Specified).
    2. CD99 Staining Patterns: Spatial distribution (e.g., Membranous, Cytoplasmic, Diffuse, Patchy).
  • Output: Structured JSON/CSV data.
• Validation Strategy:
  • Ground Truth (GT): A subset of 197 reports was manually annotated by a pediatric resident and a pediatric oncologist. Discrepancies were resolved via consensus.
  • Cross-Validation: A subset of 48 cases was used to compare LLM performance against individual human experts and the consensus GT.
  • Metrics: Accuracy, Precision, Recall, and F1-score.
• Statistical Analysis:
  • Endpoint: Overall Survival (OS).
  • Methods: Kaplan-Meier analysis (log-rank test) and multivariable Cox proportional hazards regression.
  • Adjustments: Models adjusted for metastatic status to determine independent prognostic value.

3. Key Contributions

• Scalable "Dark Data" Rescue: Demonstrated the first large-scale application of LLMs to extract structured histologic data from noisy, multi-institutional, legacy scanned pathology reports for a rare pediatric cancer.
• Superior Extraction Accuracy: Validated that an LLM (o3) can achieve 98.1% accuracy in a cross-validation subset, outperforming human annotators (Pediatric Resident: 91.4%; Pediatric Oncologist: 95.9%).
• Discovery of Novel Biomarkers: Identified Neuron-Specific Enolase (NSE) and S100 as significant, independent prognostic biomarkers in Ewing sarcoma, which are not currently standard in risk stratification models.
• Multilingual Capability: Successfully extracted data from reports in both English and French, demonstrating the model's robustness to language variations in historical archives.

4. Key Results

A. LLM Performance

• IHC Extraction: The LLM achieved a weighted average accuracy of 94% across 17 IHC markers (Weighted Precision: 0.997; F1-score: 0.966) despite significant OCR noise.
• Pattern Recognition: The model achieved 90.1% accuracy in classifying complex CD99 staining patterns (e.g., distinguishing "Membranous" vs. "Membranous Diffuse").
• Human vs. AI: In the cross-validation set, the LLM (98.1%) significantly outperformed human experts, suggesting LLMs are less susceptible to fatigue and reading errors associated with poor-quality legacy documents.

B. Prognostic Findings

• Neuron-Specific Enolase (NSE):
  • Overall: NSE positivity was associated with significantly inferior survival (HR 2.15, 95% CI 1.15–4.02, p=0.016).
  • Subgroup Analysis: The risk was most pronounced in non-metastatic disease, where NSE positivity conferred a 5.64-fold increased risk of death (p=0.0055). No significant association was found in metastatic disease.
• S100:
  • Overall: S100 positivity was associated with improved survival (HR 0.58, 95% CI 0.34–1.00, p=0.046).
  • Subgroup Analysis: A protective trend was observed in non-metastatic patients (HR 0.44), though it did not reach strict statistical significance in the subgroup alone.
• CD99: While 91% of cases were CD99 positive (essential for diagnosis), no specific staining pattern or intensity was significantly associated with survival outcomes in this cohort.

5. Significance and Implications

• Refining Risk Stratification: The study challenges the current paradigm that relies solely on clinical staging. It suggests that NSE status identifies a high-risk subset of patients with localized disease who may require more aggressive therapy, while S100 may indicate a more indolent course.
• Methodological Advancement: The workflow proves that AI can reliably convert unstructured, noisy historical medical records into high-fidelity structured datasets, unlocking vast archives for retrospective discovery without the need for new prospective data collection.
• Future Directions: The authors recommend integrating NSE and S100 status into future prospective clinical trial designs for Ewing sarcoma. The study also highlights the potential for AI to guide expert pathology annotation and image selection based on extracted staining patterns.

Conclusion: This research validates the utility of Large Language Models as a scalable, accurate tool for "rescuing" prognostic data from historical clinical trials, revealing that immunohistochemical markers (specifically NSE and S100) hold significant, independent prognostic value in Ewing sarcoma that has been previously overlooked due to data extraction limitations.