In-Domain Supervised Pathology Report Classification: A Reproducible Pipeline from Data Curation to Production-Matched Evaluation

Technical Summary: In-Domain Supervised Pathology Report Classification

Problem Statement
Supervised biomedical Natural Language Processing (NLP) models often suffer from significant performance degradation when deployed across different cancer registries due to distribution shifts. This issue arises from heterogeneities in report styles, data sources, and clinical workflows. Specifically, the MOSSAIC OncoID classifier, trained on data from the Seattle SEER registry, achieved high accuracy locally but exhibited a high false-positive rate (FPR) when deployed at the Kentucky Cancer Registry (KCR). This resulted in an unmanageable manual review burden for registry staff, as the model flagged too many non-reportable reports. Furthermore, existing supervised models often rely on datasets representing single cancer types or single facilities, limiting their generalizability.

Methodology
The authors propose a reproducible, in-domain supervised pipeline designed to train a classifier using routinely collected data from the target registry (KCR). The methodology encompasses data curation, model training, and production-matched evaluation.

• Data Curation and Stratification:
  The pipeline utilizes pathology reports from the KCR Feeds database, stratifying data into three subsets to ensure representation:

  1. Linked Reports ($M_{linked}$/$H_{linked}$): Reports linked to cancer cases in the SEERDMS system. These are stratified by cancer topography (primary site) to ensure broad representation across cancer sites.
  2. Unlinked Reports ($M_{unlinked}$/$H_{unlinked}$): Reports categorized by an Oncology Data Specialist (ODS) but not linked to a specific case. These are stratified by ODS-assigned categories (e.g., "Needs Review," "Metastases," "No Cancer").
  3. Seattle Model Negatives ($SM^-$): Reports labeled as non-reportable by the Seattle OncoID model. These are excluded from training to avoid label noise (as they lack human validation) but are included in the holdout set to simulate the real-world data stream.

  The training data is split 80:20 for linked reports and 90:10 for unlinked reports to match the observed proportions in production data streams.

• Model Architecture and Training:
  The study employs the OncoID classifier, a Hierarchical Self-Attention Network (HiSAN) architecture implemented via the BARDI and FrESCO software packages.

  • Preprocessing: Vocabulary building includes a minimum word count of 5, removal of special characters, and exclusion of common first names.
  • Class Imbalance Handling: To prioritize recall (minimizing missed cancer cases), the authors experimented with loss function weighting schemes (30:1, 15:1, and unweighted) for non-reportable to reportable classes.
  • Operating Point Selection: The goal is to maintain an extremely low False Negative Rate (FNR) while keeping the False Positive Rate (FPR) low enough to be manageable for human reviewers.

• Evaluation and Validation:

  • Production-Matched Holdout: A holdout set ($H$) of 418,166 reports mirrors the production environment, including both human-labeled and model-filtered data.
  • Blinded Manual Audit: A two-stage manual review of 600 reports was conducted to estimate true positive prevalence and quantify label noise. Two independent ODS staff reviewed reports, with a third adjudicating conflicts, strictly following 2024 SEER Coding and Reportability Guidelines.
  • Qualitative Review: Authors analyzed 30 false negatives and 30 false positives to identify patterns in misclassifications.

Key Results

• Performance Improvement: The Kentucky-trained OncoID model significantly outperformed the Seattle-trained baseline on the KCR holdout set.
  • F1 Score: Improved from 0.860 (Seattle) to 0.922 (Kentucky).
  • False Negative Rate (FNR): Reduced from 0.010 to 0.003.
  • False Positive Rate (FPR): Reduced from 0.183 to 0.097.
• Operating Point Selection: The model trained with a 15:1 loss weighting scheme was selected as the final operating point, achieving an FNR of 0.003 and an FPR of 0.097. This balance prioritizes case capture while reducing the manual review workload by approximately 47% compared to the baseline (18.3% vs. 9.7% of non-reportable reports flagged).
• Label Noise Analysis: The blinded manual audit revealed that the original dataset labels contained substantial noise. The estimated positive prevalence in the holdout set dropped from 0.500 (original label) to 0.398 (audited label), indicating that roughly 20% of reports labeled as "reportable" were actually non-reportable. Errors were concentrated in rare primary sites (e.g., tongue, mouth, gum) and specific categories like "Project-Specific."
• Site-Specific Performance: The model achieved near-zero FNRs for common cancers (breast: 0.0001, lung: 0.0005) but showed higher FNRs for rare cancers, reflecting the difficulty of acquiring balanced training data for rare conditions.

Significance and Claims
The paper claims that training a supervised classifier on in-domain data that matches local formatting, language, and terminology is essential for overcoming distribution shifts in cancer registry NLP. The primary contribution is a reproducible recipe for other cancer registries to:

1. Build in-domain training sets using routine data feeds.
2. Construct production-matched holdout sets that include both human-labeled and model-filtered data.
3. Select operating points that strictly control FNR while keeping FPR manageable for staff.

The authors emphasize that while the Kentucky model demonstrates strong potential to reduce manual review workload without missing cases, the approach is not a "one-size-fits-all" solution. It requires local data curation and acknowledges that label noise and linguistic variations across facilities remain challenges. The study does not claim to solve all generalization issues but provides a validated framework for registries to adapt models to their specific operational contexts. Future work is proposed to explore label smoothing, robustness testing against facility phrasing and HL7 changes, and the potential (but currently constrained) use of Large Language Models (LLMs) given privacy and resource limitations.