iSight: Towards expert-AI co-assessment for improved immunohistochemistry staining interpretation

Technical Summary: iSight and HPA10M for Expert–AI Co-assessment in Immunohistochemistry

Problem Statement
Immunohistochemistry (IHC) is a cornerstone of diagnostic pathology and biomedical research, providing spatially resolved molecular information essential for tumor characterization, disease classification, and therapeutic stratification. Despite its critical role, IHC interpretation remains labor-intensive, subjective, and inconsistently reproducible across institutions. While Artificial Intelligence (AI) models trained on Hematoxylin and Eosin (H&E)-stained slides have shown promise, their applicability to IHC is limited by domain-specific challenges:

1. Stain Variability: IHC exhibits substantial variations in color profiles due to different chromogens, counterstains, and protocols.
2. Complex Interpretation: Unlike H&E, where morphology is primary, IHC requires simultaneous assessment of morphological context and specific stain localization patterns.
3. Metadata Integration: IHC datasets contain rich structured metadata (e.g., expected protein localization, tissue type) often absent in H&E datasets, which, if integrated, could significantly enhance model performance.
4. Data Scarcity: There is a lack of large-scale, structured, and harmonized IHC datasets suitable for training foundation models.

Methodology
The authors address these gaps through two primary contributions: the curation of a massive dataset and the development of a specialized multi-task learning framework.

1. The HPA10M Dataset
The authors constructed HPA10M, a large-scale dataset comprising 10,495,672 IHC images sourced from the Human Protein Atlas (HPA) version 23.0.

• Curation: The dataset was transformed from unstructured web resources into a machine-learning-ready format. This involved custom web scraping, harmonization of 20 years of accumulated XML metadata, and rigorous quality control by board-certified pathologists.
• Annotations: Images are annotated with standardized ontologies (Uberon for anatomy, SNOMED CT for diagnoses, UniProt/Ensembl for targets). Metadata includes patient demographics, tissue source, and staining characteristics.
• Scope: The dataset covers 45 normal tissue types and 20 major cancer types, encompassing over 17,200 protein biomarkers.
• Structure: Images are paired with unique MD5 hashes for traceability, and the dataset is split into a training set (~10.5M images) and a held-out test set (2,000 images).

2. The iSight Model Architecture
iSight (Immunohistochemical Staining Insight with Hybrid Transformers) is a multi-task learning framework designed to integrate visual features with structured metadata.

• Tissue Encoding Module: Whole-slide images are divided into 336×336 pixel patches. A Vision Transformer (CLIP-ViT-large-patch14-336) extracts token embeddings. Instead of aggregating within patches, the model concatenates all tokens and applies a gated attention mechanism to create a weighted tissue-level representation, allowing the model to focus on diagnostically relevant areas while maintaining global context.
• Context Encoding Module: Metadata (tissue type, SNOMED codes, antibody/gene information) is processed via a CLIP text encoder. During training, text captions are concatenated with metadata to create a multimodal input; during inference, only cell type information is used to prevent data leakage.
• Multi-Task Learning: The model utilizes parallel classification heads to simultaneously predict:
  • Primary Tasks: Staining Intensity (4 classes), Staining Location (4 classes), and Staining Quantity (4 classes).
  • Auxiliary Tasks: Tissue Type (58 classes) and Malignancy Status (2 classes).
• Training: The model was trained end-to-end using the Adam optimizer for one epoch on the HPA10M training set.

Key Results
1. Performance on Held-Out Data
On the 2,000-image held-out HPA test set, iSight outperformed fine-tuned pathology foundation models (PLIP and CONCH) across all primary tasks:

• Staining Location: 85.5% accuracy (vs. 79.4% for PLIP and 73.2% for CONCH).
• Staining Intensity: 76.6% accuracy (vs. 73.0% for PLIP and 70.0% for CONCH).
• Staining Quantity: 75.7% accuracy (vs. 73.2% for PLIP and 70.1% for CONCH).
• Auxiliary Tasks: iSight achieved 95.7% accuracy for tissue type and 99.9% for malignancy status.
• Ordinal Performance: For intensity and quantity, iSight achieved high "within-one-rank" accuracy (96.6% and 94.3%, respectively) and low "beyond-one-rank" error rates, indicating that errors were typically minor deviations rather than severe misclassifications.
• Calibration: The model demonstrated well-calibrated predictions with low Expected Calibration Errors (ECE) ranging from 0.0150 to 0.0408.
• Robustness: iSight maintained stable performance under synthetic image corruptions (random cutout and salt-and-pepper noise), showing minimal degradation in accuracy.

2. Pathologist User Study
A user study involving eight pathologists (ranging from residents to attendings) evaluated 200 images from the HPA and Stanford TMAD datasets.

• Accuracy Improvement: On the HPA dataset (with ground truth), pathologist accuracy improved after receiving AI suggestions: Location (68% → 70%), Intensity (57% → 60%), and Quantity (52% → 56%). iSight alone outperformed both initial and AI-assisted pathologist scores.
• Inter-Rater Agreement: AI assistance improved consensus among pathologists. Cohen's κ increased from 0.63 to 0.70 on the HPA dataset and from 0.74 to 0.76 on the Stanford TMAD dataset.
• AI Influence: When pathologists initially disagreed with the AI, they adopted the AI suggestion in 34.3% of location disagreements, 28.0% of intensity disagreements, and 21.3% of quantity disagreements on the HPA dataset.

Significance and Claims
The paper claims that this work establishes a foundation for AI systems capable of improving IHC diagnostic accuracy and consistency. Key contributions include:

• HPA10M: The first large-scale, systematically curated IHC dataset, enabling foundation-scale learning of IHC-specific patterns, which was previously hindered by the lack of structured data.
• iSight: A demonstration that task-specific architectural innovations (multi-task learning with metadata integration) can outperform generic pathology foundation models adapted from H&E analysis.
• Human-AI Collaboration: The study provides empirical evidence that AI can serve as a decision support tool that improves diagnostic consistency and inter-observer agreement without replacing expert judgment. The authors note that while AI assistance improved pathologist performance, it did not fully close the gap to the AI model's standalone performance, suggesting that cognitive factors (e.g., anchoring to initial judgments) influence the adoption of AI guidance.

The authors position iSight as a step toward clinically deployable systems that augment pathologist expertise, with a modular design allowing for adaptation to new biomarkers and workflows. They acknowledge limitations, including the need for multi-institutional validation, the absence of paired H&E images in the current dataset, and the fact that reference labels from HPA may not represent an absolute gold standard.