Radiomics-Integrated Deep Learning with Hierarchical Loss for Osteosarcoma Histology Classification

Technical Summary: Radiomics-Integrated Deep Learning with Hierarchical Loss for Osteosarcoma Histology Classification

Problem Statement
Osteosarcoma (OS) is an aggressive primary bone malignancy where accurate histopathological assessment of viable versus non-viable tumor regions following neoadjuvant chemotherapy is critical for prognosis and treatment planning. Manual evaluation of histopathological slides is labor-intensive, subjective, and prone to inter-observer variability. While recent deep learning approaches have attempted to automate necrosis quantification, prior studies often reported tile-level generalization that did not hold up under patient-level testing. This discrepancy arises from data leakage caused by random tile-level splits (where multiple tiles from the same patient appear in both training and test sets) and the failure of "flat" multi-class models to account for the clinical hierarchy of the disease. Specifically, standard three-class classification (non-tumor, non-viable tumor, viable tumor) often overlooks the distinct clinical significance of these subtypes and the varying class prevalences inherent to the disease.

Methodology
The authors propose a multimodal hierarchical learning framework designed to align model architecture with clinical reasoning. The methodology consists of three core components:

1. Multimodal Feature Fusion: The framework integrates deep learning image features with handcrafted radiomic descriptors.

   • An image encoder ($f_{img}$) processes RGB histology tiles, while a radiomic encoder ($f_{rad}$) processes a vector of 29 first-order and 2D shape features extracted via the PyRadiomics library.
   • These modalities are fused using an adaptive attention gating mechanism. A softmax gate computes attention weights ($\alpha_{img}, \alpha_{rad}$) to create a weighted combination of the image and radiomic embeddings, allowing the model to dynamically prioritize the most informative modality for a given sample.

2. Hierarchical Task Formulation: Instead of a flat three-class classification, the problem is decomposed into two related binary tasks:

   • Task A (Coarse): Distinguishing Non-tumor vs. Tumor (viable + non-viable).
   • Task B (Fine): Distinguishing Non-viable vs. Viable tumor (applied only to samples identified as tumor in Task A).
   • The final three-way probability distribution is derived by multiplying the probabilities of the two binary tasks, ensuring the output respects the pathological hierarchy.

3. Uncertainty-Weighted Hierarchical Loss: To optimize the two tasks simultaneously, the authors employ a homoscedastic uncertainty weighting framework (Kendall et al.).

   • Learnable scalar parameters ($\lambda_A, \lambda_B$) represent the log-variance (uncertainty) of each task.
   • The joint objective function combines the weighted cross-entropy losses of both tasks, where tasks with higher learned uncertainty receive lower effective weights. This allows the model to adaptively balance the learning signal between the coarse and fine classification tasks without manual hyperparameter tuning.

Key Contributions

• Patient-Level Generalization: The study addresses the issue of data leakage by employing a strict patient-level split for training and testing, providing a more realistic assessment of model generalization compared to previous tile-level random splits.
• Multimodal Integration: The work demonstrates that incorporating handcrafted radiomic features (specifically shape and intensity descriptors) as additional input significantly improves classification performance and adds interpretability, even when derived from the same images used for deep learning.
• Hierarchical Loss Optimization: The paper introduces a structured two-head loss function with trainable uncertainty weighting. This approach outperforms flat multi-class baselines by explicitly modeling the dependency between tumor presence and viability.

Experimental Results
The framework was evaluated on the TCIA Osteosarcoma Tumor Assessment dataset, comprising 1,144 H&E-stained tiles from 1995–2015, annotated into Non-tumor, Non-viable-tumor, and Viable-tumor categories.

• Performance: The proposed configuration (InceptionV3 backbone + Hierarchical Loss + Radiomics) achieved state-of-the-art performance on this dataset.
  • Overall Accuracy: 0.86 ± 0.01 (compared to 0.73 for the baseline 3-class InceptionV3).
  • Macro-F1: 0.86 ± 0.01.
  • OvR AUC: 0.96 ± 0.00.
• Ablation Studies:
  • Adding radiomic features to a standard InceptionV3 improved Macro-F1 from 0.74 to 0.80.
  • Adding hierarchical loss to InceptionV3 improved Macro-F1 from 0.73 to 0.75.
  • The combination of both yielded the highest gains.
• Per-Class Improvements: The full model significantly improved the recognition of the challenging "Non-viable tumor" class. Sensitivity at 90% specificity (Sen@Spe90) for non-viable tumors increased from 0.69 (baseline) to 0.83. The model also maintained high performance for Non-tumor (Sen@Spe90 0.92) and Viable-tumor (Sen@Spe90 0.95) classes.

Significance and Claims
The authors claim that their work sets a new state-of-the-art performance benchmark for osteosarcoma histology classification on the open TCIA dataset. They argue that the integration of radiomics provides intrinsic interpretability through explicit shape descriptors, complementing the implicit representations of deep networks. Furthermore, the hierarchical formulation supports clinically meaningful reasoning by aligning the model's decision process with the progression of tumor viability. The study concludes that combining hierarchical supervision with multimodal fusion is essential for enhancing discrimination across all clinically relevant tissue types, particularly for the difficult-to-classify non-viable tumor subtype. The code and trained models are made available to the community.