Representation-Level Adversarial Regularization for Clinically Aligned Multitask Thyroid Ultrasound Assessment

1. Problem Statement

Thyroid ultrasound assessment involves two coupled clinical tasks: nodule segmentation (drawing contours for measurement) and TI-RADS risk stratification (categorizing malignancy risk based on sonographic criteria).

• The Challenge: In clinical practice, these tasks are performed jointly by radiologists, but deep learning models often treat them separately or sequentially.
• Annotator Variability: Ground truth labels (contours and risk scores) vary significantly between different radiologists. This creates "inconsistent supervision," where the "ground truth" reflects subjective interpretation rather than an objective label.
• Multitask Failure: Naïve multitask learning (sharing a single encoder for both tasks) often fails because the gradients for segmentation and classification compete within the shared representation, leading to negative transfer and unstable training.
• Domain Shift: Models trained on static B-mode images often fail to generalize to cine-loop (video) frames due to temporal redundancy and intra-case variability.

2. Methodology

The authors propose a Clinically Guided Multitask Framework enhanced by a novel regularization technique called RLAR.

A. Architecture Overview

The model uses a Shared Encoder (based on a pretrained nnU-Net) that feeds into two task heads:

1. Segmentation Decoder: Outputs a nodule mask.
2. Classification Head: Outputs TI-RADS risk logits.

B. Clinically Guided Embedding (Radiomics Distillation)

To ensure the risk prediction is grounded in clinical evidence rather than purely data-driven correlations:

• The classification embedding vector ($h$) is split into two parts: $h_{tirads}$ and $h_{deep}$.
• $h_{tirads}$ (13 dimensions): Trained to mimic a compact set of radiomics features (shape, margin, texture) computed from the ground-truth mask during training.
• $h_{deep}$: Captures complementary non-linear deep features for discriminative power.
• Loss: An $\ell_2$ regression loss ($L_{clin}$) forces $h_{tirads}$ to match the radiomics target.
• Note: Radiomics are only used during training as a distillation target; they are not computed at inference time, preserving deployment efficiency.

C. Representation-Level Adversarial Regularization (RLAR)

To address gradient competition in the shared representation without hard parameter disentanglement:

• Concept: Instead of "surgery" on gradients, RLAR treats task gradients as geometric vectors in the latent space.
• Mechanism:
  1. For each task $t$, compute the normalized adversarial direction ($\delta_t$) in the shared representation space (the direction that maximally increases the task loss).
  2. Calculate the cosine similarity between the adversarial directions of different task pairs (e.g., segmentation vs. classification).
  3. Penalty: The RLAR loss ($L_{RLAR}$) penalizes excessive angular alignment (high cosine similarity) between these directions.
• Goal: This encourages complementary sensitivities (tasks relying on different features) while preserving beneficial shared cues, effectively reducing destructive interference.

D. Total Objective Function

The model is optimized using a weighted sum of:
$$L_{total} = \lambda_{seg}L_{seg} + \lambda_{cls}L_{cls} + \lambda_{clin}L_{clin} + L_{RLAR}$$

3. Key Contributions

1. Clinically Aligned Multitask Framework: A unified model that mirrors the clinical workflow by jointly predicting segmentation and TI-RADS risk, using radiomics distillation to anchor risk predictions to interpretable medical criteria.
2. RLAR (Representation-Level Adversarial Regularization): A novel geometric regularizer that explicitly controls task interference in the latent space by penalizing alignment between task-specific adversarial directions, avoiding the brittleness of hard parameter partitioning.
3. Robustness to Variability: Demonstrates that managing gradient competition via RLAR stabilizes training under inconsistent annotator supervision and improves generalization to external datasets (cine-loop data).

4. Experimental Results

The model was evaluated on two public datasets: ThyroidXL (static B-mode, expert-labeled) and AIMI (cine-loop, biopsy-confirmed).

• Classification Performance (TI-RADS):

  • On ThyroidXL, the Clinically Guided model achieved the best F1 score (0.6267).
  • On AIMI (External), RLAR achieved the highest Recall (0.4033) among all multitask variants. High recall is critical clinically to avoid missing high-risk nodules.
  • Single-task models (EfficientNet, ConvNeXt) suffered significant performance drops on AIMI due to domain shift.

• Segmentation Performance:

  • RLAR maintained segmentation quality comparable to the best baselines (Dice ~0.856 on ThyroidXL) but with significantly lower variance across cross-validation folds, indicating more stable training.
  • On AIMI, RLAR achieved the best HD95 (Hausdorff Distance) among multitask models, suggesting better boundary robustness.

• Ablation Studies:

  • Placement: Applying RLAR at the bottleneck layer was most effective for in-domain data, while the mid-encoder layer worked best for the external cine-loop dataset.
  • Feature Importance: Removing individual radiomics features had minimal impact on in-domain performance but caused significant drops in recall on the external AIMI dataset, confirming that texture and shape cues are vital for cross-domain generalization.

5. Significance and Conclusion

• Clinical Relevance: The approach bridges the gap between deep learning and clinical workflow by ensuring risk predictions are grounded in interpretable radiomics features without adding inference-time computational burden.
• Technical Innovation: RLAR offers a flexible alternative to "gradient surgery" or hard parameter separation. It allows tasks to share information where beneficial while preventing them from fighting over the same representation features.
• Generalization: The method significantly improves robustness to domain shifts (static images to cine-loops) and annotator variability, making it a promising tool for real-world thyroid nodule assessment systems.

In summary, the paper presents a robust, clinically grounded multitask learning framework that leverages radiomics distillation and adversarial geometric regularization to solve the challenges of inconsistent supervision and domain shift in thyroid ultrasound analysis.