🔬 physics

HERMES: A Hybrid Ensemble for Head-and-Neck Tumor Segmentation, TN Staging, and Recurrence-Free Survival on PET/CT

The paper introduces HERMES, a containerized hybrid ensemble system that integrates STU-Net-based tumor segmentation, geometry-derived radiological staging features, and a deep-clinical survival ensemble to simultaneously achieve state-of-the-art performance in head-and-neck tumor segmentation, TN staging, and recurrence-free survival prediction on the HECKTOR 2026 challenge.

Original authors: Kai Wang, Meixu Chen, Elie Nasr, Ryan Lanning, Moyed Miften

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kai Wang, Meixu Chen, Elie Nasr, Ryan Lanning, Moyed Miften

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where doctors have a super-powered X-ray vision that can see not just the shape of a tumor, but also how hungry it is for sugar. This is the world of PET/CT scans, a medical technology that combines a standard CT scan (which gives a detailed 3D map of the body's bones and organs) with a PET scan (which lights up areas where cells are gobbling up sugar, a sign of cancer). But having a super-powerful map is only half the battle; the real challenge is reading it. Doctors need to do three tricky things at once: draw a precise outline around the cancer, figure out how far it has spread to nearby lymph nodes (like checking if a fire has jumped to neighboring houses), and predict how likely the patient is to stay cancer-free in the future.

For a long time, computers have been trying to help with this, but they often act like a student who memorizes the textbook but fails the test because they can't connect the dots. They might be great at drawing the outline, but then they get confused when trying to guess the stage or the future outcome. This is where a new approach called HERMES comes in. It's a smart computer program designed to do all three jobs in one smooth motion, acting like a single, highly trained detective who doesn't just look at the clues, but understands exactly what the clues mean for the final verdict.

The Story of HERMES: A Digital Detective for Head and Neck Cancer

Meet HERMES, a clever new computer algorithm created by a team of researchers to help doctors fight head-and-neck cancer. Think of HERMES as a super-efficient factory line that takes a patient's medical data—specifically a special sugar-hungry scan (PET/CT) and a digital health record—and spits out three critical answers: "Where exactly is the tumor?", "How bad is it?", and "What does the future look like?"

The First Job: Drawing the Map
First, HERMES has to find the enemy. It looks at the scan and draws a digital outline around the main tumor and any swollen lymph nodes. To do this, it doesn't just use one brain; it uses a "committee" of ten slightly different AI models working together. Imagine ten different artists sketching the same monster on a piece of paper, and then HERMES takes the average of all their drawings to get the most accurate picture possible. This team is called an "ensemble," and it helps the computer avoid mistakes that a single model might make.

The Second Job: The "Geometry" Trick
Once the tumor is outlined, HERMES needs to figure out the "Stage" of the cancer. In the medical world, the stage depends on specific rules: How big is the tumor? How many lymph nodes are involved? Are they on one side or both?

Here is where HERMES gets really clever. Most computer programs try to guess the stage by looking at a long, complicated list of texture details (like how rough or smooth the tumor looks). But HERMES realized that for lymph nodes, the rules are actually very simple: it's all about count and size.

So, instead of using a complex list of textures, HERMES measures the shape of the tumor it just drew. It counts the number of separate lumps, measures the size of the biggest one, and calculates the total volume. It's like a teacher grading a test: instead of reading the student's entire essay to guess their grade, HERMES just counts the number of correct answers and the length of the longest sentence. The paper suggests that this "geometry" approach is much better at guessing the lymph node stage than the old, complicated texture methods. In fact, when the researchers tested this on perfect, hand-drawn maps, the geometry method was significantly more accurate. However, because the computer's own drawings aren't perfect, the real-world improvement is a strong trend rather than a guaranteed win.

The Third Job: Predicting the Future
Finally, HERMES tries to predict "Recurrence-Free Survival," which is a fancy way of asking: "Will the cancer come back?" To do this, HERMES acts like a panel of experts. It has a "Deep Learning" expert that looks at the scan images and a "Clinical" expert that looks at the patient's age, habits, and history.

Usually, these experts argue about how much weight to give each other. But HERMES decided to keep things simple and fair: it gives every expert an equal vote. One of these experts is special because it was trained with a unique "scorecard" that directly tracks how well it is predicting the future, rather than just trying to minimize a confusing math error. This makes the training process more honest and reliable, even if it doesn't magically make the predictions perfect.

The Results: A Work in Progress
When HERMES was tested on a group of about 50 patients (a small sample size), it did a solid job. It correctly outlined the tumors about 64% of the time (a score called Mean Dice), guessed the stage of the cancer with about 58% to 64% accuracy, and predicted survival outcomes with a score of 0.679.

The researchers are careful to say that while HERMES is a big step forward, it's not a magic wand. The biggest bottleneck is still the first step: if the computer's drawing of the tumor isn't perfect, the rest of the answers will be a little fuzzy. The paper suggests that if we can get the drawing part even better, the predictions for the cancer stage will get much sharper.

In the end, HERMES is a promising new tool that shows us a better way to organize medical AI. Instead of trying to be a genius at everything at once, it breaks the problem down, uses simple rules where they work best, and lets a team of experts vote on the answer. It's a reminder that sometimes, the smartest thing a computer can do is to stop overthinking and start measuring the obvious things—like size and count—just like a human doctor would.

Technical Summary of HERMES: A Hybrid Ensemble for Head-and-Neck Tumor Segmentation, TN Staging, and Recurrence-Free Survival

Problem Statement
The paper addresses the HECKTOR 2026 challenge, which requires a single, containerized algorithm to perform three distinct tasks on paired FDG-PET/CT scans and electronic health records (EHR) for head-and-neck (H&N) oropharyngeal cancer:

Segmentation: Delineation of the primary tumor (GTVp) and involved lymph nodes (GTVn).
Staging: Radiological T and N staging based on AJCC/UICC 7th-edition criteria.
Prognosis: Prediction of recurrence-free survival (RFS).

Unlike previous editions that allowed separate algorithms for each task, HECKTOR 2026 mandates a unified pipeline where segmentation outputs directly inform staging and survival estimation, reflecting a clinical workflow where delineation drives downstream decision-making.

Methodology
The proposed system, HERMES (Hybrid Ensemble for Radiotherapy-target segmentation, Malignancy staging, and Event-free Survival), employs a modular, multi-task architecture selected via rigorous out-of-fold (OOF) cross-validation to prevent overfitting on the small public validation set (~50 patients).

Segmentation:
- Architecture: A 10-fold ensemble of STU-Net Small networks (approx. 14M parameters), a scalable residual U-Net variant.
- Training: Trained from random initialization on 160×160×192 patches with deep supervision, Dice + Cross-Entropy loss, and extensive augmentation (including modality-specific resolution simulation and PET channel dropout).
- Ensemble: The final deployment uses 30 checkpoints (top 3 per fold) with test-time augmentation (flips, gamma).
- Post-processing: Connected-component filtering removes small components (GTVp < 1000 mm³, GTVn < 500 mm³), retains the top 2 GTVp and top 8 GTVn components, and applies an SUV > 2.5 gate to remove low-uptake nodal false positives.
TN Staging:
- Core Innovation: Instead of passing generic radiomics vectors to staging models, the system derives a compact set of geometry features directly from the predicted masks. These features align with the 7th-edition staging axes:
  - N-stage: Nodal count, largest component volume/extent, dominance fraction, and total nodal burden.
  - T-stage: Primary tumor extent and volume.
- Models:
  - N-stage: An L1-regularized logistic regression using clinical data and the 7-dimensional geometry block (replacing a 30-dimensional radiomics block).
  - T-stage: An equal-weight fusion of a deep T-head (3D ResNet-18) and a logistic regression over clinical, radiomics, and primary geometry features.
- Deep Expert: A DualHeadFusionResNet processes a 96³ patch centered on the mask centroid, fusing image features with an 18-dimensional clinical embedding.
Recurrence-Free Survival (RFS):
- Ensemble Strategy: An equal-weight, z-averaged ensemble of risk experts to avoid overfitting fusion weights.
- Members: Includes multi-scale deep survival models, single-task deep models, a classical clinical Cox model, and a novel concordance-tracking loss expert.
- Concordance-Loss Expert: Trained using a pairwise sigmoid loss that approximates $1 - \text{C-index}$ . Unlike the Cox partial likelihood, this loss value directly tracks the C-index during training, providing an interpretable signal for early stopping and checkpoint selection.

Key Contributions

Geometry-Aligned Features for Staging: The paper demonstrates that features explicitly mirroring the definition of radiological staging criteria (node count and size) outperform generic intensity/texture radiomics for N-staging. This approach reduces dimensionality (7 features vs. 30) and improves balanced accuracy.
Unified Multi-Task Pipeline: A single containerized system where segmentation masks drive downstream staging and survival tasks, validated on a strict OOF protocol.
Interpretable Survival Training: The adoption of a concordance-tracking loss provides a training signal that is directly aligned with the evaluation metric (C-index), offering interpretability over the standard Cox loss.
Regularization-Oriented Selection: A protocol prioritizing stability and generalizability (e.g., equal-weight ensembles, L1 regularization, clinical-only risk models for cross-center transport) over marginal point-estimate gains on small validation sets.

Results
On the HECKTOR 2026 held-out validation set (~50 patients), the deployed HERMES+ configuration achieved:

Segmentation: Mean Dice of 0.641 (GTVp: 0.714, GTVn: 0.688 on OOF).
Staging: Balanced Accuracy of 0.580 (T-stage) and 0.642 (N-stage).
Survival: C-index of 0.679.

Internal OOF Analysis (782 patients):

N-Stage: Replacing radiomics with geometry features increased balanced accuracy from 0.691 to 0.720. While the 95% CI for this gain includes zero on predicted masks, analysis using ground-truth masks showed a statistically significant improvement (0.837 vs. 0.897), confirming the gain stems from the feature geometry rather than mask artifacts.
Survival: The concordance-loss expert achieved a C-index of 0.713 compared to 0.693 for a Cox-trained twin on OOF data. However, across three training seeds, the difference was within noise (0.706 vs. 0.701), suggesting the primary value of this loss is its interpretability rather than a decisive accuracy boost.

Significance and Claims
The authors claim that the primary contribution is the demonstration that aligning downstream features with the specific decision boundaries of clinical tasks (specifically the size and number axes of N-staging) is more effective than using generic descriptors. The paper posits that for tasks defined by geometric criteria, features that instantiate those definitions are superior.

The authors are modest regarding statistical significance, noting that most design gains (including the survival loss improvement) fall within cross-validation variability. They emphasize that the N-stage improvement is a "well-supported trend" rather than a statistically significant breakthrough on predicted masks, though the effect is robust when segmentation quality is ideal. The work highlights that segmentation quality remains the shared bottleneck for all downstream tasks; improvements in GTVn Dice would directly propagate to better N-staging. The system qualified for the testing phase of the HECKTOR 2026 challenge.

The Story of HERMES: A Digital Detective for Head and Neck Cancer

Technical Summary of HERMES: A Hybrid Ensemble for Head-and-Neck Tumor Segmentation, TN Staging, and Recurrence-Free Survival

More like this