In search of truth: Evaluating concordance of AI-based anatomy segmentation models

This paper introduces a practical framework for evaluating concordance among AI-based anatomy segmentation models on datasets lacking ground truth by harmonizing outputs into a standard representation and providing interactive visualization tools, demonstrating its utility in comparing six open-source models on NLST CT scans to flag discrepancies and prioritize cases of inter-model disagreement for expert review.

The Gist

Imagine you have a massive library of medical X-ray scans (specifically CT scans of chests) from thousands of people. Doctors want to study these scans to understand diseases, but looking at them one by one is impossible. They need a robot to automatically draw outlines around organs like lungs, hearts, and ribs so they can measure them.

Recently, six different "robot chefs" (AI models) were invented to do this job. They all try to draw these outlines automatically, but they don't always agree with each other. But here's the problem: No one has the "answer key." There are no perfect, human-drawn outlines to check against to see who is right.

This paper is like a taste test where the judges don't know the recipe, but they can still figure out which chefs are making the messiest soup by spotting where they disagree.

The Big Problem: The "Answer Key" is Missing

Usually, to test a student, you give them a test and compare their answers to the teacher's key. In this case, the "teacher" (the perfect human annotation) doesn't exist for these thousands of scans. If you just ask the six robots to draw the lungs, how do you know if they are drawing the same lung? One might call it "Left Lung," another "Lung Left," and a third might draw the outline slightly differently. It's like trying to compare six maps of the same city where everyone uses different names for the streets and draws the borders in different places.

The Solution: A Universal Translator and a "Group Hug"

The researchers built a toolkit to solve this in three clever steps:

1. The Universal Translator (Harmonization)
First, they built a translator. They took the messy, different labels from all six AI models and forced them to speak the same language. They made sure that when Model A says "Liver" and Model B says "Hepar," the computer knows they are talking about the exact same organ. They also gave every organ a standard color, so you don't have to guess which red blob is the heart and which is the liver.

2. The "Group Hug" (Consensus)
Since there is no teacher's key, they used the majority vote. Imagine six people trying to draw a circle on a piece of paper. If five of them draw almost the same circle, but one draws a square, you can guess the circle is probably right.

• They overlaid all six drawings on top of each other.
• Where all six agreed, that's the "Consensus" (the safe zone).
• Where they disagreed, that's a red flag.

Important Note: Just because the models agree (consensus) does NOT mean the answer is correct. They could all be making the same mistake! However, agreement is a useful signal that things are likely okay, while disagreement is a strong signal that something is wrong and needs attention.

3. The Detective Tools (Visualization)
They built two special tools to help humans spot the trouble:

• The "Outlier Radar" (Interactive Plots): Instead of looking at thousands of numbers, they made a chart. If a model's drawing is way off from the group hug, it pops up as a bright dot on the chart. You can click that dot, and it instantly flies you to the 3D scan to see the mistake.
• The "Side-by-Side" Viewer (3D Slicer): They created a split-screen viewer that lets you look at the same slice of a patient's chest from all six models simultaneously. It's like having six different security cameras showing the same room at the exact same time, so you can instantly see if one camera is looking at the wrong angle.

What Did They Find? (The Taste Test Results)

They tested these robots on 18 chest scans. Here is what they discovered:

• The Lungs (The Stars): All the models were great at drawing lungs. They all agreed almost perfectly. It's like all six chefs agreed on how to chop an apple.
• The Heart (The Confused One): Most models agreed, but one model (CADS) drew the heart as a tiny, compact ball, while the others drew it as a larger shape including the big blood vessels. It turned out CADS was using a different definition of what counts as "the heart."
• The Ribs and Spine (The Disaster Zone): This is where things got messy. Four of the models (which happened to be trained on the same dataset) kept making the same mistakes. They would accidentally glue two ribs together or merge two vertebrae (backbone bones) into one giant blob. It was like a chef who keeps accidentally gluing two spoons together.
  • Why? The training data they used had bad examples to begin with.
  • The Fix: The other two models (MOOSE and CADS) didn't use that bad training data, and they drew the bones correctly.

Why Does This Matter?

This paper isn't just about saying "Model X is bad." It's about finding where the models DISAGREE, so human experts know where to look first.

The researchers showed that even without a teacher to grade the work, you can still find where things might be going wrong by:

1. Making everyone speak the same language.
2. Checking where the group agrees and where they don't.
3. Using smart tools to flag the disagreements for human review.

They are now sharing all their "translator" tools, their "outlier radar," and their "side-by-side" viewer for free. This means other scientists can use these tools to evaluate how well different AI models agree on their own medical data and flag areas of disagreement for closer inspection.

In short: They built a toolkit to help us spot where AI models disagree, so that human experts can prioritize reviewing those cases — ensuring that when we automate medical research, we catch potential mistakes before they propagate.

Technical Summary

1. Problem Statement

The rapid proliferation of AI-based whole-body anatomy segmentation models has created a challenge for researchers and clinicians: how to evaluate concordance among multiple models on large-scale imaging datasets and identify cases where models disagree, prioritizing those for expert review.

• Context: Large public datasets like the National Lung Screening Trial (NLST) contain thousands of CT scans but lack segmentation labels.
• Challenge: Existing models (e.g., TotalSegmentator, MOOSE, CADS) vary in architecture, training data, and output formats. Direct comparison is difficult because:
  • Anatomical structures are labeled differently across models.
  • Boundaries are defined inconsistently.
  • There is a lack of standardized tools to visualize and compare multiple model outputs simultaneously without ground truth.
• Goal: To develop a practical framework for harmonizing, visualizing, and quantitatively comparing multiple segmentation models to identify and flag discrepancies among model outputs for prioritized expert review, enabling informed decisions about model suitability.

2. Methodology

The authors established a four-step framework to evaluate six open-source segmentation models on a sample of 18 CT scans (9 studies) from the NLST dataset.

A. Data Harmonization

• Format Standardization: All model outputs (originally in NIfTI format) were converted to DICOM SEG format. This ensures interoperability with standard medical viewers and archives.
• Semantic Mapping: The authors mapped model-specific labels to a unified SNOMED-CT coding system. This allowed for consistent terminology-based labeling of structures (e.g., mapping "lung_upper_lobe" from different models to the same SNOMED code).
• Color Assignment: A harmonized color map was generated to ensure consistent visual representation of structures across different models.

B. Quantitative Evaluation (Consensus-Based)

Since ground truth was absent, the authors used inter-model agreement as a proxy for quality:

• Consensus Segmentation: For each structure, a consensus mask was created containing only voxels segmented by all models.
• Metrics:
  • Dice Similarity Coefficient (DSC): Measured spatial overlap between each model's output and the consensus.
  • Volume Ratio: Compared the volume of individual model segmentations against the consensus volume.
• Filtering: Structures were excluded if they were segmented by only one model, had incomplete coverage in the chest CTs (e.g., abdominal organs), or were segmented by only 2–3 models (to ensure statistical reliability of the consensus).

C. Visualization and Qualitative Analysis

• Interactive Plots: Custom scatterplots were developed to visualize DSC and volume agreement. Points in the plots are hyperlinked to the specific CT scan and segmentation.
• Tools Developed:
  • OHIF Viewer: Used for browser-based, zero-footprint visualization of the harmonized DICOM SEG files.
  • CrossSegmentationExplorer: A new 3D Slicer extension allowing side-by-side, synchronized 2D/3D comparison of multiple segmentations for the same structure.
• Expert Review: Cases with low agreement (outliers in the plots) were flagged for detailed visual inspection by a radiology expert.

D. Models Evaluated

The study compared six models:

1. TotalSegmentator 1.5
2. TotalSegmentator 2.6
3. Auto3DSeg (trained on TotalSegmentator data)
4. MOOSE 3.0
5. MultiTalent (trained on TotalSegmentator data)
6. CADS (trained on NLST data)

3. Key Contributions

1. Harmonization Framework: A pipeline to convert diverse AI segmentation outputs into a standardized, interoperable DICOM SEG format with unified SNOMED-CT semantics.
2. Open-Source Toolkit:
   • CrossSegmentationExplorer: A 3D Slicer module for efficient, side-by-side comparison of multiple models.
   • Interactive Dashboards: Web-based plots for rapid outlier detection.
   • Public Data: The harmonized dataset, mapping scripts, and results are publicly available via the Imaging Data Commons (IDC).
3. Consensus-Based Evaluation Strategy: A methodology for evaluating inter-model agreement in the absence of ground truth, flagging disagreements for prioritized human review. Note: consensus among models does not guarantee correctness but indicates areas of agreement vs. areas requiring further scrutiny.

4. Key Results

The study analyzed 24 anatomical structures (lungs, heart, ribs, vertebrae, sternum) across the 6 models.

• Lungs: Showed excellent agreement across all models (DSC > 95%, volume overlap > 90%). Minor boundary differences were observed in the right middle lobe.
• Heart: Showed moderate agreement initially. The discrepancy was traced to CADS, which defined the heart as a compact structure excluding major vessels, whereas other models included atria and vessels. Excluding CADS improved agreement significantly (DSC > 95%).
• Ribs and Vertebrae: Showed poor agreement (DSC as low as 0–65% for some vertebrae).
  • Root Cause: Four models (TotalSegmentator 1.5/2.6, Auto3DSeg, MultiTalent) shared the same training dataset, which contained labeling errors. These models frequently:
    • Merged adjacent vertebrae.
    • Included parts of neighboring ribs.
    • Failed to segment the costovertebral joints correctly.
  • Outliers: MOOSE and CADS (which did not rely on the flawed TotalSegmentator training data) showed significantly better agreement with each other (DSC > 90% for vertebrae).
• Sternum: Showed moderate agreement, with differences primarily in the definition of the xiphoid process.

5. Significance and Conclusion

• Flagging Systematic Errors: The framework successfully flagged systematic disagreements in bony structures among models sharing the same training data, revealing a shared training-data artifact that would not have been visible without cross-model comparison.
• Scalability: The approach is scalable and dataset-agnostic. It allows for the evaluation of large cohorts without the prohibitive cost of manual expert annotation.
• Community Impact: By releasing the tools and harmonized data, the authors enable the broader community to:
  • Reproduce the evaluation workflow.
  • Evaluate concordance among models on their own datasets and identify areas of disagreement requiring closer inspection.
  • Provide feedback to model developers to improve training data and architectures.
• Future Work: The authors plan to extend this evaluation to the full NLST cohort and incorporate more advanced consensus methods (like STAPLE) to further refine concordance-based quality assessment for public release via the Imaging Data Commons.

In summary, this paper provides a critical "toolkit" for navigating the "truth" in AI segmentation when ground truth is missing, demonstrating that inter-model concordance analysis is a viable and effective strategy for quality control in medical imaging, enabling researchers to flag and prioritize cases of disagreement for expert review.