How errors from prompt-assisted longitudinal tumour segmentation change RECIST 1.1 target-lesion response categories: a retrospective out-of-fold analysis

Technical Summary: How Errors from Prompt-Assisted Longitudinal Tumour Segmentation Change RECIST 1.1 Target-Lesion Response Categories

Problem Statement
The Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 is the standard for assessing tumor response in oncology trials and routine care. It relies on clinicians measuring selected target lesions at baseline and follow-up to calculate the Sum of Longest Diameters (SLD) and assign a response category (Complete Response, Partial Response, Stable Disease, or Progressive Disease). While automated segmentation promises reproducibility and scalability, small errors in boundary delineation, lesion tracking, or detection can cross the strict thresholds defined by RECIST 1.1 (e.g., a 30% decrease for Partial Response), potentially altering clinical categorization.

Prior research has focused heavily on "missed lesions" (omission) as the primary safety risk. However, semi-automatic tools that rely on user prompts can still fail in other ways: tracking a lesion to the wrong location, retaining a lesion that has resolved (false persistence), or measuring a correctly located lesion inaccurately. This study addresses a gap in the literature by quantifying how often these specific error types change the target-lesion response category and whether they introduce a systematic bias toward apparent benefit or harm.

Methodology
The authors conducted a retrospective, out-of-fold analysis using the public autoPET/CT longitudinal whole-body CT dataset, comprising patients with metastatic melanoma.

• Model and Task: The study utilized LongiSeg, a prompt-assisted longitudinal segmentation model based on an nnU-Net residual-encoder backbone. The task was defined as "assisted tracking": the model received the baseline CT, the expert baseline lesion mask, and a location click for the follow-up scan. Crucially, the expert follow-up mask was withheld; the model had to generate the follow-up outline based on the prompt and the baseline context.

• Evaluation Protocol: To ensure an unbiased evaluation, the study employed a strict patient-level out-of-fold (OOF) approach. Each of the 258 response-evaluable patients was processed by a model checkpoint trained on the other four folds of the 300-patient development cohort, ensuring the patient's scans were never seen during weight fitting.

• Response Calculation: The study focused exclusively on the target-lesion component of RECIST 1.1. Targets were selected from expert baseline masks (up to five total, max two per organ/lesion type). The SLD was calculated for both the expert reference (expert baseline + expert follow-up) and the model (expert baseline + model follow-up).

• Error Taxonomy and Attribution: Disagreements between the model and reference response categories were decomposed into four error types:

  1. Prompted Omission: The lesion was present in the reference but missing in the model.
  2. Tracking Mislocalization: The model output did not overlap with the reference lesion.
  3. False Persistence: The model retained a lesion that had resolved in the reference.
  4. Measurement (Sizing) Error: The model and reference overlapped, but the measured diameters differed.

  To handle cases with multiple simultaneous errors, the authors used Shapley attribution from cooperative game theory. This method averaged the contribution of each error type across all possible correction orders to determine how much each error type drove the category disagreement.

Key Results

• Disagreement Rate: The model's target-lesion response category disagreed with the expert reference in 43 out of 258 patients (16.7%) (95% Wilson CI: 12.6–21.7).

• Error Contributions: Using Shapley attribution, the study found that no single error channel clearly dominated the disagreements:

  • Measurement (Sizing) Error: Contributed 8.3% (95% CI: 5.0–11.8).
  • Detection/Localization (Omission + Mislocalization): Contributed 5.2% (95% CI: 2.7–7.9).
  • False Persistence: Contributed 3.1% (95% CI: 1.2–5.4).

  While sizing had the largest point estimate, it was not statistically separated from the combined detection/localization channel (difference of +3.1 percentage points, 95% CI: −1.2 to 7.4).

• Direction of Bias: In a post-hoc analysis, 28 disagreements moved toward "apparent benefit" (e.g., Stable Disease becoming Partial Response) and 15 toward "apparent harm." However, this imbalance was not stable across different target-selection rules and did not constitute a confirmed bias toward inflating objective response rates.

• Simulation Validation: Synthetic simulations resampling observed sizing errors produced a mean "flip rate" of 9.3% (central 95% simulation interval 6.2–12.4), which encompassed the observed 8.3% sizing contribution. This suggests that the magnitude of boundary errors observed in the model is sufficient to plausibly explain the rate of category changes attributed to sizing.

Significance and Claims
The paper claims that measurement error is a critical, yet often overlooked, failure mode in automated RECIST assessment. The findings challenge the prevailing assumption that detection (omission) is the sole or primary safety concern. The study demonstrates that even when a model correctly locates a lesion, small boundary inaccuracies can be sufficient to shift a patient's response category.

The authors emphasize that these results are specific to:

1. Prompt-assisted tracking (not autonomous detection).
2. Follow-up errors conditional on a correct expert baseline.
3. Target-lesion components only (excluding non-target disease and new lesions).
4. One specific model architecture (LongiSeg) evaluated internally.

The study concludes that future evaluations of automated response systems must audit boundary accuracy and measurement precision with the same rigor as lesion coverage. The authors call for independent confirmation, particularly regarding the direction of disagreements, and note that their findings do not prove changes in clinical decision-making or patient outcomes.