💻 computer science

An ablation measured twice: small component effects can change sign across repeated training runs in breast-cancer histopathology segmentation

This study demonstrates that in breast-cancer histopathology segmentation, the measured contributions of small architectural components in ablation studies can vary significantly in magnitude and even change direction across repeated training runs, highlighting the critical need for replication before drawing definitive conclusions about component efficacy.

Original authors: Md Mostafizur Rahman

Published 2026-09-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Mostafizur Rahman

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the field of medical imaging, computers are increasingly tasked with reading microscope slides to find cancer. These digital tools, known as deep learning models, are trained to recognize patterns in tissue samples that the human eye might miss. To build these tools, researchers combine various techniques, such as adjusting the colors of the images to account for differences between laboratories, or teaching the computer to look at the image in different levels of detail at once. When a new model is built, scientists create a summary table to show which of these techniques actually helped. This table, often called an ablation study, lists the specific contribution of each part, telling the reader, "this piece added value, that piece did nothing." The goal is to separate the useful tools from the noise so that future doctors and researchers know exactly what to trust.

However, a hidden problem lurks beneath these summaries. Even when a computer program is run twice with the exact same settings, the results are rarely identical. Tiny, invisible fluctuations in the computer's calculations can cause the final score to wiggle up or down. If this natural wiggle is large enough, it can make a helpful tool look useless, or a useless tool look helpful, simply by chance. This uncertainty makes it difficult to know if a reported improvement is a real discovery or just a lucky roll of the dice.

A researcher at San Francisco Bay University decided to test how much this natural wiggle matters in the specific task of identifying breast cancer in tissue samples. The study focused on a popular benchmark dataset containing thousands of image patches from patient slides. The researcher built a computer model designed to spot tumors and then tested seven different versions of it. Each version turned on or off three specific features: a method to standardize the colors of the images, a way to look at the image in multiple sizes simultaneously, and a mechanism that lets the computer compare fine details with broad overviews. The goal was to see which features truly improved the model's ability to find cancer.

To get a true sense of the computer's natural instability, the researcher first ran a single version of the model twelve times in a row, changing nothing but the random starting point of the calculation. The difference between these paired runs was measured, revealing that the model's score naturally shifted by a small but consistent amount every time it was restarted. This established a baseline for how much the numbers could move without any real change to the model itself.

With this baseline in hand, the researcher then ran the full experiment of seven different model versions. However, the subsequent repetition of the entire experiment was not originally planned as a design choice; it was necessitated because the per-patch prediction files from the first measurement were lost to a version-control mistake. To recover the data, the researcher repeated the grid, performing twenty-one fresh training runs on the same seven configurations using the same data and starting points. This created a second, independent set of results to compare against the first.

The comparison revealed a striking pattern. The large improvements seen in the first round mostly held up in the second round. For instance, the combination of color standardization and multi-size viewing consistently improved the model's performance, though the exact amount of improvement was slightly smaller the second time. However, the smaller, more subtle effects were completely unreliable. One specific feature, which acts like a bridge between fine details and broad views, showed a negative effect in the first round, suggesting it hurt the model's performance. In the second round, that same feature showed a positive effect, suggesting it helped. Another small effect flipped from positive to negative.

The study found that the size of the shift between the two rounds was roughly the same for every feature, regardless of how big the feature's effect was. This means that when a feature's effect is small, the natural wiggle of the computer is large enough to completely drown it out or even reverse its direction. The researcher concluded that for small effects, a single experiment is not enough to draw a conclusion. If a feature's impact is comparable to the natural noise of the system, its result is not stable enough to be trusted.

The only finding that survived this rigorous test was a specific behavior of the color standardization tool. While it did not change the overall average score of the model very much, it consistently helped the weakest hospitals in the dataset the most, bringing their performance up to match the stronger ones. This pattern was clear in both the first and second runs, proving that the tool was genuinely useful for making the system fairer across different locations, even if the overall numbers didn't look dramatic.

Ultimately, this work serves as a cautionary tale for how we interpret computer science results. It shows that when a computer model's improvement is small, it is impossible to tell if it is a real breakthrough or just a random fluctuation without running the experiment multiple times. The study argues that before declaring a specific part of a model to be good or bad, researchers must first measure how much the numbers wiggle on their own. If the effect is smaller than that wiggle, the result is not yet a fact, but merely a suggestion that needs further proof.

Technical Summary: An Ablation Measured Twice

Problem Statement
In deep learning for computational pathology, ablation studies are standard for quantifying the contribution of individual architectural components. However, these studies often lack context regarding the inherent run-to-run variability of the training pipeline. Without a measured scale of this noise, small performance differences (e.g., a 0.004 Dice improvement) appear indistinguishable from measurement noise, and null results cannot be reliably separated from unstable instruments. This paper investigates whether component-level contributions in breast-cancer histopathology segmentation are reproducible when the entire ablation grid is re-run, specifically examining whether small effects change sign or magnitude across repeated training runs.

Methodology
The study utilizes the BCSS (Breast Cancer Semantic Segmentation) benchmark, employing an institution-held-out protocol to prevent data leakage from shared staining protocols or scanners. The dataset consists of 5,273 training, 1,087 validation, and 2,256 test patches, with a specific focus on 1,563 tumor-bearing patches for paired statistical testing.

Pipeline and Architecture: The base architecture is a ResNet-34 encoder with a U-Net-style decoder. The ablation investigates three axes:
- Stain Processing: HED jitter (augmentation) vs. none.
- Multi-scale Encoding: Retaining the encoder hierarchy vs. using only the deepest map.
- Cross-scale Attention: A fine feature map querying a coarse pooled map (tested in windowed and full global variants).
  This results in a seven-configuration ablation (Cells A–G).
Experimental Design:
- Variability Measurement: A single configuration was trained twice at each of twelve seeds to establish a baseline run-to-run variability ( $R$ ).
- The Ablation: The seven configurations were trained at three seeds each.
- Replication: Crucially, the entire seven-configuration grid was re-run from scratch (21 fresh training runs) using the same frozen data splits and seeds to generate a second independent measurement of every contrast. This replication was necessitated by the loss of per-patch prediction files from the first run due to a version-control error.
Statistical Analysis:
- Contrasts were computed as mean paired per-patch Dice differences over tumor-bearing patches.
- Significance was assessed using a 10,000-resample percentile bootstrap and Wilcoxon signed-rank tests, Holm-corrected.
- Two resampling units were compared: patches (conventional) and slides (clustered bootstrap), with the latter used for inference to account for the nested nature of patches within slides.
- The study explicitly avoids using a fixed threshold (e.g., $R$ ) to classify results, instead analyzing the relationship between contrast magnitude and the displacement observed between the two measurements.

Key Results

Run-to-Run Variability: The mean absolute difference between two runs of the same configuration across twelve seed pairs was $R = 0.0081$ Dice. The individual differences ranged from 0.0001 to 0.0248.
Contrast Stability vs. Magnitude: The displacement between the two measurements of the ablation contrasts averaged 0.0046 Dice and varied by a factor of 4.6. In contrast, the magnitudes of the contrasts themselves varied by a factor of 27.
- Large Contrasts: The three largest contrasts (adding stain + multi-scale, all three components, and multi-scale alone) retained 59–87% of their magnitude and maintained their sign and statistical significance.
- Medium Contrast: The stain-only contrast retained its sign but only 13% of its magnitude.
- Small Contrasts: The three smallest contrasts (all involving cross-scale attention) were comparable in size to the observed displacement. Two of these changed sign upon re-running (e.g., cross-scale attention measured as $-0.0035$ in the first run and $+0.0022$ in the second).
Cross-Scale Attention: The initial measurement suggested a consistent negative effect for cross-scale attention. The replication showed two of the three paths became positive, with one becoming significantly positive under patch-level analysis (though not under slide-clustered analysis). The paper concludes that no definitive claim about cross-scale attention can be made on this benchmark.
Stain Processing: While the pooled mean improvement for stain processing collapsed from $+0.0075$ to $+0.0010$ (becoming statistically insignificant), a reproducible cross-site pattern emerged. Stain processing consistently raised performance at the three weakest held-out institutions while slightly lowering it at the strongest, effectively narrowing the inter-institution performance spread.

Claims and Significance
The paper makes the following specific claims:

Measurement of Noise: It provides an empirical measure of run-to-run variability ( $R \approx 0.0081$ Dice) for this specific pipeline, serving as an order-of-magnitude reference for how much a component contrast might shift upon re-running.
Rejection of Thresholds: The author argues against using a fixed threshold (like $R$ ) to determine reproducibility. Instead, they propose a "graded reading": the stability of a contrast depends on its magnitude relative to the displacement. Small contrasts (comparable to the displacement) are not directionally stable and warrant replication before interpretation.
Replication Necessity: The study demonstrates that a "clean" negative result (cross-scale attention is harmful) can be an artifact of a single run's noise, as the direction flipped upon replication. Conversely, a finding that appears weak in pooled means (stain processing) can be robust and practically significant when analyzed by sub-group (institution).
Methodological Shift: The paper advocates for reporting run-to-run variability alongside ablation tables and suggests that small component effects in deep learning pipelines require targeted replication rather than interpretation from a single grid.

The author explicitly states that their findings are limited to the BCSS binary segmentation task and do not claim to establish a universal law, but rather illustrate the relationship between contrast size and reproducibility in this specific context. They emphasize that the "finding" is the instability of small effects, not the specific performance of the tested architectures.

Technical Summary: An Ablation Measured Twice

More like this