Technical Summary: BBR-Net: Boundary-Balanced Replay for Continual Medical Image Segmentation

1. Problem Statement

Continual learning (CL) for medical image segmentation faces significant challenges under domain shift, particularly the phenomenon of catastrophic forgetting. Existing replay-based methods typically prioritize preserving appearance-level information (pixel distributions or feature embeddings) or rely on random/heuristic sample selection. However, in medical imaging, image appearance varies substantially across devices and protocols, while the underlying anatomical structure remains relatively stable.

The authors argue that current CL approaches suffer from a fundamental mismatch: they prioritize variable appearance cues over invariant anatomical structures. Consequently, models may retain numerical performance metrics while failing to preserve anatomically consistent representations, leading to fragmented predictions and inconsistent boundaries in clinical applications. Furthermore, the effectiveness of replay mechanisms in medical CL has not been systematically analyzed regarding the structural reliability of stored samples or the order of tasks (e.g., learning from clean data first vs. noisy data first).

2. Methodology: BBR-Net

The paper proposes the Boundary-Balanced Replay Network (BBR-Net), a framework designed to integrate structural information into the memory selection process for continual cardiac ultrasound segmentation.

Core Architecture

The framework utilizes a dual-head U-Net backbone:

• Segmentation Head ($f_{seg}$): Predicts the segmentation mask.
• Boundary Head ($f_{bnd}$): Predicts anatomical boundary maps as an auxiliary constraint to reinforce shape consistency.

Boundary-Aware Replay Strategy

Unlike standard replay which treats all samples equally, BBR-Net constructs a replay buffer based on a combined priority score that quantifies structural complexity and class balance:

1. Boundary-Aware Priority ($S_b$): For a given mask $y$, a boundary map $B(y)$ is generated. The structural importance score is the density of boundary pixels:
   $$S_b(y) = \frac{1}{HW} \sum_{i,j} B(y)_{ij}$$
   Samples with higher boundary complexity (richer anatomical contours) are deemed more informative for retaining structural knowledge.

2. Class-Balanced Priority ($S_c$): To address class imbalance, an inverse-frequency weight is applied to the presence of each anatomical class. Rare structures are prioritized to ensure they are not forgotten.
   $$S_c(y) = \sum_{c=1}^{C} w_c \cdot c_c(y), \quad \text{where } w_c = \frac{1}{n_c}$$

3. Priority-Based Buffer Management: The final priority score $P(x, y)$ combines both factors:
   $$P(x, y) = \alpha S_c(y) + \beta S_b(y)$$
   The replay buffer ($\mathcal{M}$) stores samples based on this score. During target task training, samples are sampled from the buffer with probability proportional to their priority, ensuring that structurally informative and under-represented classes are replayed more frequently.

Training Objective

The model is trained using a multi-task loss function that combines segmentation loss (Cross-Entropy + Dice), boundary loss (Binary Cross-Entropy), and replay loss:
$$\mathcal{L} = \mathcal{L}_{seg} + \lambda_b \mathcal{L}_{bnd} + \lambda_r \mathcal{L}_{replay}$$
where $\lambda_b$ and $\lambda_r$ control the contributions of boundary supervision and replay, respectively.

3. Experimental Setup

The method was evaluated on two echocardiographic datasets under a sequential learning setting:

• Source Task: CAMUS (clean, well-defined annotations).
• Target Task: CardiacNet (noisy, high variability, diverse clinical conditions).

Two task orders were tested:

1. Forward: CAMUS $\to$ CardiacNet.
2. Reverse: CardiacNet $\to$ CAMUS.

Additionally, a Controlled Structural Perturbation Analysis was conducted where the structural reliability of source-task masks in the replay buffer was progressively degraded (by eroding/dilating boundaries) while keeping all other variables fixed, to isolate the effect of structural quality on forgetting.

4. Key Results

Forward Task Order (CAMUS $\to$ CardiacNet)

• Performance: BBR-Net achieved a Dice score of 0.893 on the source task (CAMUS) after learning the target, with a forgetting score of 0.009. This is nearly identical to the offline joint-training reference (0.902) and significantly outperforms standard fine-tuning (0.320) and plain replay (0.882).
• Ablation: The combination of boundary-aware and class-aware prioritization yielded the best balance between source-task retention and target-task adaptation. Boundary-only prioritization slightly outperformed class-only prioritization, suggesting structural cues are critical.
• Qualitative: BBR-Net maintained smooth, continuous anatomical boundaries and correct topological relationships, whereas baselines exhibited fragmented predictions and structural collapse in hard cases.

Reverse Task Order (CardiacNet $\to$ CAMUS)

• Performance: A striking asymmetry was observed. When the model learned from the noisy CardiacNet dataset first, BBR-Net failed, exhibiting catastrophic forgetting (Forgetting $\approx$ 0.595) comparable to fine-tuning.
• Comparison: In this reverse setting, "Weighted Sampler Replay" (which does not rely on structural priors) outperformed BBR-Net, retaining CardiacNet knowledge with a forgetting score of 0.015.
• Implication: Structure-aware replay is not universally superior; it fails when the initial structural priors learned from the first task are unreliable or noisy.

Controlled Structural Perturbation

• Findings: As the structural reliability of replay masks was degraded (corruption severity 0 $\to$ 5), source-task retention decreased monotonically (0.890 $\to$ 0.848), and forgetting increased (0.011 $\to$ 0.052).
• Conclusion: Replay effectiveness is strongly dependent on the quality of stored structural information, not just memory capacity or sample frequency.

5. Significance and Claims

The authors claim that this work shifts the paradigm of continual medical image segmentation from appearance preservation to structural preservation.

• Structural Consistency as a Driver: The study demonstrates that anatomical consistency, rather than appearance similarity, is the key factor governing knowledge retention in medical CL.
• Conditional Effectiveness of Replay: The paper challenges the assumption that replay is a universal solution. It reveals that the effectiveness of structure-aware replay is conditional: it succeeds when stable anatomical priors are learned early (forward order) but fails when initial representations are noisy (reverse order).
• Role of Structural Reliability: Through controlled perturbation, the authors provide empirical evidence that the degradation of replay performance is directly linked to the corruption of structural information in the memory buffer, independent of dataset difficulty or task order.
• Practical Implication: For robust deployment in heterogeneous clinical environments, replay mechanisms must account for the structural reliability of stored samples. If initial data is noisy, relying on boundary-based prioritization may amplify errors rather than correct them.

The paper concludes that future CL frameworks should explicitly model geometric and anatomical constraints rather than relying solely on feature-level alignment, and that understanding the interplay between task order and structural reliability is essential for developing robust medical AI systems.