Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis

1. Problem Statement

Counterfactual image generation aims to simulate "what-if" scenarios (e.g., "How would this patient's disease progress if treated differently?") to support clinical decision-making and data augmentation. While existing methods like Deep Structural Causal Models (DSCMs) combined with Counterfactual Fine-Tuning (CFT) have advanced the field, they face two critical limitations in medical imaging:

• Global vs. Local Interventions: Previous CFT methods (e.g., Reg-CFT) rely on global scalar measurements (e.g., total plaque area) or subject-level factors (e.g., age). Consequently, interventions often result in global artifacts rather than localized structural changes. For instance, increasing "plaque area" might artificially brighten the entire vessel rather than adding plaque to a specific segment.
• Limitations of Pixel-Level Guidance: While using segmentation masks for guidance offers spatial control, it requires user-defined counterfactual masks for every intervention. This is tedious, impractical for large-scale datasets, and introduces ambiguity regarding the causal role of the masks.
• Spurious Correlations: Regression-based fine-tuning often fails to disentangle spatial dependencies because regressors tend to exploit spurious correlations (e.g., vessel brightness) rather than true anatomical cues, leading to "leakage" where an intervention in one region affects unrelated regions.

2. Methodology: Positional Segmentor-Guided CFT (Pos-Seg-CFT)

The authors propose Pos-Seg-CFT, an extension of the existing Seg-CFT framework. Instead of using global measurements or pixel-level masks, Pos-Seg-CFT introduces region-specific supervision derived from a pretrained segmentation model.

Core Mechanism

1. Regional Partitioning: The image is divided into $K$ disjoint spatial regions (e.g., proximal, mid, and distal segments of a coronary artery).
2. Segmentation-Derived Measurements: A pretrained, weight-frozen segmentation model ($s_\psi$) generates a segmentation map ($\hat{m}$) from the counterfactual image ($\hat{x}$).
3. Regional Aggregation: Instead of summing the total area globally, the method computes independent scalar measurements for each region $R_k$ by summing the pixel probabilities within that specific mask:
   $$\hat{c}_{pa^{(k)}_x} = \sum_{(i,j) \in R_k} \hat{m}_{i,j}$$
4. Fine-Tuning Objective: The generative model (HVAE) is fine-tuned to minimize the difference between the predicted regional measurements and the target counterfactual values ($f_{pa^{(k)}_x}$) for each region independently:
   $$\mathcal{L}_{\text{Pos-Seg-CFT}} = \sum_{k=1}^{K} \mathcal{L}(\hat{c}_{pa^{(k)}_x}, f_{pa^{(k)}_x})$$
   where $\mathcal{L}$ is typically an L1 or L2 loss.

Key Advantages of the Approach

• No Pixel-Level Inputs: It avoids the need for user-provided counterfactual masks, relying instead on the segmentation model's ability to detect structure changes.
• Spatial Independence: By supervising specific regions independently, the model learns to modify anatomy only where intended, preventing "leakage" to adjacent areas.
• Reusability: The same pretrained segmentor is reused for all regions by masking out irrelevant areas, making the approach flexible and scalable.

3. Experimental Setup

• Dataset: Internal Coronary Computed Tomography Angiography (CCTA) dataset containing 65,706 images of the Left Anterior Descending (LAD) artery.
• Preprocessing: Images were converted to straightened curvilinear planar reformations (sCPR) and cropped to $64 \times 384$ pixels.
• Variables: Three structure-specific variables were modeled:
  • Calcified Plaque Area (CPA)
  • Non-Calcified Plaque Area (NCPA)
  • Lumen Area (LA)
• Interventions: Each variable was intervened upon in three distinct positional regions: Proximal, Mid, and Distal.
• Baselines:
  • No-CFT: Standard DSCM without counterfactual fine-tuning.
  • Reg-CFT: Uses pre-trained ResNet-based regressors to predict global or regional scalar targets.

4. Key Results

The study evaluated effectiveness, defined as the alignment between the desired intervention and the actual change in the generated image.

• Quantitative Performance:

  • Pos-Seg-CFT consistently achieved the lowest error rates across all interventions and regions compared to No-CFT and Reg-CFT.
  • Localization: In Reg-CFT, intervening on the "Mid" region often caused significant unintended changes in the "Proximal" or "Distal" regions (positional leakage). Pos-Seg-CFT significantly reduced these off-target effects.
  • Example: For do(NCPA-mid), Reg-CFT had an error of 26.32 in the mid-region, while Pos-Seg-CFT reduced this to 23.44, with even more dramatic improvements in preventing leakage to other regions.

• Visual Evidence:

  • Visualizations showed that Reg-CFT interventions often triggered changes across the entire vessel (global artifacts).
  • Pos-Seg-CFT produced anatomically coherent modifications confined strictly to the targeted segment (e.g., adding plaque only to the mid-section), with difference maps confirming spatially aligned responses.

5. Key Contributions

1. Novel Framework: Introduction of Pos-Seg-CFT, the first method to enable spatially localized counterfactual generation without requiring user-defined counterfactual masks.
2. Decomposition of Variables: The strategy of decomposing global structural variables into regional measurements allows for fine-grained control over causal interventions.
3. Solving the Leakage Problem: Demonstrated that segmentation-derived regional supervision effectively disentangles spatial dependencies, preventing the "global artifact" problem common in regression-based fine-tuning.
4. Practicality: The method leverages a single, frozen segmentation model, making it computationally efficient and applicable to arbitrary region definitions without manual annotation.

6. Significance

This work addresses a critical gap in medical image synthesis: the ability to model localized disease progression (e.g., plaque growth in a specific artery segment) rather than global changes. By providing a mechanism for spatially targeted interventions, Pos-Seg-CFT enhances the realism and interpretability of counterfactual images. This has profound implications for:

• Clinical Decision Support: Simulating the specific impact of localized treatments.
• Data Augmentation: Generating realistic, region-specific variations for training robust diagnostic models.
• Bias Mitigation: Correcting localized anatomical biases in datasets.

The paper establishes a new standard for structure-aware causal modeling, moving beyond subject-level factors to precise, anatomically grounded image synthesis.