Technical Summary: Factored Classifier-Free Guidance

Problem Statement

Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions (e.g., "What would this patient's disease progression look like if they received treatment A instead of B?"). While diffusion models combined with DDIM inversion and Classifier-Free Guidance (CFG) have become the state-of-the-art for this task, the authors identify a critical limitation in standard CFG: attribute amplification.

Standard CFG applies a single global guidance weight ($\omega$) to the entire conditioning embedding. In counterfactual generation, this uniform strengthening inadvertently amplifies not only the intervened attributes but also attributes that should remain invariant under the causal intervention. This "off-target" amplification violates the intended causal graph, modifies attributes outside the causal pathway, and causes the generation trajectory to drift from the original data manifold, thereby degrading identity preservation and the axiomatic soundness of the inferred counterfactuals.

Methodology: Factored Classifier-Free Guidance (FCFG)

To mitigate spurious attribute amplification without altering the underlying diffusion model, the authors propose Factored Classifier-Free Guidance (FCFG). This is a flexible, model-agnostic inference-time technique that enables attribute-wise control following a causal graph.

1. Attribute-Split Conditioning Embedding

The method introduces a structural change to the conditioning mechanism. Instead of entangling multiple attributes into a single vector, FCFG employs an attribute-split embedder. Each semantic parent attribute $pa_i$ is embedded independently via a dedicated Multi-Layer Perceptron (MLP), $E_i$. The final conditioning vector $c$ is formed by concatenating these independent embeddings:
$$c = \text{concat}(E_1(pa_1), E_2(pa_2), \dots, E_K(pa_K))$$
This architecture preserves the identity of each attribute in the embedding space, allowing for selective null-tokenization or modulation of individual attributes during inference.

2. Group-Wise Guidance Formulation

FCFG partitions the semantic attributes into $M$ disjoint groups (e.g., an "affected" group containing intervened attributes and their descendants, and an "invariant" group containing attributes expected to remain unchanged). Under the assumption that these groups are conditionally independent given the latent variable $x_t$, the authors derive a factorized proxy posterior:
$$p_\omega(x_t | pa) \propto p(x_t) \prod_{m=1}^M p(pa^{(m)} | x_t)^{\omega_m}$$
where $\omega_m$ is a distinct guidance weight for group $m$.

The guided noise prediction $\epsilon_{\text{FCFG}}$ is computed by summing the unconditional prediction with weighted differences for each group:
$$\epsilon_{\text{FCFG}}(x_t, t, c) = \epsilon_\theta(x_t, t, \emptyset) + \sum_{m=1}^M \omega_m \left( \epsilon_\theta(x_t, t, c^{(m)}) - \epsilon_\theta(x_t, t, \emptyset) \right)$$
Here, $c^{(m)}$ is a masked embedding where only the attributes belonging to group $m$ are retained, and others are replaced with null tokens.

3. Integration with Counterfactual Inference

FCFG is integrated into the standard three-step counterfactual inference pipeline (Abduction, Action, Prediction):

• Abduction: Uses DDIM inversion to estimate exogenous noise $u$ from the observed image and factual attributes.
• Action: Applies the causal intervention to generate counterfactual attributes.
• Prediction: Uses the FCFG-guided reverse DDIM trajectory. Crucially, the prediction step applies distinct weights ($\omega_{\text{aff}}$ for affected attributes, $\omega_{\text{inv}}$ for invariant attributes) to steer the generation toward the intervention while preserving invariant factors.

The framework is designed to be compatible with advanced guidance schemes, including CFG++ and Adaptive Projected Guidance (APG), by applying the same group-wise decomposition to their respective score formulations.

Key Contributions

1. Identification of CFG-Induced Failure: The authors analyze and demonstrate that the global guidance mechanism in standard CFG is a primary source of attribute amplification, causing spurious changes in invariant attributes even when the causal graph is known.
2. Proposal of FCFG: They introduce Factored Classifier-Free Guidance, a simple extension that assigns separate guidance weights to attribute groups. This allows for fine-grained, interpretable control over the generative process without requiring changes to the model architecture or retraining.
3. Empirical Validation: Through extensive experiments on natural (CelebA-HQ) and medical (EMBED, MIMIC-CXR) image datasets, the paper demonstrates that FCFG significantly reduces off-target amplification while maintaining or improving intervention effectiveness.

Experimental Results

The authors evaluate FCFG against standard CFG and other counterfactual generation methods (HVAE, HVAE-soft, SA-DCG) across three datasets:

• CelebA-HQ: In interventions targeting attributes like "Smiling," "Male," or "Young," global CFG with high guidance weights ($\omega \ge 2.0$) caused significant unintended changes in non-intervened attributes (e.g., changing age or identity when only smiling was targeted). FCFG, using a lower weight for invariant attributes ($\omega_{\text{inv}} \approx 1.0-1.2$) and a higher weight for affected attributes ($\omega_{\text{aff}} \ge 2.0$), achieved comparable target attribute changes while keeping invariant attributes stable. This resulted in lower Mean Absolute Error (MAE) and LPIPS scores for reversibility, indicating better identity preservation.
• EMBED (Breast Mammography): When intervening on breast density, global CFG inadvertently amplified the presence of circular skin markers. FCFG suppressed this spurious shift, maintaining stable performance on the non-intervened attribute.
• MIMIC-CXR (Chest X-Rays): Interventions on disease findings ("finding") with global CFG led to unintended changes in race and sex attributes. FCFG successfully localized the changes to the disease finding, reducing off-target AUC changes from large positive values (indicating amplification) to near-zero or negative values.

In all cases, FCFG improved the trade-off between intervention effectiveness and off-target amplification. Furthermore, FCFG demonstrated compatibility with CFG++ and APG, mitigating the attribute amplification inherent in those advanced methods as well.

Significance and Claims

The paper claims that FCFG addresses a fundamental limitation in guided diffusion models for causal reasoning. By decoupling the guidance signal across semantically or causally defined groups, FCFG enables the generation of counterfactuals that are axiomatically sounder (adhering to the causal graph) and more reversible (faithfully preserving identity).

The authors emphasize that FCFG is a simple, inference-time remedy that does not require:

• Additional predictor-based training objectives (unlike HVAE variants).
• Complex diffusion-autoencoder frameworks (unlike SA-DCG).
• Causal discovery (it assumes access to a pre-specified causal graph or semantic grouping).

The work suggests that the "global" nature of standard CFG is a bottleneck for reliable counterfactual inference and that factoring the guidance signal is a necessary step for precise, controlled image editing in causal contexts. The authors note limitations, specifically that FCFG relies on structured conditioning variables and pre-specified groupings, and that guidance weights are currently chosen empirically rather than adaptively.