Beyond the False Trade-off: Adaptive EWC for Stealthy and Generalizable T2I Backdoors

Technical Summary: Beyond the False Trade-off: Adaptive EWC for Stealthy and Generalizable T2I Backdoors

Problem Statement
The paper addresses a critical vulnerability in Text-to-Image (T2I) diffusion models: the difficulty of implementing stealthy text-side backdoor attacks that preserve the model's clean behavior (fidelity) while maintaining a high Attack Success Rate (ASR). This challenge is particularly acute in practical deployment scenarios involving LoRA-adapted experts (models fine-tuned for specific styles like anime or photorealism).

The authors identify a "false trade-off" in existing approaches. Prior work often relies on output-space teacher-student alignment (Learning without Forgetting, LwF) or static Elastic Weight Consolidation (EWC) to preserve clean behavior. The paper demonstrates that static EWC, which applies a fixed penalty to parameter changes, fails in these scenarios. Specifically, a strong static consolidation weight ($\lambda$) intended to preserve clean behavior often suppresses the backdoor trigger entirely (collapsing ASR to 0), especially for weak, stealthy triggers (e.g., Unicode substitutions) or on LoRA experts. This occurs because the static regularizer cannot distinguish between the model "forgetting" clean behavior (which requires penalization) and the model "underfitting" the backdoor (which requires relaxation).

Methodology: Cosine-Aware Adaptive EWC (AEWC)
To resolve the tension between retaining clean capabilities and learning the backdoor, the authors propose Cosine-Aware Adaptive EWC (AEWC). This method transforms EWC from a static constraint into a context-sensitive, sensor-regulator system that operates solely during training with zero inference overhead.

The framework consists of three core components:

1. Teacher-Student Pipeline: A frozen clean encoder ($T$) acts as the teacher, while a trainable encoder ($S$) is the student. The student is trained to map poisoned prompts to a target embedding while maintaining alignment with $T$ on clean prompts.
2. Semantic Sensor (Clean Utility): Instead of using Mean Squared Error (MSE), AEWC employs a cosine similarity-based utility loss ($L_{utl-cos}$) on clean prompts. This measures the semantic drift between the student and the teacher.
3. Adaptive Regulator: The consolidation strength $\lambda$ is dynamically modulated based on the ratio of the clean utility loss to the backdoor loss ($L_{bd}$).
   • An Exponential Moving Average (EMA) stabilizes this ratio ($\hat{r}_t$).
   • If the ratio is high (indicating the model is forgetting clean behavior), $\lambda$ increases to strengthen consolidation.
   • If the ratio is low (indicating the backdoor is underfitting), $\lambda$ decreases to relax the constraint, allowing the trigger to be learned.
   • The final adaptive weight is clipped within a defined range $[\lambda_{min}, \lambda_{max}]$.

Key Contributions

1. Diagnosis of Static EWC Failure: The paper provides a systematic study showing that static EWC induces a catastrophic failure mode in LoRA-adapted experts and with weak triggers. It creates a false appearance of high fidelity by suppressing the attack entirely (ASR $\approx$ 0), rather than successfully balancing retention and learning.
2. AEWC Framework: The authors introduce a sensor-regulator design that couples a cosine semantic sensor with an adaptive regulator driven by the clean-to-backdoor loss ratio. Ablation studies confirm that both the cosine sensor and the adaptive regulator are necessary to achieve the optimal ASR-fidelity trade-off.
3. Empirical Evidence: The study demonstrates that AEWC achieves a superior Pareto frontier for ASR and fidelity compared to static EWC, LwF, and plain training. Crucially, it is the only baseline that maintains high ASR while preserving stylization on LoRA experts and generalizing to off-distribution (OOD) text.

Experimental Results
The authors evaluated their method across three trigger families (Syntactic, Unicode/Homoglyph, and Phrase) on Stable Diffusion v1.5, including LoRA-adapted "Anime" and "Photoreal" experts and OOD text from AG News.

• In-Domain Performance: Under matched training budgets, AEWC consistently achieved higher Clean-Cosine similarity (fidelity) at comparable or higher ASR than static baselines. For Unicode triggers, AEWC achieved an ASR of 98.7% with a Clean-Cos of 0.972, significantly outperforming static EWC which struggled to learn the trigger without degrading fidelity.
• LoRA Experts: On both Anime and Photoreal LoRA experts, AEWC maintained a perfect ASR of 1.00 while preserving stylization (low FID and LPIPS). In contrast, static EWC collapsed the attack (ASR = 0) when attempting to preserve stylization, and LwF occasionally leaked backdoor artifacts into clean outputs.
• Off-Distribution (OOD) Generalization: AEWC demonstrated strong zero-shot transfer to OOD text (AG News), maintaining high fidelity and ASR where static EWC failed to generalize effectively.
• Ablation Studies: Removing the adaptive regulator (using Static+Cos) or the cosine sensor degraded performance, confirming the necessity of the full sensor-regulator synergy.

Significance and Claims
The paper claims that AEWC resolves the "retain-learn" tension in T2I backdoors by making the regularization mechanism semantics-aware. By distinguishing between semantic forgetting and backdoor underfitting, AEWC avoids the "trigger suppression" failure mode inherent in static consolidation.

The authors position this work as a practical design principle for stealthy text-side backdoors, particularly in the context of LoRA adapters and weak triggers. They argue that parameter-space consolidation, when made adaptive and context-sensitive, offers a robust alternative to output-space distillation, enabling high-fidelity attacks that generalize beyond the training distribution without incurring inference-time computational costs. The work is framed as a contribution to understanding the feasibility and mechanics of such attacks, with the authors noting the dual-use nature of the research and releasing configuration snapshots for verification while gating sensitive artifacts.