Technical Summary: FedOT - Ownership Verification and Leakage Tracing via Watermarks for Federated LDMs

1. Problem Statement

The paper addresses the critical vulnerability of Federated Latent Diffusion Models (FedLDMs) regarding intellectual property (IP) protection. While Federated Learning (FL) preserves data privacy by keeping raw data local, it requires sharing global model updates, exposing the model architecture to malicious clients. These clients may unauthorizedly distribute or resell the fine-tuned models.

Existing solutions face two fundamental limitations when applied to FedLDMs:

1. Lack of Traceability: Current methods can verify if a model belongs to a federated group (ownership verification) but cannot identify the specific malicious client responsible for the leak.
2. Vulnerability to Replacement Attacks: Standard VAE-based watermarking techniques are fragile in FL. Since clients typically only update the U-Net parameters while the Variational Autoencoder (VAE) remains frozen, a malicious client can simply replace the watermarked VAE decoder with a clean, open-source counterpart. This removes the watermark without degrading the image generation quality, rendering the protection ineffective.

2. Methodology: FedOT Framework

The authors propose FedOT, the first framework designed to simultaneously enable ownership verification and leakage tracing for FedLDMs. The framework consists of two core components: a Chunked Watermark mechanism and Latent Vector Transformation (LVT).

2.1 Chunked Watermark Design

To solve the traceability gap, FedOT employs a binary watermark of length $n$ bits, divided into two distinct segments embedded into the VAE decoder of each client's model:

• Ownership Segment ($r$ bits): A shared prefix used to verify if a leaked model originates from the federated group.
• Tracing Segment ($n-r$ bits): A unique suffix assigned to each specific client.

Workflow:

1. Verification: If the extracted watermark's first $r$ bits match the group's prefix (above a threshold $\tau$), ownership is confirmed.
2. Tracing: Only upon successful ownership verification does the system analyze the remaining $n-r$ bits to identify the specific client index $j$ responsible for the leak. This two-stage process improves efficiency in large-scale deployments.

2.2 Latent Vector Transformation (LVT)

To counter the VAE replacement attack, FedOT introduces LVT, which creates a strong dependency between the VAE and the U-Net. The core insight is that the U-Net in LDMs adapts to the specific latent distribution produced by the VAE encoder during training. By modifying this distribution, the U-Net becomes "bound" to the specific VAE.

LVT Training Process (Two Stages):

1. Stage I (Encoder Training): The server fine-tunes the VAE encoder to learn a specific transformation $T$ applied to the latent vector $z$. The decoder is fixed. The encoder learns to map input images to a transformed latent space $z' = T(z)$.
2. Stage II (Decoder Training): The encoder is frozen. The decoder is fine-tuned to reconstruct images from the transformed latent space $z^*$, effectively learning the inverse transformation $T^{-1}$.

Transformation Strategies:
The paper explores three deterministic strategies to shift the latent distribution:

• Translation ($FedOT_{tran}$): Adds a constant vector $c$ to the latent space ($z' = z + c$).
• Mirror ($FedOT_{mir}$): Inverts the latent vector ($z' = -z$).
• Negative ($FedOT_{neg}$): Applies pixel-wise inversion to the input image ($x^- = 1-x$) before encoding, forcing the latent space to adapt to this inverted manifold.

Security Mechanism:
If a malicious client replaces the watermarked VAE with a clean one, the U-Net (trained on the transformed distribution) will receive inputs from a mismatched latent space. This causes severe degradation in image synthesis quality, making the stolen model unusable and deterring the replacement attack.

3. Key Contributions

The paper claims the following contributions:

1. First Framework for FedLDMs: FedOT is the first system to enable both ownership verification and specific client leakage tracing for Federated Latent Diffusion Models.
2. Robust Component Binding: The proposed LVT establishes a strong dependency between the VAE and U-Net. It ensures that removing the watermark via VAE replacement results in significant image quality degradation, effectively neutralizing zero-cost replacement attacks.
3. Comprehensive Evaluation: The authors provide extensive experiments against various attacks, including VAE replacement, model purification (fine-tuning on clean data), and image-based attacks, demonstrating the framework's reliability.

4. Experimental Results

Experiments were conducted using Stable Diffusion v2.1 with datasets COCO2017 (for LVT/watermark training) and LAION-10K (for federated fine-tuning).

• Defense Against Replacement Attacks:
  • Baseline methods (Stable Signature* and FedOT without LVT) showed almost no degradation in image quality (FID) after VAE replacement, and watermarks were undetectable.
  • FedOT with LVT: All LVT variants caused significant FID increases (quality degradation) upon replacement.
    • $FedOT_{neg}$ showed the best balance: FID increased by ~20 points (from ~20.3 to ~40.5), while maintaining high generation quality in the clean state.
    • $FedOT_{mir}$ caused the most severe semantic disruption (CLIP-Score drop of -0.064), rendering generated content practically unusable.
• Ownership and Tracing Performance:
  • FedOT achieved high detection rates (>0.96) and bit accuracy (>0.94) for both ownership verification and client tracing.
  • The chunked watermark design allowed efficient tracing only when necessary.
• Robustness:
  • Purification Attacks: Fine-tuning the model on clean data for 300 epochs failed to remove the watermark without degrading image quality (FID increased significantly).
  • Federated Conditions: Performance remained stable across different numbers of clients (5, 10, 20) and non-i.i.d. data distributions (Dirichlet $\alpha$).

5. Significance and Claims

The paper positions FedOT as a necessary evolution in protecting generative AI within privacy-preserving federated settings. The authors claim that:

• Existing FL watermarking solutions are insufficient for complex generative models like LDMs due to architectural differences and the specific vulnerability of frozen VAEs.
• The trade-off introduced by LVT is a deliberate and effective security feature: the cost of removing the watermark is the destruction of the model's utility. This makes the attack impractical for malicious actors.
• The framework fills a critical gap in the literature by moving beyond simple ownership verification to source tracing, allowing federated groups to identify and hold accountable the specific client responsible for model leakage.

The authors conclude that FedOT provides a reliable mechanism for securing FedLDMs, ensuring that intellectual property rights are maintained even in decentralized training environments.