Technical Summary: Rethinking Forgery Attacks on Semantic Watermarks in Black-Box Settings

1. Problem Statement

The proliferation of AI-generated content (AIGC) has necessitated robust mechanisms for source attribution and trust verification. Semantic watermarks, which embed information into the initial latent noise of diffusion models, have emerged as a promising solution due to their robustness against common image transformations. However, recent studies have demonstrated that these systems are vulnerable to black-box forgery attacks. In such attacks, an adversary with only black-box access to a service provider's (SP) model can use a proxy diffusion model to invert a watermarked image, recover an estimated latent representation, and regenerate a new image that preserves the SP's watermark while falsifying the content's provenance.

Existing literature primarily relies on empirical evidence to demonstrate these attacks, lacking a rigorous theoretical understanding of the conditions under which forgery succeeds or fails. Consequently, it remains unclear what fundamental security limits semantic watermarking systems possess in practice or how to design defenses that generalize beyond specific attack implementations.

2. Methodology and Theoretical Framework

Rate-Distortion Perspective

The authors reframe the black-box forgery problem through the lens of rate-distortion theory in the latent space. They model the adversary's task as a lossy compression problem where the proxy model acts as a decoder attempting to approximate a target watermarked latent.

• Irreducible Distortion Floor: The authors prove that a mismatch between the target model ($\Theta$) and the adversary's proxy model ($\Theta_A$) induces an irreducible information loss ($D_{irr}$). This loss creates a lower bound on the achievable distortion, meaning that even with infinite computational resources, a black-box adversary cannot achieve arbitrarily small inversion distortion if the models differ.
• Effective Rate Penalty: The model mismatch also imposes an effective rate penalty ($I_{pen}$), reducing the adversary's usable information rate and slowing the decay of distortion as the information rate increases.

Geometric Characterization of Distortion

Moving beyond stochastic noise assumptions, the paper characterizes the distortion introduced by proxy models as structured geometric deviations on the intrinsic latent manifold. These deviations manifest in two specific forms:

1. Global Drift (Directional): On the hypersphere of the latent space, forgery perturbations cause a systematic angular displacement. This is quantified using Spherical Angular Distortion (SAD), measured via cosine similarity. Theoretically, the orientation of a forged latent is expected to deviate significantly from the original watermarked latent, whereas a correctly recovered latent remains well-aligned.
2. Local Deformation (Structural): Forgery attacks disrupt the local structural integrity of the latent manifold. The authors map local token statistics to the manifold of Symmetric Positive Definite (SPD) matrices and measure the deviation using the Affine-Invariant Riemannian Metric (AIRM). This yields a metric called Local SPD Geometric Inconsistency (LGI), which captures non-congruent deformations in the local neighborhood of tokens that are absent in genuine watermarked samples.

Detection Scheme

Leveraging these geometric insights, the authors propose a scheme-agnostic detection method. This method operates as a pre-verification step, analyzing the recovered latent noise ($\hat{z}_T$) before attempting to extract the watermark. By evaluating the joint geometric space of global drift (G-Cos) and local deformation (L-SPD), the detector can distinguish forged samples from genuine watermarked ones without modifying existing watermarking schemes.

3. Key Contributions

• Theoretical Formalization: The paper is the first to formalize black-box forgery attacks within a rate-distortion framework, identifying an irreducible distortion floor that fundamentally constrains an adversary's capability when using a mismatched proxy model.
• Geometric Characterization: It characterizes forgery-induced distortion not as random noise, but as structured geometric deviations (global drift and local deformation) on the latent manifold.
• Robust Detection: The authors propose a detection method based on these geometric findings. Extensive evaluations demonstrate its effectiveness across diverse black-box scenarios, including both guidance-based and optimization-based attacks, while maintaining robustness against common image distortions.

4. Experimental Results

The authors evaluated their method against six state-of-the-art semantic watermarking schemes (Tree-Ring, RingID, HSTR, HSQR, Gaussian Shading, and TAG) across five target models (SD2.1, SDXL, PixArt-$\Sigma$, FLUX.1, SD3) using SD2.1 and SD3 as proxies.

• Forgery Success: Experiments confirmed that forgery attacks are highly successful when the target and proxy models are identical (e.g., SD2.1 $\to$ SD2.1), achieving near-perfect True Positive Rates (TPR) for watermark retention. However, attack efficacy drops significantly in cross-model scenarios (e.g., SD3 target with SD2.1 proxy), consistent with the theoretical distortion floor.
• Detection Performance: The proposed geometric metrics (G-Cos and L-SPD) achieved near-perfect Area Under the Curve (AUC) values (often exceeding 0.99) in cross-model scenarios. Even in the challenging identical-model setting (where the theoretical distortion floor is minimized), the method retained strong detection performance (e.g., AUC > 0.95 for G-Cos).
• Robustness: The detection method maintained high consistency under various post-processing distortions (e.g., JPEG compression, noise, cropping), although performance degraded slightly under stochastic noise and nonlinear signal degradations that corrupt latent structural coherence.
• Comparison: The method outperformed or matched alternative metrics and remained effective even against advanced schemes like PRC watermarks, which rely on pseudo-randomness to resist forgery.

5. Significance and Claims

The paper claims to offer a new perspective on the security of semantic watermarking systems. By identifying that the proxy-target model mismatch is the fundamental source of vulnerability for adversaries (and the source of detectable signal for defenders), the authors provide a principled basis for forgery detection.

• Orthogonal Defense: The proposed method serves as an orthogonal defense that complements existing watermarking schemes. It does not require retraining the generative model or altering the watermarking algorithm.
• Practical Utility: The detection framework operates exclusively on the latent noise, adhering to the zero-storage principles inherent in watermarking, making it lightweight and compatible with large-scale deployments.
• Security Boundaries: The work clarifies that while identical-model settings represent a security boundary where architectural mismatch is minimized, unavoidable internal accumulated errors (due to forward-generation vs. backward-inversion asymmetry) still induce measurable latent discrepancies, allowing for robust detection even in these extreme cases.

The authors conclude that their findings highlight the intrinsic limitations of black-box forgery attacks and suggest that future robust semantic watermarking designs should account for these geometric properties to enhance security.