RLSpoofer: A Lightweight Evaluator for LLM Watermark Spoofing Resilience

1. Problem Statement

Large Language Model (LLM) watermarking is a critical mechanism for detecting AI-generated text and attributing its source. However, the robustness of these watermarks against black-box spoofing attacks remains insufficiently evaluated. Existing evaluation methods suffer from three major limitations:

1. High Data Requirements: They often require massive training corpora (up to 10,000 samples).
2. White-Box Access: Many rely on access to the watermarking algorithm's internals or the detector API.
3. Semantic Degradation: Attacks often fail to generate text that is semantically coherent with the original context.

The core challenge is to develop a lightweight, black-box attack that can effectively spoof watermarks using minimal data (e.g., 100 pairs) while maintaining high semantic fidelity, thereby serving as a rigorous stress test for current watermarking schemes.

2. Methodology: RLSpoofer

The authors propose RLSpoofer, a reinforcement learning (RL) based black-box attack. The approach is grounded in a distributional perspective, aiming to shift the paraphraser's output distribution from a "human-like" distribution ($P_h$) toward a "watermarked" distribution ($P_{wm}$).

A. Theoretical Foundation: Local Capacity Bottleneck

The paper establishes a theoretical local capacity bottleneck (Theorem 1).

• Concept: Under a KL-bounded local update (to preserve semantic fidelity), there is a limit to how much probability mass can be reallocated from the dominant human-like token to other tokens.
• Metric: This limit is quantified by the local capacity mass ($c_t = 1 - \max_{x} P_h(x|ht)$).
• Insight: If the human-like distribution is highly concentrated (low $c_t$), there is little room to shift probability mass without breaking semantic coherence. Conversely, positions with high $c_t$ allow for flexible redistribution. This insight guides the reward function to focus on semantically flexible positions.

B. Algorithm Design

RLSpoofer utilizes Group Relative Policy Optimization (GRPO) to optimize a policy model ($\pi_\theta$) using only 100 human-watermarked paraphrase pairs.

1. Surrogate Distributions: Since the true watermarked and human distributions are unknown, RLSpoofer approximates them using a lightweight reference model ($\pi_{ref}$):
   • $P_h$ is approximated by conditioning $\pi_{ref}$ on the original human text.
   • $P_{wm}$ is approximated by conditioning $\pi_{ref}$ on the watermarked rewrite.
2. Reward Function: The training objective combines three components:
   • Capacity-Aware Token-Level Reward: $r_t = c_t \cdot \log \frac{P_{wm}}{P_h}$. This rewards tokens that are preferred by the watermarked distribution but penalizes shifts at positions where the human distribution is rigid (low $c_t$).
   • Sequence-Level Semantic Reward: Based on the P-SP score, ensuring the generated text remains semantically similar to both the original and the watermarked rewrite. A conservative "minimum" strategy is used to enforce strict semantic preservation.
   • Cross-Entropy (CE) Anchor: A teacher-forced loss term that anchors the policy to the target watermarked rewrite, preventing the RL optimization from drifting too far from the desired distribution.

3. Key Contributions

1. Local Capacity Bottleneck Theory: The paper provides a theoretical characterization of the trade-off between KL-bounded updates and semantic fidelity, defining a "local capacity mass" that dictates where watermark injection is feasible.
2. RLSpoofer Framework: A novel, sample-efficient black-box attack that requires zero access to watermark internals or detectors and only 100 training pairs.
3. Comprehensive Evaluation: The method is tested across 5 attacker models (ranging from 0.6B to 8B parameters) and 6 watermarking schemes (including logit-based and sampling-based distortion-free methods).

4. Experimental Results

The experiments demonstrate that RLSpoofer significantly outperforms existing baselines (Distill, DITTO, DPO) across various scenarios:

• Performance on Logit-Based Watermarks: RLSpoofer achieves high Spoof Success Rates (SSR), often exceeding 50-60% on schemes like EWD, SWEET, and KGW, while maintaining high semantic similarity (P-SP > 0.7).
• Performance on Sampling-Based (Distortion-Free) Watermarks: This is the most significant finding. Baselines (like Distill trained on 10k samples) struggle immensely with distortion-free watermarks like PF-Watermark (achieving ~6% SSR). In contrast, RLSpoofer achieves a 62.0% SSR on PF-Watermark using a 4B model with only 100 samples.
• Data Efficiency: RLSpoofer with 100 samples outperforms Distill trained on 10,000 samples.
• Ablation Studies:
  • Removing the local capacity weighting ($c_t$) significantly drops performance, proving the importance of targeting semantically flexible tokens.
  • Removing the CE anchor causes optimization instability and performance drops.
  • Using a conservative semantic reward (Min. of two similarities) yields better results than averaging or single-reference approaches.

5. Significance and Implications

• Vulnerability Exposure: The study reveals that current LLM watermarking paradigms, even advanced distortion-free ones, are highly vulnerable to distributional spoofing attacks when the attacker uses RL to learn the statistical shift rather than just mimicking text.
• Evaluation Standard: RLSpoofer provides a lightweight, practical stress test for the community. It suggests that relying on large-scale distillation for evaluation is insufficient; robustness must be tested against adaptive, sample-efficient RL attacks.
• Call for Robustness: The results indicate an urgent need for watermarking schemes that are not only statistically detectable but also resilient to distributional shifting attacks that preserve semantic meaning.

In conclusion, RLSpoofer fundamentally challenges the reliability of current LLM watermarks by demonstrating that a small, well-tuned RL agent can effectively "steal" watermarks without needing to know the secret key or the detector, highlighting a critical gap in current defense mechanisms.