🤖 AI

Vaporizer: Breaking Watermarking Schemes for Large Language Model Outputs

This paper demonstrates that state-of-the-art large language model watermarking schemes can be effectively broken by various semantic-preserving text modification attacks, revealing critical vulnerabilities in current systems and highlighting the need for more robust security designs.

Original authors: Jonathan Hong Jin Ng, Anh Tu Ngo, Anupam Chattopadhyay

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jonathan Hong Jin Ng, Anh Tu Ngo, Anupam Chattopadhyay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical stamp that leaves an invisible mark on every sentence a computer writes. This "watermark" is supposed to prove that a human didn't write the story, but a robot did. The creators of these stamps claim they are unbreakable, like a secret code that can't be erased without ruining the story itself.

This paper is like a group of security experts trying to break those stamps. They asked a simple question: "Can we wash off the invisible ink without scrubbing the paper until it tears?"

Here is what they found, explained through everyday analogies:

The Three Types of "Invisible Stamps"

The researchers tested three different ways companies are trying to watermark AI text:

The "Green Light" System (Provable Robust): Imagine a traffic light that subtly nudges the AI to pick "green" words slightly more often than "red" ones. It's a statistical trick. The paper claims this is the toughest to break, but even this one isn't unbreakable.
The "Secret Handshake" System (SynthID): This is like a game where the AI picks words based on a secret scoring rule. If the words fit the secret pattern, it's watermarked. The researchers found this is moderately vulnerable.
The "Digital Signature" System (Publicly Detectable): This tries to hide a cryptographic code directly into the text, like a hidden barcode. The researchers found this was completely broken. Every single attack they tried wiped this signature out instantly.

The "Wash" Methods (The Attacks)

The team didn't just use a sponge; they used three different "laundry" techniques to try and remove the marks:

The Thesaurus Swap (Lexical Changes): Like swapping "happy" for "joyful" or "big" for "huge." They found this was the weakest method. It barely touched the watermark and often made the text sound awkward or full of grammar errors.
The Translation Loop: Imagine translating a sentence from English to French, then back to English. It's like running a message through a game of "telephone." This removed some watermarks but often made the text sound clunky or changed the meaning slightly.
The "AI Rewriter" (Neural Paraphrasing): This was the nuclear option. They used a super-smart AI (like Pegasus) to rewrite the text. It's like hiring a professional editor to completely rephrase your story while keeping the exact same meaning.
- The Result: This was the most effective weapon. It successfully erased the "Green Light" and "Secret Handshake" stamps in many cases, and it wiped out the "Digital Signature" 100% of the time.

The Catch: The "Quality vs. Security" Trade-off

Here is the most important part of the story: You can't have it both ways.

The paper found a clear rule: To break the watermark, you usually have to damage the text.

If you tried to remove the stamp too aggressively, the text became full of grammar mistakes, hard to read, or lost its original meaning.
However, the "AI Rewriter" (Pegasus) was the master thief. It managed to remove the watermarks while keeping the text looking mostly normal. It was the best balance of "breaking the code" without "ruining the paper."

The Big Takeaway

The researchers concluded that the current "invisible stamps" are fragile.

The "Digital Signature" stamp is already useless; it breaks with the slightest touch.
The "Green Light" and "Secret Handshake" stamps are stronger, but a smart AI rewriter can still peel them off.

The Bottom Line:
The paper argues that we cannot rely on these current watermarks to prove if text is AI-generated. The "locks" are too easy to pick. To fix this, the authors suggest we need to stop hiding the mark in the words themselves (like the color of the ink) and start hiding it in the deep meaning of the story, which is much harder to change without destroying the story itself.

Until then, if someone tries to hide the fact that an AI wrote something by rewriting it, current technology can't reliably stop them.

Technical Summary: Vaporizer - Breaking Watermarking Schemes for Large Language Model Outputs

Problem Statement

The rapid adoption of Large Language Models (LLMs) has raised critical concerns regarding content authenticity, plagiarism, and model attribution. To address this, watermarking techniques have been proposed to embed invisible signals into generated text to verify authorship. However, existing schemes are claimed to be robust and production-grade, yet their vulnerability to semantic-preserving text manipulations remains under-explored. The core problem addressed in this paper is that current watermarking methods, whether statistical or cryptographic, may be susceptible to attacks that alter the text's surface structure while preserving its semantic integrity, effectively destroying the watermark signal without rendering the text unusable.

Methodology

The authors developed a systematic evaluation framework to test the robustness of three state-of-the-art LLM watermarking schemes against a comprehensive collection of modified text attacks. The study utilized a dataset of 100 diverse writing prompts generated by ChatGPT, which were processed through the target watermarking systems to create watermarked outputs. A control group of unwatermarked text was also generated for baseline comparison.

Target Watermarking Schemes

The study evaluated three distinct approaches representing current advancements:

Provable Robust Watermarking (Zhao et al., 2024): A statistical method using "green" and "red" token lists to bias token selection during generation.
Publicly Detectable Watermarking (Fairoze et al., 2025): A cryptographic approach encoding digital signatures within specific token sequences using hash functions.
SynthID (Dathathri et al., 2024): A hybrid strategy by Google DeepMind using tournament-based sampling and a secret-key seeded scoring function to select tokens.

Attack Strategies

The authors implemented three primary categories of attacks, balancing effectiveness with text quality preservation:

Lexical Transformations: Included synonym replacement (using WordNet), random word deletion, random word swapping, and adjacent word swapping.
Machine Translation: Bidirectional translation (English $\leftrightarrow$ French/German/Spanish) using the MarianMT framework.
Neural Paraphrasing: Utilized BART (facebook/bart-large-cnn) and Pegasus models to rephrase text while maintaining meaning.

Both single-step and incremental attacks were performed to analyze the relationship between modification intensity and watermark degradation.

Evaluation Metrics

The efficacy of the attacks was measured against two primary criteria:

Watermark Removal: Quantified by detection scores (Z-scores for statistical methods, g-scores for SynthID). A score below a specific threshold (e.g., 0 or 0.5) indicated successful removal.
Semantic and Quality Preservation: Assessed using:
- BERT Scores: To measure semantic similarity to the original text.
- Text Complexity: Measured via the Gunning Fog Index.
- Readability: Assessed using Flesch Reading Ease indices.
- Grammatical Errors: Detected and classified using LanguageTool.

Key Results

The experimental results revealed varying levels of vulnerability across the three schemes, with a consistent finding that watermarks can be removed with reasonable effort.

1. Provable Robust Watermarking

Vulnerability: Demonstrated the strongest resistance among the three.
Best Attack: Neural paraphrasing, specifically Pegasus Paraphrase, achieved the highest success rate (23% of samples fell below the detection threshold) with the lowest mean Z-score (0.93).
Other Attacks: Translation-based attacks showed moderate effectiveness (4-5% success), while lexical transformations like synonym replacement were least effective (1-4% success).
Quality Trade-off: Pegasus Paraphrase maintained reasonable grammatical error rates while achieving removal, whereas simple paraphrasing and synonym replacement caused severe quality degradation (e.g., 797% increase in grammatical errors for synonym replacement).

2. SynthID

Vulnerability: Showed moderate vulnerability.
Best Attack: Pegasus Paraphrase again proved most effective with a 14% success rate. Synonym replacement (11%) and word swaps (9%) followed.
Quality Trade-off: Neural paraphrasing offered the best balance, limiting BERT score reduction to 12.8% while achieving removal. Translation-based attacks were less effective (2-3% success) but preserved semantic content well.

3. Publicly Detectable Watermarking

Vulnerability: Demonstrated complete vulnerability.
Result: All tested attack methods achieved a 100% success rate in removing the watermark.
Quality Trade-off: While removal was universal, the impact on text quality varied. Word swapping preserved semantic content well (9.25% BERT reduction), whereas translation-based attacks caused severe distortion (e.g., German translation increased text complexity by 315%).

General Findings

Neural Paraphrasing Dominance: Across all schemes, neural paraphrasing techniques (particularly Pegasus) were the most effective at degrading watermark detectability while preserving linguistic quality.
Inverse Relationship: There is a clear trade-off between attack effectiveness and text quality. Successful removal invariably degraded semantic preservation, readability, or grammatical correctness.
Incremental Attacks: Progressive text modifications gradually reduced watermark detectability in a near-linear relationship with the degradation of semantic preservation.

Key Contributions

The paper makes the following contributions:

First Comprehensive Demonstration: It provides empirical evidence that leading LLM watermarking schemes are vulnerable to a wide range of semantic-preserving attacks.
Systematic Evaluation Framework: The authors developed a framework encompassing multiple attack vectors (lexical, translation, neural) and detection metrics (Z-scores, BERT, readability, errors).
Quantification of Trade-offs: The study quantifies the specific trade-off between watermark removal effectiveness and text quality degradation for different attack types.
Analysis of Incremental Modifications: It analyzes how incremental text changes impact both detection scores and semantic preservation.
Design Recommendations: The authors provide insights suggesting that future systems should focus on obstructing neural paraphrasing rather than simple lexical manipulations and consider hybrid detection approaches.

Significance and Claims

The authors state that their work sheds light on the strengths and weaknesses of existing LLM watermarking systems. The primary significance lies in the empirical proof of the fragility of current schemes, particularly the total failure of the Publicly Detectable Watermarking scheme and the vulnerability of even "provable" robust methods to sophisticated neural paraphrasing.

The paper claims that while watermarking aims to introduce imperceptible modifications, adaptive transformations exploit the inherent flexibility of language to erase these signals. The authors conclude that:

Current schemes are insufficient for reliable attribution in adversarial environments.
There is an unavoidable trade-off between removal efficiency and text fidelity, suggesting that hybrid detection systems incorporating quality metrics (e.g., flagging content with marginal watermark scores but degraded quality) could enhance robustness.
Future watermarking designs must move beyond surface-level text features and embed signals within deep semantic structures to withstand neural paraphrasing.
Among the tested schemes, Provable Robust Watermarking currently offers the most secure option, though it remains vulnerable to sophisticated attacks.

The study emphasizes that these findings are critical for reinforcing the reliability of watermarking methods in real-world applications and guiding the development of more secure content authentication technologies.