📊 statistics

Improving Detection of Watermarked Language Models

This paper proposes and evaluates hybrid detection schemes that combine watermark-based and non-watermark-based detectors to improve the identification of watermarked language model generations, particularly in scenarios where limited entropy hinders standalone watermark detection.

Original authors: Dara Bahri, John Wieting

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dara Bahri, John Wieting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to spot which books in a massive library were written by a specific, very popular AI robot versus those written by humans or other robots. This paper is about a new, smarter way to do that spotting.

Here is the breakdown of their idea using simple analogies:

The Two Old Ways of Spotting the AI

The paper says there are currently two main ways to catch an AI, but both have flaws:

The "Invisible Ink" Method (Watermarking):
Imagine the AI robot has a secret stamp. Every time it writes a word, it subtly changes the ink color just a tiny bit, invisible to the naked eye, but detectable if you have the special UV light (the secret key).
- The Problem: This only works well if the robot has a lot of freedom to choose words. If the robot is asked a simple fact like "What is the capital of France?", it has no choice but to write "Paris." There's no room to hide the invisible ink. But if asked to "Write a poem about a sad cloud," it has many choices, and the ink is easy to find.
- The Issue: In the real world, many prompts are simple or rigid, so the "invisible ink" often fails to show up.
The "Style Detective" Method (Non-Watermark Detection):
This is like hiring a literary critic who reads the text and says, "This sounds too perfect, too robotic, or too predictable." They look for statistical patterns that humans usually don't produce.
- The Problem: As AI gets smarter, it learns to sound more human. The critic gets confused and starts making mistakes, thinking human writing is AI, or vice versa.

The New Idea: The "Hybrid Team"

The authors realized that relying on just one detective is risky. Instead, they built a team that uses both methods together.

Think of it like a security checkpoint at an airport:

The Watermark Detector is the metal detector. It's fast and cheap to run. If it beeps, you know you have a problem.
The Non-Watermark Detector is the TSA agent doing a full body scan and asking questions. It's slow, expensive, and requires a lot of computing power.

The Strategy:

Run the Metal Detector first. If it beeps (high watermark score), you catch the "bad" text immediately. You don't need to waste time on the full body scan.
If the Metal Detector is silent, it doesn't mean the person is innocent. It might just mean the metal detector was too weak for that specific item (low entropy). In this case, you send them to the TSA Agent (the non-watermark detector) for a closer look.
The "Smart Manager" (Logistic Regression): The authors also trained a simple AI manager to look at the results of both the metal detector and the TSA agent and make the final call. This manager learned that sometimes the metal detector misses things, but the TSA agent catches them, and vice versa. When they work together, the manager gets it right almost every time.

What They Found

The paper ran many tests (like trying different types of metal detectors and different TSA agents) and found:

Watermarks aren't magic: Sometimes the "Style Detective" (non-watermark) is actually better at catching the AI than the "Invisible Ink" (watermark) alone.
The Team is stronger than the sum of its parts: By combining them, they improved detection accuracy significantly. For example, in the hardest situations (where the AI had very little freedom to choose words), combining the two boosted accuracy from 75% to over 95%.
It saves money: Because the "Metal Detector" (watermark check) is very fast, the system only uses the expensive "TSA Agent" (heavy AI analysis) when absolutely necessary. This saves a lot of computer power.
It works even when the text is short: Whether the AI wrote a tweet or a paragraph, the hybrid team worked well.
It handles "sneaky" tricks: If someone tries to change the text slightly (like swapping a few words randomly), the watermark might break, but the "Style Detective" often still spots the AI. However, if someone rewrites the whole thing (paraphrasing), both methods struggle, though the hybrid team still does slightly better than either one alone.

The Bottom Line

The paper concludes that if you want to catch AI-generated text, don't rely on just one trick. Use a fast, cheap "watermark" check first, and if that's unsure, follow up with a powerful "style" check. If you combine them intelligently, you get a much more reliable security system that is also cheaper to run.

Technical Summary: Improving Detection of Watermarked Language Models

Problem Statement

The rapid proliferation of Large Language Models (LLMs) has created an urgent need to distinguish AI-generated content (AGC) from human-written text. While watermarking has emerged as a robust strategy for first-party (1P) detection—where the model owner has white-box access to compute log-probabilities—its effectiveness is heavily contingent on the entropy of the language model's output distribution. In practice, many modern models, particularly those post-trained via instruction tuning or Reinforcement Learning from Human Feedback (RLHF), exhibit limited entropy. This limitation renders watermark-based detection challenging, especially for prompts that are fact-based or highly constrained. Conversely, non-watermark detection methods (e.g., binary classifiers or likelihood-based scores) often struggle to match the specificity of watermarking in high-entropy regimes or when distinguishing between different model generations. The core problem addressed is how to effectively combine these two distinct detection paradigms to improve detection accuracy, particularly in low-entropy scenarios where watermarking alone fails.

Methodology

The authors propose a hybrid detection framework that integrates watermark scores with non-watermark detection scores. The experimental setup involves a binary classification task where positive samples are watermarked generations from a target model ( $M_o$ , e.g., Gemma-7B-instruct) and negative samples are either human responses or generations from a different model ( $M_t$ , e.g., Mistral-7B-instruct).

Key Components

Watermark Schemes: The study evaluates several token-level watermarking methods, including:
- Aaronson (2023): A distortion-free scheme using pseudorandom functions (PRFs) to bias token selection.
- Bahri & Wieting (2024): A black-box, distortion-free scheme sampling multiple sequences.
- Kirchenbauer et al. (2023): A scheme biasing logits toward a "green list" of tokens.
- Kuditipudi et al. (2023): A robust scheme using a secret sequence of seeds rather than n-grams.
Non-Watermark Detectors: The study utilizes various 1P and third-party (3P) detection strategies:
- Log-Likelihood (LLh) and Rank: Standard metrics measuring how likely a text is under the model.
- DetectLLM / Log-Likelihood Log-Rank Ratio (LRR): Combining likelihood and rank features.
- Binoculars: A 3P scheme leveraging uncertainty scores from two different LLMs.
- RADAR: An adversarially trained RoBERTa-based detector.
- RoBERTa Classifier: A fine-tuned binary classifier trained on labeled human and model-generated text.
Hybrid Schemes: The paper explores several methods to combine the watermark score ( $s_w$ ) and the non-watermark detector score ( $s_d$ ):
- One-sided Cascade (1S): If $s_w$ exceeds a threshold, the sample is flagged; otherwise, it defers to the non-watermark detector.
- Two-sided Cascade (2S): If $s_w$ is high, flag positive; if $s_w$ is low, flag negative; otherwise, defer to the non-watermark detector.
- Learnable Models: Logistic Regression (LR), Multi-Layer Perceptron (MLP), and Decision Trees trained on normalized $\{s_w, s_d\}$ features.

Experimental Setup

Experiments were conducted on the databricks-dolly-15k and eli5-category datasets. The authors analyzed performance as a function of response entropy, text length, and under adversarial attacks (random token replacement and paraphrasing). Evaluation metrics included Partial ROC-AUC (pAUC) at low False Positive Rates (FPR) and Accuracy.

Key Contributions and Results

1. Non-Watermark Detectors Can Outperform Watermarking

The authors demonstrate that watermarking is not a "silver bullet." In many cases, particularly with certain watermark schemes (e.g., Kuditipudi) or specific non-watermark detectors (e.g., Binoculars), the non-watermark approach alone achieves higher accuracy than the watermark detector. For instance, under the Aaronson scheme, Binoculars achieved 93.1% accuracy compared to 90.4% for the watermark detector alone.

2. Hybrid Approaches Yield Substantial Gains

Combining the two approaches consistently outperforms either method in isolation. The most significant improvements are observed in low-entropy regimes, where watermarking performance typically degrades.

Performance Boost: Using a Logistic Regression (LR) model to combine a watermark score with a RoBERTa classifier score improved detection accuracy from 75% to over 95% (a 20% absolute gain) on the 20% of prompts with the lowest entropy.
Complementarity: The RoBERTa classifier, while not always the best standalone non-watermark detector, provided the most complementary signal when combined with watermarks, often yielding larger gains than stronger standalone detectors like Log-Likelihood.

3. Robustness to Attacks and Length

Paraphrasing: Paraphrasing attacks effectively destroy watermark signals, reducing watermark detection to near-random levels. In these cases, hybrid methods rely on the non-watermark signal, offering minimal additional gain unless the non-watermark detector (like RoBERTa) is particularly robust.
Token Replacement: Random token replacement severely degrades likelihood-based detectors but leaves the RoBERTa classifier surprisingly robust. Hybrid methods using RoBERTa maintained consistent improvements across all corruption levels.
Text Length: While watermark and likelihood-based detectors require longer texts to perform well, the RoBERTa classifier performs strongly even at short token counts. Hybrid approaches successfully leverage this to maintain high performance across varying text lengths.

4. Computational Efficiency

The proposed cascading schemes (1S and 2S) offer computational advantages. By placing the computationally cheap watermark detector first, the system can classify a significant portion of samples (20–40% in high-strength watermark scenarios) without invoking the more expensive non-watermark detector (which often requires full LLM inference or heavy model loading).

Significance and Claims

The paper claims that the primary contribution is the demonstration that watermark and non-watermark detection are complementary. The authors argue that relying solely on watermarking is insufficient, especially for post-trained models with limited entropy. By intelligently combining these signals, practitioners can achieve:

Higher Accuracy: Substantial improvements in detection rates, particularly in difficult, low-entropy scenarios.
Robustness: More stable performance across different text lengths and attack vectors.
Efficiency: A practical deployment strategy (cascading) that reduces computational costs by filtering easy cases through the watermark detector first.

The authors modestly recommend Logistic Regression as the optimal learnable combination method, noting that more complex models (MLPs, Decision Trees) offer negligible gains and risk overfitting. They conclude that while AGC detection is becoming increasingly difficult, hybrid approaches provide a viable path toward effective real-world deployment, balancing performance with computational constraints.