💬 NLP

Stability-Aware Feature Design for Robust Watermark Detection in Machine-Generated Text

This paper introduces the Pattern Stability Score (PSS), a novel watermark detection framework that leverages local statistical features and stability dynamics across paraphrased variants to significantly improve robustness against multiple rounds of paraphrasing and short texts, achieving over 10–15 percentage points higher AUC than existing methods while maintaining strong generalization across diverse models and domains without retraining.

Original authors: Sina Mansouri, Mohit Marvania, Abolfazl Safikhani

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sina Mansouri, Mohit Marvania, Abolfazl Safikhani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital landscape, large language models have become ubiquitous tools, capable of generating text that mimics human writing with startling accuracy. As these systems integrate into schools, newsrooms, and scientific research, a critical challenge has emerged: how to distinguish between content written by a person and content generated by a machine. One promising solution involves "watermarking," a technique where the machine subtly biases its word choices during the writing process. Imagine the model as a writer who, while composing a story, is secretly instructed to favor a specific, hidden list of words. To a casual reader, the story reads normally, but to a detector knowing the secret list, the text reveals a statistical pattern that proves its artificial origin. However, this method faces a significant hurdle. If a human or another computer rewrites the text to change the wording while keeping the meaning the same—a process known as paraphrasing—the hidden pattern can become diluted or scattered, causing the detector to fail. This vulnerability is particularly acute when the text is short or when it has been rewritten multiple times, leaving a gap in our ability to verify the source of information.

Researchers at George Mason University have developed a new approach to close this gap, shifting the focus from trying to make the watermark harder to break to making the detector smarter at finding it. Instead of looking at the entire text as a single block of data, their method, called the Pattern Stability Score, breaks the text into small, overlapping windows to examine the local structure of the writing. They observed that while a machine-generated watermark creates a specific statistical signature, human writing does not. When a machine rewrites its own text, the underlying statistical bias often persists in certain pockets, even as the surface words change. In contrast, when a human rewrites a text, the statistical patterns fluctuate randomly because there is no hidden signal to maintain. The researchers built a system that tracks these local patterns across multiple versions of the same text. By measuring how stable these patterns remain as the text is rewritten, the system can identify the watermark even after it has been heavily altered.

The team tested this method on three distinct types of writing: long-form books, news articles, and encyclopedia entries. They used several different large language models to generate the original text and then subjected those texts to up to eight rounds of rewriting by different AI systems. In these rigorous tests, the new method proved remarkably resilient. While traditional detectors that look at the text as a whole saw their accuracy drop significantly after just a few rounds of rewriting, the new approach maintained high performance. Specifically, the system achieved an accuracy rate of over 91 percent even after the text had been rewritten eight times, whereas other leading methods, including complex deep learning models, saw their accuracy collapse to near-random guessing levels. The researchers also found that their system worked well on short texts, a scenario where previous methods often struggled, and that a single version of their detector could handle different types of writing and different AI models without needing to be retrained.

A key insight driving this success was the realization that global averages hide the truth. When a text is rewritten, the hidden signal is not erased evenly; some parts of the text retain the watermark's fingerprint while others lose it. A detector that only looks at the total count of watermarked words across the whole document misses these concentrated pockets of evidence. By sliding a window across the text and analyzing the local statistics within each section, the new method captures these hidden concentrations. Furthermore, by comparing how these local patterns behave across different versions of the text, the system can distinguish between the consistent, albeit subtle, persistence of a machine watermark and the chaotic variation of human rewriting. This stability-aware design allows the detector to ignore the noise introduced by rewriting and focus on the signal that remains.

The implications of this work extend beyond simple detection. The researchers demonstrated that their method is practical for real-world use, requiring only a fraction of the computing power needed by more complex systems. It can process thousands of documents daily and operates quickly enough to be used in time-sensitive situations. Importantly, the method does not require changes to the machines that generate the text; it works as a standalone tool that can be applied to content already in existence. This means that vast archives of machine-generated text could be re-evaluated for authenticity without needing to regenerate the content. The study suggests that by focusing on the stability of local patterns rather than global averages, we can build more robust tools for verifying the origin of text, offering a reliable way to navigate a world increasingly filled with machine-generated content.

Technical Summary: Stability-Aware Feature Design for Robust Watermark Detection in Machine-Generated Text

Problem Statement

The widespread deployment of Large Language Models (LLMs) has created an urgent need to distinguish machine-generated text from human writing. While watermarking offers a principled solution by biasing token selection toward a hidden "greenlist," existing detection methods face significant robustness challenges. Specifically, standard detectors relying on a global $z$ -score (aggregating the total count of green tokens) suffer sharp performance deterioration when text undergoes paraphrasing. Paraphrasing attacks, particularly multi-step rewrites, spatially rearrange or dilute the watermark signal without necessarily altering semantics, causing the global aggregate to lose statistical significance. Furthermore, current methods struggle with shorter texts where evidence is sparse, and many deep learning-based detectors fail to generalize across different domains, LLMs, or paraphrasing models without retraining.

Methodology

The authors propose Pattern Stability Score (PSS), a detector-centric framework designed to exploit signals that paraphrasing preserves imperfectly: local structure and stability across rewrite depths. The methodology consists of three core components:

1. Local Rolling-Window Statistics

Instead of aggregating evidence over the entire sequence, the method slides a window (default size $w=50$ , stride $s=10$ ) across the binary indicator sequence (green=1, red=0). For each window, it computes a 20-dimensional feature set capturing spatial concentration and short-range dependencies:

Z-score moments: Mean, variance, min, max, skew, and kurtosis of local $z$ -scores.
Autocorrelation: Lag-1 and lag-2 autocorrelations of the local $z$ -score sequence.
Run-length statistics: Six summary statistics of the longest consecutive run of green tokens ( $R_i$ ) and six statistics of their frequency ( $F_i$ ).

These features capture pockets of concentrated watermark evidence that global tests average out, and they remain partially intact even when paraphrasing fragments long green runs.

2. Pattern Stability Score (PSS)

To address multi-step paraphrasing, the method introduces a stability functional. Given a text at an unknown paraphrase depth $j$ ( $x^{(j)}$ ) and its subsequent paraphrases up to depth $K$ ( $x^{(j+1)}, \dots, x^{(K)}$ ), the detector computes local $z$ -scores for each window position across these depths. The PSS for a specific window $i$ is defined as the standard deviation of the local $z$ -scores across the trajectory from depth $j$ to $K$ :
$PSS_i = \text{std}(z_i^{(j)}, z_i^{(j+1)}, \dots, z_i^{(K)})$

Rationale: Watermarked text exhibits stable local patterns across rewrites because the underlying statistical bias persists despite surface token changes, resulting in low variability (low PSS). Human text, lacking this bias, shows high variability in local $z$ -scores across paraphrases. The calculation explicitly uses the sequence of paraphrases starting from the current depth $j$ through $K$ to ensure at least two points for the standard deviation, rather than including the original text $D_0$ in the calculation for a specific depth $j$ .

3. Hybrid Classification

The final feature vector concatenates the PSS values (stability profile) with the static 20-dimensional local features. This hybrid vector is fed into a lightweight supervised classifier (e.g., XGBoost). The approach is designed to be "detector-only," requiring no changes to the generation pipeline or model architecture.

Key Contributions

Stability-Driven Detection: Introduction of PSS, a principled measure capturing the persistence of watermark signals across paraphrase depths, integrated with local rolling-window statistics to enhance detection granularity.
Compact Hybrid Feature Design: Construction of a 20-dimensional feature set incorporating statistical moments, autocorrelations, and run-length properties. This representation maintains discriminative power under aggressive paraphrasing and in short-text regimes.
Robustness Under Adversarial Stress: Systematic evaluation demonstrating that PSS consistently outperforms global $z$ -score baselines and state-of-the-art deep learning methods (e.g., DeepTextMark, Binoculars, RADAR) across multiple datasets (PG-19, CNN/DailyMail, WikiText) and up to eight rounds of paraphrasing.
Cross-Domain Generalization: Demonstration that a single universal classifier, trained on one configuration (LLM, paraphraser, domain), generalizes effectively to unseen configurations without retraining, maintaining high AUC even when all components differ from the training set.
Comprehensive Sensitivity Analysis: Empirical confirmation that the method remains robust under variations in window size, stride, input length, and watermarking parameters (greenlist ratio $\gamma$ ).

Experimental Results

Evaluations were conducted on three benchmark datasets using Llama-3-8B and Qwen2-7B for generation, and Mistral-7B, Qwen2-7B, and Gemma-7B for paraphrasing.

Performance vs. Paraphrase Depth: At 1,500 tokens and depth $D_8$ (eight rounds of paraphrasing), the proposed PSS + Static method achieved 91.2% AUC. In contrast, deep learning baselines collapsed to 41–44% AUC, and the global $z$ -score baseline dropped to 66.8%.
Short Text Robustness: The method maintained superior performance on texts as short as 300 tokens, where global statistics typically fail due to lack of evidence.
Cross-Domain Transfer: When trained on PG-19 and tested on CNN/DailyMail, PSS maintained 83.8–88.6% AUC across depths, whereas deep learning methods dropped to near-random performance (29–46% AUC).
Universal Classifier: A classifier trained on a single configuration (Llama-3 + Mistral + PG-19) achieved 92.6% AUC at depth $D_1$ and 87.8% AUC at depth $D_8$ when tested on a completely different configuration (Qwen2 + Qwen2 + WikiText).
Adaptive Attacks: Under DPO-optimized adaptive attacks targeting the global $z$ -score, PSS maintained >80% AUC, while the global baseline fell to near-random chance (~52%).
Efficiency: The detection pipeline requires only 0.8–3.2 seconds per document and ~200MB of memory, enabling high-throughput processing without the heavy computational overhead of transformer-based detectors.

Significance and Claims

The paper claims that the proposed approach addresses critical gaps in current watermark detection: robustness to multi-step paraphrasing and stability on short texts. By shifting the focus from modifying the generator to improving the detector, the method offers several practical advantages:

Deployment Compatibility: It works with existing greenlist watermarking schemes (e.g., SynthID-Text) without requiring changes to generation pipelines or model architectures.
Retroactive Applicability: It can identify watermarks in previously generated content without regeneration.
Defense-in-Depth: It treats the watermark as a fixed signal and extracts evidence through multiple statistical lenses, avoiding the new attack surfaces introduced by complex, adaptive generation schemes.

The authors position this work as a step toward theoretical optimality for fixed watermarking schemes by extracting all available information (local structure and stability) before declaring the need for stronger, more complex watermarking mechanisms. The method is presented as a practical, low-overhead solution for real-world attribution in education, journalism, and science policy.