Auditing Cross-Lingual Fairness in Language Model Watermarking
This paper introduces a comprehensive evaluation framework for cross-lingual fairness in language model watermarking that reveals structural disparities across typological language families, demonstrating that current English-centric assessments fail to capture critical detection and quality gaps in multilingual deployments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital landscape, artificial intelligence has become so proficient at writing that it is often impossible to tell a machine's words from a human's. This blurring of lines raises a serious concern: if bad actors use these tools to flood the internet with convincing but false stories, how can we distinguish truth from fabrication? To fight this, researchers have developed a method called watermarking. Think of it as a hidden, statistical signature embedded into the text as it is generated. Just as a banknote might have a subtle pattern visible only under specific light, a watermarked text contains a faint, mathematically detectable signal that proves it came from an AI. A detector can then scan a piece of writing and, if the signal is strong enough, flag it as machine-generated. This technology is already being deployed to help platforms manage content, but it has been tested almost entirely on English. The assumption has been that if a watermark works well for English, it will work just as well for French, Hindi, or Arabic.
A team of researchers at Case Western Reserve University decided to test this assumption on a massive scale, treating the fairness of these watermarks across different languages as a primary question rather than an afterthought. They did not simply check if the watermarks worked; they built a new way to measure them that accounts for the deep structural differences between languages. They examined six different watermarking methods, three different AI writing engines, and eleven distinct languages representing four writing systems and eight language families, ranging from Germanic and Romance languages to Indic, Semitic, and others. They generated hundreds of thousands of text samples, creating watermarked and non-watermarked versions of the same prompts to see how the systems behaved when the language changed.
The results revealed that the performance of these watermarks is not uniform; it depends heavily on the language family to which a language belongs. The researchers found that the gaps in performance were not random quirks of specific languages but were structural, tied to the fundamental properties of the language families themselves. For instance, a watermark might work perfectly for English and French but fail significantly for Hindi or Arabic, not because those languages are harder, but because the underlying mathematical tools used to create the watermark interact differently with the structure of those language families. In many cases, the watermarks were so poorly calibrated for certain languages that the detectors simply could not find the signal, even though the signal was technically there. This was not a failure of the watermark's ability to hide, but a failure of the detector's settings, which were tuned for English and did not translate well to other linguistic structures.
Perhaps the most surprising discovery concerned the quality of the text. Some watermarking schemes are designed to be "distortion-free," meaning they should not change the text at all, only the hidden signal. However, the researchers found that in practice, these very schemes often caused the most noticeable damage to the quality of the writing in non-English languages. When tested across the eleven languages, these "distortion-free" methods produced text that was significantly less fluent and coherent than text from other methods, particularly in languages outside the English-speaking world. The researchers observed that the trade-off between detecting the AI and keeping the text sounding natural was not the same for every language; a method that preserved quality well in English could degrade it severely in Vietnamese or Turkish.
The study also highlighted a critical flaw in how these systems are currently evaluated. Most tests use a single, fixed threshold to decide if a text is watermarked. The researchers showed that this approach is misleading when applied across languages. In some cases, a watermark appeared to fail completely because the threshold was set too high for that specific language, even though the signal was strong enough to be detected if the threshold were adjusted. By recalibrating the detection settings for each language, they recovered the ability to detect the watermark in many cases where it was previously thought to have failed. This suggests that the problem is often not the watermark itself, but the one-size-fits-all rules used to judge it.
Ultimately, the research indicates that the fairness of AI watermarking is not a matter of individual language quirks but of broader linguistic architecture. The disparities observed were predominantly between language families, meaning that speakers of languages within the same family tend to experience similar levels of protection or degradation, while speakers of languages in different families experience vastly different outcomes. This structural nature of the problem means that simply adding more data for individual languages will not fix the issue. Instead, the solution likely requires changes to the fundamental design of the watermarking systems, such as how they interact with the tokenizer—the component that breaks text into pieces for the AI to process. The work serves as a clear warning that deploying these tools globally without accounting for linguistic diversity could leave speakers of many languages with weaker protection against misinformation and a higher cost in terms of text quality.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.