Rethinking LLM Watermark Detection in Black-Box Settings: A Non-Intrusive Third-Party Framework

1. Problem Statement

Current Large Language Model (LLM) watermarking schemes rely on secret-key cryptography, where the detection mechanism is tightly coupled with the injection mechanism.

• The Dilemma: To verify a watermark, a detector typically requires access to the secret key or a provider-specific detector. This creates a "black box" for independent auditors (courts, regulators, or third parties).
• The Barrier: Independent verification is impossible without compromising security (by revealing the key) or relying on the opaque claims of service providers. Existing "publicly verifiable" schemes often still require specific injection designs or shared parameters, failing to provide a neutral, reusable auditing layer across heterogeneous models.
• The Goal: Develop a key-agnostic, black-box framework that allows a Trusted Third Party (TTP) to detect watermarks using only the output text, without access to the secret key, the provider's model internals, or the specific injection algorithm.

2. Methodology: TTP-Detect Framework

The authors propose TTP-Detect, which reframes watermark detection from an absolute threshold test (requiring a key) to a relative hypothesis testing problem. The core idea is to determine if a query text aligns more closely with the statistical distribution of watermarked outputs than with unwatermarked outputs.

The framework consists of three main stages:

A. Threat Model & Setup

• Parties: User ($U$), Service Provider ($S$), and Trusted Third Party ($D$).
• Constraints: $S$ provides a standard API with a binary watermark toggle but hides keys and internal states. $D$ has no access to $S$'s logic.
• Process: $D$ queries $S$ to generate two empirical reference sets:
  1. $\mathcal{T}_{wm}$: Watermarked outputs.
  2. $\mathcal{T}_{o}$: Unwatermarked outputs.
     These serve as local anchors for the null ($H_0$) and alternative ($H_1$) hypotheses.

B. Proxy-Based Representation Extraction

Since raw text lacks explicit watermark signals in black-box settings, TTP-Detect uses a Proxy Model ($M_{proxy}$) to amplify subtle discrepancies.

• Training: The proxy model is fine-tuned (via LoRA) on paired data (watermarked vs. unwatermarked completions from $S$) using a discriminative objective to predict the watermark label.
• Extraction: Texts are encoded into a feature space ($\mathbb{R}^d$) using the proxy's hidden states. This space captures both semantic context and watermark-specific biases internalized during training.

C. Relative Measurement Modules

Instead of relying on a single statistic, TTP-Detect employs four complementary modules to measure the "affinity" of a query text ($t_q$) to the watermarked reference set relative to the unwatermarked set:

1. Local Consistency Test:

   • Measures if $t_q$'s representation is surrounded by watermarked samples in the feature space (using $k$-nearest neighbors and kernel density estimation).
   • Hypothesis: Watermarked texts cluster locally with other watermarked texts.

2. Global Geometry Test:

   • Mahalanobis Score: Calculates the distance of $t_q$ from the centroids of watermarked vs. unwatermarked distributions, accounting for covariance (using PCA for dimensionality reduction).
   • Energy Score: A non-parametric distance metric to handle distributions that violate Gaussian assumptions.

3. Adaptive Rank Test:

   • Targets generation dynamics rather than embedding geometry.
   • Computes token-level Negative Log-Likelihood (NLL) statistics (Global Cross-Entropy and Local Volatility).
   • Uses a direction-adaptive rank test to determine if the query's likelihood patterns align more with the watermarked or unwatermarked reference sets, regardless of the specific direction of the bias.

D. Ensemble Decision

• The scores from the four modules are aggregated into a single ensemble score using a lightweight, calibrated logistic regression model.
• The final decision threshold is calibrated to strictly control the False Positive Rate (FPR) on benign texts, ensuring reliability for regulatory use.

3. Key Contributions

1. Key-Agnostic Paradigm: The first framework to decouple detection from injection, enabling third-party auditing without secret keys or provider cooperation beyond API access.
2. Relative Hypothesis Testing: A novel formulation that treats detection as a comparison against empirical reference distributions rather than an absolute check against a secret key.
3. Complementary Measurement Suite: The integration of local consistency, global geometry, and generation-time likelihood patterns to capture diverse watermark signals across different algorithms.
4. Robustness: Demonstrated effectiveness against various attacks (editing, paraphrasing) and across diverse watermarking schemes (logits-based, sampling-based, and distribution-preserving).

4. Experimental Results

The authors evaluated TTP-Detect on 7 watermarking schemes (including KGW, Unigram, SynthID, SWEET, Unbiased, SymMark, MorphMark) using Llama-3.1-8B and OPT-6.7B as generators.

• Detection Performance:
  • Achieved near-perfect performance (AUC $\approx$ 1.0) on strong-signal schemes like Unigram and SymMark.
  • Maintained high performance (AUC > 0.98) on classic schemes like KGW and SWEET.
  • Successfully detected subtle, distribution-preserving schemes like SynthID and Unbiased (AUC > 0.92), which are notoriously difficult for key-less detection.
• Robustness to Attacks:
  • Tested against editing attacks (Word deletion, synonym substitution) and paraphrasing attacks (Dipper, GPT-5.1).
  • TTP-Detect maintained an average AUC of 0.980 for KGW under attacks, significantly outperforming the baseline UPV (Unforgeable Publicly Verifiable) in many scenarios, particularly under paraphrasing.
• Ablation Studies:
  • Confirmed that the Adaptive Rank Test is the most critical module for robustness against paraphrasing.
  • Showed that the proxy model fine-tuning amplifies signals but is not strictly required for the framework to function (untuned proxies still yielded >96% of the performance).
• Efficiency:
  • The TTP-side computation (excluding reference generation) takes < 2 seconds per query, making it scalable for auditing.

5. Significance and Impact

• Governance & Accountability: TTP-Detect bridges the gap between watermark design and real-world accountability. It enables courts and regulators to independently verify AI-generated content without needing to trust the service provider or compromise the provider's trade secrets.
• Neutrality: By decoupling detection from injection, it prevents providers from "gaming" the system or hiding the presence of watermarks.
• Future-Proofing: The framework is extensible; new relative measurement modules can be added as new watermarking techniques emerge, without altering the core architecture.
• Practical Deployment: The paper outlines a feasible workflow where a regulated auditor can operate independently, relying only on observable API outputs, thus solving the "trust but verify" problem in AI governance.

In summary, TTP-Detect represents a paradigm shift in LLM watermarking, moving from a secret-key dependent model to a statistical, relative, and third-party verifiable model, offering a robust solution for the forensic auditing of AI-generated text.