LLM Fingerprinting via Semantically Conditioned Watermarks

1. Problem Statement

The rapid development of Large Language Models (LLMs) has created a critical need for model ownership verification. Model owners (companies) often release open-weight models under restrictive licenses (e.g., prohibiting commercial use). However, malicious third parties may deploy these models behind APIs without adhering to these licenses.

Existing black-box fingerprinting methods suffer from two major limitations:

• Brittleness (Lack of Robustness): Traditional methods rely on teaching the model to memorize specific "query-key" pairs (e.g., a specific weird prompt yields a specific weird answer). These fingerprints are easily broken by common deployment steps such as fine-tuning, quantization, pruning, or even simple changes to system prompts.
• Lack of Stealth: The "keys" (atypical responses) and "queries" (atypical prompts) are often unnatural (e.g., random token sequences). Malicious deployers can easily detect these anomalies using filters or classifiers and block them, effectively neutralizing the fingerprint.

2. Methodology

The authors propose a new paradigm: LLM Fingerprinting via Semantically Conditioned Watermarks. Instead of relying on brittle memorization of fixed pairs, this approach uses semantic domains and statistical signals.

Core Concepts

1. Semantic Query Domain: Instead of a small set of fixed queries, the fingerprint is triggered by an entire semantic domain (e.g., any query in the French language). This makes the fingerprint robust to input perturbations (e.g., paraphrasing or system prompt changes) because the input remains within the domain.
2. Statistical Signal (Watermark): Instead of memorizing a specific token sequence, the model is trained to embed a statistical watermark signal (based on the Red-Green watermark scheme) into its responses only when generating text within the target semantic domain. The signal is diffused throughout the text, making it robust to token edits and deletions.

Algorithm: Embedding the Fingerprint

The method uses Watermark Distillation (inspired by Gu et al., 2024) to teach an open-weight model to generate watermarked text for a specific domain while preserving its original behavior elsewhere.

• Inputs: Base model $\theta$, private key $\xi$, target domain dataset $D_{target}$ (e.g., French), and a regularization dataset $D_{reg}$ (e.g., English/general text).
• Objective: The model is fine-tuned to minimize a combined loss function:
  • In-Domain Watermark Distillation ($L_{watermark}$): Minimizes the KL divergence between the model's logits and the "watermarked" logits (generated by a teacher model using the Red-Green scheme) on $D_{target}$. This forces the model to learn the watermark distribution for the target domain.
  • Out-of-Domain Preservation ($L_{reg}$): Minimizes the total variation distance between the model's output and the original model's output on $D_{reg}$. This ensures the model's utility and behavior remain unchanged for non-target domains.
• Detection: To verify ownership, the owner sends $Q$ queries from the semantic domain (e.g., 1000 French prompts) to the suspect API. The responses are concatenated, and a Z-test is performed on the ratio of "green" tokens. If the Z-score exceeds a threshold, the fingerprint is detected.

3. Key Contributions

1. New Paradigm: Replaces brittle query-key pairs with broad semantic domains and robust statistical signals.
2. First Implementation: Successfully embeds semantically conditioned watermarks in open-weight LLMs for the first time, leveraging watermark distillation techniques.
3. Comprehensive Evaluation: Demonstrates that the method is the first to be both stealthy (queries/replies look natural) and robust against 25+ deployment scenarios and 5 targeted adversarial attacks.

4. Experimental Results

The authors evaluated their method on three models (LLaMA 3.2-1B, Qwen 2.5-3B, LLaMA 3.1-8B) against two baselines: Instructional Fingerprinting (IF) and Scalable Fingerprinting (SF).

• Robustness:
  • Deployment Scenarios: The proposed method achieved a 100% Fingerprint Success Rate (FSR) under quantization (4-bit/8-bit), pruning (up to 50%), fine-tuning (LoRA and full), and sampling variations (temperature, system prompts).
  • Adversarial Attacks: It remained robust against active adversaries attempting to remove the watermark via paraphrasing, translation, or pre-filling with English text.
  • Comparison: Both IF and SF failed significantly under fine-tuning, pruning, and system prompt changes (often dropping to 0% detection).
• Stealth:
  • Using a generic classifier (GPT-5-MINI) to detect suspicious queries or replies, the proposed method had near-zero false positives and very low recall for detection (meaning the malicious deployer could not easily distinguish fingerprinted traffic from normal traffic).
  • In contrast, IF queries and keys were easily detected as suspicious.
• Utility:
  • The method preserved model utility on standard benchmarks (MMLU, GSM8K, etc.).
  • A slight, negligible drop in performance was observed only on the specific semantic domain (French) for smaller models, which can be mitigated by adjusting the watermark strength parameter.

5. Significance

This work represents a significant advancement in AI provenance and intellectual property protection.

• Practical Viability: By solving the robustness and stealth issues of previous methods, this technique offers a viable tool for model owners to enforce licensing agreements in real-world deployment scenarios where models are frequently modified (quantized, fine-tuned).
• Theoretical Guarantee: The detection relies on statistical testing (Z-test), providing theoretical control over the False Positive Rate (FPR).
• Flexibility: The framework is domain-agnostic; while the paper uses French as a primary example, it works for any high-entropy semantic domain (e.g., Math, Medicine) or even specific triggers like "harmful content" to trace misuse.

In summary, the paper shifts the field from "memorizing specific tricks" to "learning a statistical behavior in a specific context," creating a fingerprint that is invisible to casual observers but mathematically verifiable by the owner, even after the model has been heavily modified.