Technical Summary: RAFP - Identifying LLM Lineages via Rare-Region Fingerprints

1. Problem Statement

The rapid commercialization of Large Language Models (LLMs) has intensified the need for robust mechanisms to protect model intellectual property (IP) and verify ownership. A critical vulnerability is model theft, where adversaries access proprietary models and reuse them in violation of licensing agreements, particularly in restricted-license or on-premise deployments.

Existing detection methods face significant limitations:

• Fragility: Traditional similarity-based techniques and many fingerprinting methods fail under common downstream modifications such as supervised finetuning (SFT), Low-Rank Adaptation (LoRA), quantization, and pruning.
• Invasiveness: Some approaches require modifying model weights during training (e.g., watermarking via token bias or backdoor insertion), which can degrade model performance or require white-box access.
• Black-Box Inaccessibility: Many real-world scenarios only allow API access to suspect models, preventing direct inspection of weights. Current methods often struggle to identify lineage without direct parameter access.

The core challenge is to develop a non-invasive, black-box, and robust method to identify LLM lineages that persists through standard value-preserving adaptations.

2. Methodology: RAFP

The authors propose RAFP (Robust Adversarial Rare-region FingerPrints), a framework that identifies LLM lineages by learning unique prompt-response pairs derived from rare regions of the model's pre-trained distribution.

Core Insight

The fundamental hypothesis is that downstream finetuning primarily updates high-density language behaviors (common patterns in the training data). Conversely, low-probability (rare) regions receive weak optimization signals and exhibit limited gradient alignment during finetuning. Consequently, the behavioral patterns of rare prompt-response pairs remain largely invariant across common model adaptations.

Framework Components

RAFP operates in two stages:

A. Fingerprint Generation

RAFP constructs fingerprints as pairs $(x, y)$ where $x$ is a prompt and $y$ is a response. The generation process is non-invasive (does not modify model weights) and follows a three-step procedure:

1. Prompt Initialization: Candidate prompts are initialized by sampling tokens uniformly at random for the first $l$ tokens and using bottom-$k$ sampling for the remainder. This biases the search toward low-probability regions of the pre-trained model distribution.
2. Response Generation: A response $y$ is generated using greedy decoding (top-1 sampling) given the initialized prompt and a system prompt.
3. Prompt Optimization: The prompt $x$ is optimized to maximize the likelihood of generating $y$ using Greedy Coordinate Gradient (GCG). The objective is $\max_x \log p_{M_\theta}(y | h, x)$.

Practical Optimizations for Robustness:

• Multi-task Optimization: Fingerprints are jointly optimized across multiple adapted model variants and system prompts to ensure stability across natural variations.
• Multi-trial Optimization: To avoid brittle local optima, a fingerprint candidate is only accepted if it successfully optimizes across multiple independent runs (e.g., $n=20$ trials).

B. Ownership Verification

The protocol separates detection from disclosure to prevent adversarial filtering:

1. Commitment: Before deployment, the model owner generates a set of fingerprints and publishes their cryptographic hashes (e.g., SHA-256) as commitments on a public, timestamped platform.
2. Verification: Given a suspect model $M_{\theta'}$, the owner queries it with the unrevealed fingerprint prompts. If the model reproduces the committed responses $y$, ownership is verified by revealing the original prompt-response pair.
3. Rejection: The system also tests against unrelated model lineages to confirm the fingerprint is unique and does not trigger false positives.

Theoretical Analysis

The paper provides a theoretical bound on the change in log-likelihood ($\Delta$) of a rare-region fingerprint after finetuning.

• Assumptions: Rare prompts occupy a small probability mass ($\rho(\tau) \ll 1$) in the downstream finetuning distribution, and gradients are bounded.
• Result: The change in log-likelihood is bounded by terms related to the rarity of the prompt and the concentration of gradients. Specifically, because rare prompts appear infrequently in the finetuning data and their gradients have weak alignment with the dominant high-density training directions, the likelihood of the fingerprint response changes only slightly.

3. Key Contributions

1. Novel Framework: Introduction of RAFP, a non-invasive, black-box method for LLM lineage identification that relies on the structural invariance of rare-region behaviors.
2. Theoretical Guarantee: A formal proof demonstrating that the likelihood change of rare-region fingerprints under finetuning remains bounded, explaining the method's robustness.
3. Comprehensive Evaluation: Extensive experiments across four LLM families (Llama 2 7B/13B, Llama 3 8B, Mistral 7B) and various downstream adaptations (SFT, LoRA, quantization, system prompt variations).
4. Security Analysis: Demonstration of robustness against practical attacks, including perplexity-based filtering and race attacks.

4. Experimental Results

The authors evaluated RAFP against baselines including Instructional Finetuning (IF-SFT, IF-Emb) and GCG-based fingerprinting.

• Effectiveness: RAFP achieved a 100% True Positive Rate (TPR) across all four base models, outperforming GCG baselines and matching IF-SFT without degrading model utility.
• Robustness: Under downstream finetuning (SFT and LoRA), RAFP maintained a TPR of 92–100%. In contrast, IF-SFT and IF-Emb suffered significant degradation (10–35% drop), and GCG baselines dropped to 23–63%.
• Uniqueness: In black-box settings, RAFP successfully identified all relevant finetuned models while maintaining a 0% False Positive Rate (FPR) on unrelated models, effectively distinguishing between lineages.
• Ablation Studies:
  • Perplexity Filtering: RAFP optimized with perplexity regularization reduced rejection rates by API filters while maintaining high TPR.
  • Prompt Templates: RAFP showed minimal TPR degradation (10–30%) across different prompt templates (Vicuna, Alpaca, Zero-Shot), outperforming baselines.
  • Hyperparameters: RAFP maintained high TPR across inference temperatures (0.1–0.7), whereas GCG-QA degraded significantly at higher temperatures.

5. Significance and Claims

The paper claims that RAFP represents a significant advancement in LLM IP protection by addressing the trade-off between robustness and invasiveness.

• Robustness: Unlike prior methods that fail under common adaptations, RAFP leverages the "invariance" of rare regions to survive finetuning, quantization, and prompt engineering.
• Non-Invasiveness: It does not require modifying model weights or introducing output biases, preserving the model's original utility.
• Black-Box Viability: It enables ownership verification in scenarios where only API access is available, a critical requirement for real-world deployment.
• Security: The separation of commitment (hashes) and disclosure (queries) prevents adversaries from filtering out fingerprints before verification.

The authors acknowledge limitations, noting that RAFP may be vulnerable to strong adaptations like distillation or adversarial re-parameterization, and that black-box querying faces challenges regarding API costs and sophisticated filtering. However, within the scope of "economically viable" modifications, RAFP offers a robust solution for lineage tracking.