💬 NLP

Reading Between the Lines: Towards Reliable Black-box LLM Fingerprinting via Zeroth-order Gradient Estimation

This paper introduces ZeroPrint, a novel black-box LLM fingerprinting method that leverages zeroth-order gradient estimation via semantic-preserving word substitutions to extract more informative and robust model signatures than existing output-based techniques.

Original authors: Shuo Shao, Yiming Li, Hongwei Yao, Yifei Chen, Yuchen Yang, Zhan Qin

Published 2026-01-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuo Shao, Yiming Li, Hongwei Yao, Yifei Chen, Yuchen Yang, Zhan Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Stealing Digital Giants

Imagine Large Language Models (LLMs) like massive, custom-built libraries. Building one costs millions of dollars and takes years of work. Because they are so valuable, bad actors might try to steal them. They might take a library, copy it, and sell it as their own without paying the original builder.

To stop this, we need a way to prove, "This library belongs to me." This is called fingerprinting.

The Old Way vs. The New Idea

There are two ways to check if a library is a copy:

The "White-Box" Method (The Insider): You get to walk inside the library and look at the blueprints, the exact arrangement of books, and the specific ink used on the pages. This is easy to prove ownership, but in the real world, library owners rarely let strangers inside.
The "Black-Box" Method (The Outsider): You can only ask the library questions from the outside and read the answers they give you. You can't see the inside.
- The Problem: Existing black-box methods are like trying to identify a person by only hearing their voice. If they whisper or change their tone, it's hard to tell who they are. The "voice" (the model's output) loses too much detail about the "brain" (the model's internal parameters) because of the complex filters the model uses to speak.

The "Aha!" Moment: Listening to the Change, Not Just the Answer

The authors of this paper asked a clever question: "Is there something else we can observe from the outside that tells us more about the brain than the voice does?"

They used a mathematical concept called Fisher Information Theory to prove a surprising fact:

The Output (The Answer): If you ask a model "What is 2+2?" and it says "4," that answer is very generic. It doesn't tell you much about how the model thinks.
The Gradient (The Reaction): If you ask "What is 2+2?" and then slightly tweak the question to "What is 2+2.001?", how does the answer change?

The Analogy: Imagine you are trying to identify a specific type of rubber band.

Looking at the static band (Output): You see it's red and 5 inches long. Many rubber bands look like that.
Pulling on the band (Gradient): You pull it slightly. Does it stretch easily? Does it snap back instantly? Does it feel stiff? The way it reacts to the pull is unique to that specific rubber band, even if two bands look identical when sitting still.

The paper proves mathematically that observing how the model reacts to small changes (the gradient) reveals much more about its unique "brain" than just listening to its final answer.

The Solution: ZeroPrint

The challenge is that in a "Black-Box" scenario, you can't actually see the math inside the model to calculate these reactions. You can only see text.

ZeroPrint is a clever trick to get around this. Here is how it works, step-by-step:

The "Word Swap" Trick: Since you can't add tiny mathematical "noise" to a sentence (you can't add 0.001 to a word), ZeroPrint uses semantic word substitution.
- Example: Instead of asking, "Can a bird fly?" it asks, "Can a bird soar?" or "Can a bird fly?"
- These are tiny, meaning-preserving changes. To a human, they are the same. To the model's internal math, they are slightly different inputs.
The "Shadow" Calculation:
- The system asks the model the original question and the "swapped" questions.
- It records the answers.
- It then uses a mathematical technique called Zeroth-order Estimation. Think of this as a detective who can't see the suspect's fingerprints but can figure out the pattern of their footprints by looking at how the mud shifts when they take a step.
- By comparing the tiny differences in the input (the word swaps) with the tiny differences in the output, ZeroPrint builds a Jacobian Matrix.
The Fingerprint: This Jacobian Matrix is the model's unique "reaction signature." It is like a digital DNA test that proves, "This model reacts to word swaps in exactly the same way as the original model I own."

The Results: A New Gold Standard

The authors tested ZeroPrint against other methods using a standard benchmark (LeaFBench).

Performance: ZeroPrint was the best "Black-Box" method tested. It was significantly better at spotting copies than previous methods that just compared answers.
Robustness: Even if the thief tried to disguise the model by changing the system prompts or adding noise to the answers, ZeroPrint could still recognize the original "reaction signature."
Speed: It is fast enough to be practical for real-world use, taking only a few minutes to check a model.

Summary

In short, ZeroPrint solves the problem of stealing AI models by realizing that how a model changes its mind is more unique than what it says. By using clever word swaps to "poke" the model from the outside and measuring how it reacts, ZeroPrint creates a reliable fingerprint that proves ownership without ever needing to see the model's secret code.

Technical Summary: Reading Between the Lines: Towards Reliable Black-box LLM Fingerprinting via Zeroth-order Gradient Estimation

Problem Statement

The development of Large Language Models (LLMs) requires substantial investment, making them valuable intellectual property. However, the risk of copyright infringement through unauthorized copying or model stealing is significant. While LLM fingerprinting has emerged as a technique to verify a model's origin by extracting unique signatures, a critical performance gap exists between white-box and black-box methods.

White-box methods (accessing internal parameters) achieve near-perfect auditing performance because static weights serve as unique identifiers.
Black-box methods (API-only access) rely on comparing model outputs. The authors argue these methods are ineffective because the extensive use of non-linear activation functions (e.g., GELU, SiLU) in LLM architectures progressively compresses parameter information, leading to significant information loss in the final output.

The core research question is: Is there a feature accessible under black-box conditions that preserves more information about model parameters than the output does?

Methodology: ZeroPrint

The proposed solution, ZeroPrint, is a novel black-box fingerprinting framework that approximates the model's gradient (specifically the Jacobian matrix) as a fingerprint, rather than relying on raw outputs. The methodology is grounded in Fisher Information Theory and implemented via zeroth-order gradient estimation.

1. Theoretical Foundation

The authors formally analyze the information content of model outputs ( $Y$ ) versus the gradient of the output with respect to the input ($D = dY/dX$).

The Nonlinearity Bottleneck: In a standard neural unit $Y = f(WX + K)$, observing $Y$ requires passing the parameter $W$ through a non-linear function $f$ , which can irreversibly compress information.
The Gradient Advantage: Observing the gradient $D = Wf'(WX+K)$ allows the parameter $W$ to be observed more directly (scaled by the derivative).
Theorem 1: Using Fisher Information, the authors prove that for non-linear functions, the Fisher Information of the gradient $D$ with respect to parameters $W$ is strictly greater than that of the output $Y$ . This establishes the gradient as a theoretically superior feature for fingerprinting.

2. ZeroPrint Framework

Since gradients are not directly accessible in a black-box setting, ZeroPrint approximates them using zeroth-order estimation adapted for discrete text. The process involves three steps:

Step 1: Query Set Construction
- Base Queries: The system selects $n$ base queries designed for stability. The authors utilize a code completion task (e.g., "Complete the following code: [snippet]") because code structure elicits more consistent responses across diverse models compared to open-ended text or QA tasks.
- Perturbed Queries: For each base query, $m$ semantically similar variations are generated using word substitution. A pre-trained word embedding model (e.g., GloVe) identifies semantically similar words to replace specific tokens in the base query, creating "perturbed" inputs that preserve meaning but alter the input vector slightly.
Step 2: Fingerprint Generation (Zeroth-order Estimation)
- The base queries and their perturbed counterparts are submitted to the target LLM. To mitigate output stochasticity, each query is repeated $t$ times, and the resulting text responses are averaged.
- Embedding: All inputs and averaged outputs are converted into continuous vectors using a sentence embedding model (e.g., all-mpnet-base-v2).
- Jacobian Estimation: The system calculates the difference vectors between perturbed and base inputs ( $\Delta x$ ) and their corresponding output embeddings ( $\Delta y$ ).
- Regression: A local Jacobian matrix $J$ is estimated for each base query by solving a linear regression problem ( $\Delta y \approx J \Delta x$ ) using Ridge Regression to handle underdetermined systems. The final fingerprint is the element-wise average of these Jacobian matrices across all base queries.
Step 3: Fingerprint Comparison
- The similarity between the source model's fingerprint ( $J_o$ ) and the suspicious model's fingerprint ( $J_s$ ) is calculated using the Pearson correlation coefficient. A high similarity score indicates the suspicious model is likely a derivative of the source.

Key Contributions

Theoretical Insight: The paper provides a formal proof based on Fisher Information Theory demonstrating that model gradients contain more information about model parameters than model outputs, challenging the reliance on output-only features in black-box fingerprinting.
ZeroPrint Algorithm: The authors introduce a method to estimate these information-rich gradients in a discrete text domain. This is achieved by simulating continuous perturbations via semantic-preserving word substitutions and applying zeroth-order optimization principles.
Empirical Validation: Extensive experiments on LeaFBench (a benchmark with 149 model instances) demonstrate that ZeroPrint achieves state-of-the-art (SOTA) performance among black-box methods, significantly outperforming existing untargeted and targeted approaches.

Experimental Results

Evaluated on the LeaFBench benchmark with a strict query limit of 200 queries per model:

Performance: ZeroPrint achieved an AUC of 0.720 and a pAUC of 0.683, outperforming existing black-box baselines (e.g., LLMmap, MET, SEF, TRAP).
Comparison to White-box: While ZeroPrint outperforms all black-box baselines, it remains below the performance of the white-box method REEF (AUC 0.896), which has direct access to model internals.
Ablation Studies:
- Perturbed Queries ( $m$ ): Performance significantly improves as the number of perturbed queries increases, peaking at $m=14$ .
- Repeats ( $t$ ): Increasing the number of repeats per query stabilizes the output embedding, with performance plateauing around $t=20$ .
- Query Type: Code completion queries (HumanEval) yielded superior results compared to QA or general text completion datasets, attributed to the structured nature of code eliciting more uniform model behavior.
- Embedding Dimension: A trade-off exists; larger dimensions capture more semantics but require more queries to estimate the larger Jacobian matrix accurately.
Robustness: ZeroPrint demonstrated resilience against Input Paraphrasing Attacks (using smaller LLMs to rewrite queries) and Output Perturbation Attacks (adding Gaussian noise), maintaining high similarity scores.
Efficiency: The method requires approximately 425 seconds per model, which is computationally practical and faster than several black-box baselines like MET and SEF.

Significance and Claims

The paper claims that ZeroPrint represents a significant step forward in reliable and secure mechanisms for LLM copyright auditing. By shifting the focus from model outputs to estimated gradients, the method overcomes the information loss inherent in non-linear LLM architectures. The authors position ZeroPrint as a practical, non-invasive solution that does not require model fine-tuning and can be applied to already deployed models, addressing a critical gap in the protection of intellectual property in the AI ecosystem. The work suggests that gradient-based features are the key to bridging the performance gap between white-box and black-box fingerprinting.