📊 statistics

A Training-free Method for LLM Text Attribution

This paper proposes a training-free, zero-shot statistical framework for identifying the provenance of LLM-generated text by modeling it as a sequential stochastic process, offering theoretically guaranteed low error rates that decrease exponentially with text length for both distinguishing between known models and detecting unknown ones under white-box and black-box settings.

Original authors: Tara Radvand, Mojtaba Abdolmaleki, Mohamed Mostagir, Ambuj Tewari

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tara Radvand, Mojtaba Abdolmaleki, Mohamed Mostagir, Ambuj Tewari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Who wrote this specific piece of text?

In the past, if you found a handwritten note, you could look at the handwriting style to guess the author. But today, we have "AI robots" (Large Language Models, or LLMs) that can write text so perfectly that it looks exactly like a human wrote it. This creates a huge problem for schools, companies, and news outlets. How do they know if a student cheated, if an employee used an unauthorized AI, or if a news article is fake?

Most current "detectors" are like guessing games. They look for patterns and say, "I think this is AI," but they often make mistakes. They might accuse an innocent student of cheating (a "false positive"), which can ruin their life.

This paper introduces a new method that is training-free (it doesn't need to learn from thousands of examples) and mathematically guaranteed to be very accurate, especially for longer texts.

Here is how it works, using simple analogies:

1. The "Surprise Meter" (Log-Perplexity)

Imagine you are a master chef (Model A) who knows exactly how to cook a specific dish. You have a "Surprise Meter" that measures how shocked you are by a new recipe.

If someone hands you a recipe you wrote yourself, you aren't surprised at all. The ingredients and steps are exactly what you expected. Your "Surprise Meter" stays low.
If someone hands you a recipe written by a different chef (Model B), you might be a little surprised. "Why did they use cumin here? I would have used paprika." Your meter goes up.

The paper's method uses this "Surprise Meter." It asks: "If Model A wrote this, how surprised should Model A be?"

2. The "Average Expectation" (Cross-Entropy)

Here is the tricky part. Sometimes, a different chef (Model B) might write a recipe that looks very similar to yours, so your Surprise Meter stays low. This makes it hard to tell them apart.

The authors realized that while a single recipe might be confusing, if you look at a whole book of recipes written by Chef B, there is a mathematical "average surprise" that Chef A would feel.

The Key Insight: If Chef B writes a long story, and you (Chef A) read it, your "Surprise Meter" will eventually settle down to a specific, predictable average number. It won't be zero (because you didn't write it), but it will be a stable number that represents the "distance" between your style and Chef B's style.

3. The "Fingerprint" of Length

The paper proves a beautiful mathematical fact: The longer the text, the clearer the fingerprint.

Short Text (e.g., a tweet): It's like looking at a single fingerprint smudge. It's hard to be 100% sure who it belongs to.
Long Text (e.g., an essay): It's like looking at a full handprint with all the lines and ridges. The math shows that as the text gets longer, the "Surprise Meter" settles into its pattern so quickly that you can tell with near-perfect certainty who wrote it.

The authors proved that the chance of making a mistake drops exponentially as the text gets longer. This means if you double the length of the text, the error rate doesn't just go down a little; it crashes down to almost zero.

4. The "Black Box" Problem

Sometimes, you don't have access to the AI's internal "Surprise Meter" (the probability numbers). You only have the text it produced. This is called a "Black Box."

The authors showed that even in this case, you can play a game of "Guess the Probability." You ask the AI to generate the next word 100 times and see what it picks. By doing this, you can build a "fake" version of the Surprise Meter that is almost as good as the real one. They calculated exactly how many times you need to ask the AI to get a reliable answer.

Why This Matters

No Training Needed: Unlike other methods that need to be "taught" with thousands of examples (which can be biased or outdated), this method works instantly using math.
Fairness: It guarantees a very low rate of falsely accusing innocent people. In a classroom, this means fewer students get punished for things they didn't do.
Short Texts: It works surprisingly well even on short texts (like 50-100 words), which is where most other detectors fail.

The Bottom Line

Think of this method as a mathematical lie detector for AI. Instead of guessing based on vague patterns, it measures the "statistical heartbeat" of the text. If the heartbeat matches the AI's own rhythm, it's likely the AI. If it matches a different rhythm, it's likely someone else. And the longer the text, the louder and clearer that heartbeat becomes.

1. Problem Statement

The paper addresses the critical challenge of LLM text attribution and provenance verification. As Large Language Models (LLMs) generate text indistinguishable from human writing, organizations (educational institutions, corporations, social media platforms) struggle to:

Assign responsibility for content (e.g., detecting academic dishonesty).
Enforce compliance with internal policies (e.g., ensuring only sanctioned in-house LLMs are used).
Prevent "model collapse" by filtering AI-generated content from training data.

The Core Gap: Existing detection methods suffer from two main limitations:

Lack of Theoretical Guarantees: Most methods are heuristic or rely on supervised learning, offering no user-controllable bounds on False Positive Rates (FPR). High FPRs lead to severe consequences (e.g., punishing innocent students).
Poor Performance on Short Texts: Current methods lack finite-sample guarantees for short texts (e.g., tweets, chat messages), where statistical signals are weak.

The authors propose a training-free, zero-shot statistical method that guarantees a low false positive rate and provides exponential error decay as text length increases, applicable even to short strings.

2. Methodology

The proposed method models LLM text generation as a sequential stochastic process with complete dependence on history (auto-regressive), rather than assuming independent and identically distributed (i.i.d.) tokens.

Core Statistical Concepts

The method relies on the convergence of two statistical properties:

Log-Perplexity ( $l_A$ ): The average negative log-likelihood assigned by an evaluator model $A$ to a text string $Y_N$ . It measures how "surprising" the text is to model $A$ .
Average Cross-Entropy ( $h_N(B, A)$ ): The expected log-perplexity of text generated by a source $B$ when evaluated by model $A$ .

The Key Insight:

If text is generated by model $A$ , the log-perplexity $l_A(Y_N)$ converges to the entropy of $A$ ( $h_N(A, A)$ ).
If text is generated by a different model $B$ , the log-perplexity $l_A(Y_N)$ converges to the cross-entropy between $B$ and $A$ ( $h_N(B, A)$ ).
The difference between these values serves as a detection signal.

Technical Innovation: Martingale Concentration

Since LLM tokens are dependent (non-i.i.d.), standard concentration inequalities (like Hoeffding's) do not apply. The authors' primary technical contribution is:

Defining per-token random variables such that the deviation between log-perplexity and its target (entropy/cross-entropy) forms a martingale difference sequence.
Applying concentration inequalities for martingales to derive non-asymptotic bounds.
Result: They prove that the probability of the log-perplexity deviating from its expected value decays exponentially with the text length $N$ .

Testing Frameworks

The paper defines two hypothesis testing scenarios:

Attribution (Set vs. Set): Distinguishing whether text comes from a set of non-sanctioned models ( $\mathcal{A}$ ) or a set of in-house models ( $\mathcal{B}$ ).
Detection (Model vs. Not-Model): Distinguishing whether text was generated by a specific known model $A$ or by any unknown source (human or other LLM).

White-Box vs. Black-Box

White-Box: Assumes full access to the evaluator model's conditional probability distributions ( $p_A$ ).
Black-Box: Assumes only sampling access (API). The authors derive the minimum sample size required to approximate the true probabilities sufficiently well to maintain the same theoretical error bounds as the white-box setting.

3. Key Contributions

First Zero-Shot Statistical Tests with Finite-Sample Guarantees:
The paper provides the first theoretical framework for LLM attribution that offers explicit, non-asymptotic upper bounds on Type I (False Positive) and Type II (False Negative) errors for finite-length texts.
Exponential Error Decay:
The authors prove that both Type I and Type II errors decrease exponentially with text length ( $N$ ). This means reliability improves rapidly even for short texts (e.g., ~100 tokens).
Martingale Concentration for Dependent Sequences:
They generalize existing concentration inequalities (previously limited to i.i.d. settings) to the dependent token sequences characteristic of auto-regressive LLMs. This is achieved by modeling the deviation as a martingale.
Information-Theoretic Lower Bound:
The paper establishes a lower bound showing that no detector can achieve error decay faster than exponential in text length, proving that their method is near-optimal.
Robustness to Adversarial Attacks:
Empirical results demonstrate that the method remains robust against various adversarial post-editing attacks (e.g., synonym swapping, paraphrasing, whitespace addition) that typically fool other detectors.

4. Experimental Results

The authors validated their theory using datasets like SQuAD (fact-based), XSum (news summaries), and Reddit WritingPrompts (creative writing), along with the RAID benchmark (6M+ texts with adversarial attacks).

Attribution Performance: In distinguishing between models (e.g., LLaMA2 vs. LLaMA3), the method achieved very low False Positive Rates (often 0–2%) while maintaining high True Positive Rates.
Detection Performance (A vs. Not-A):
- On fact-heavy datasets (SQuAD), the method outperformed state-of-the-art zero-shot baselines (like Binoculars, DetectGPT) significantly, especially when maintaining a strict 1% False Positive Rate.
- The method showed superior robustness to decoding strategies (greedy vs. sampling) compared to likelihood-only baselines.
Adversarial Robustness: In the RAID benchmark, the method maintained high detection rates (TPR) across various attacks (e.g., synonym swaps, paraphrasing) where other detectors (like GLTR or standard neural classifiers) saw significant performance drops.
Short Text Convergence: Experiments confirmed that log-perplexity converges to the theoretical cross-entropy target with as few as 100 tokens, validating the applicability of the method to short messages.

5. Significance and Impact

Regulatory Compliance: The ability to guarantee a user-specified, extremely low False Positive Rate makes this method suitable for high-stakes regulatory environments (e.g., academic integrity, legal compliance) where false accusations are unacceptable.
No Training Required: Being a "training-free" zero-shot method, it avoids the data privacy issues, generalization failures, and computational costs associated with training neural classifiers.
Theoretical Rigor: By moving from heuristic detection to rigorous statistical hypothesis testing with proven bounds, the paper sets a new standard for the reliability of AI detection tools.
Practical Applicability: The extension to black-box settings (via sampling) ensures the method is applicable to closed-weight commercial models (e.g., GPT-4, Claude) where log-probabilities are not directly accessible.

In summary, this paper provides a mathematically grounded, robust, and training-free solution for verifying the origin of AI-generated text, solving the critical problem of false positives in short-text scenarios through advanced statistical concentration inequalities.