📊 statistics

A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering

This paper introduces a distribution-free framework that leverages knockoff filtering to convert arbitrary rewrite-based detectors into tools with finite-sample false discovery rate guarantees for identifying LLM-generated text, without requiring model retraining.

Original authors: Yi Liu

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to spot which essays in a pile were written by students and which were secretly written by an AI. You have a tool that can "rewrite" any essay. If the tool rewrites an AI essay, it looks very similar to the original. But if it rewrites a human essay, the rewrite looks quite different because the AI is trying to force a human style into its own robotic patterns.

This paper proposes a new, clever way to use that rewriting tool to catch AI text without needing to train a new, complex AI detector from scratch. Here is how it works, using simple analogies:

1. The Problem: The "Guessing Game" is Risky

Currently, many detectors give you a score (like a "suspicion meter"). If the score is high, you flag the text as AI. But there's a big problem: if you have 1,000 essays, and you set your "suspicion meter" too low, you might accuse 100 innocent students of cheating. You don't just want to catch the cheaters; you want to make sure you aren't falsely accusing too many good students.

2. The Solution: The "Knockoff" Trick

The authors realized that the act of rewriting text creates a perfect setup for a statistical trick called Knockoff Filtering.

Think of it like a magic mirror:

The Original Text: You have an essay (let's call it "Alice").
The Knockoff: You ask an AI to rewrite "Alice." This rewrite is the "Knockoff."

The Magic Rule:

If "Alice" was actually written by an AI, the original and the rewrite are like identical twins. They are so similar that you can't tell which is which. They are "exchangeable."
If "Alice" was written by a human, the rewrite is like a bad impersonator. The AI tries to mimic the human, but the result looks weird or different. The original and the rewrite are not twins; they are distinct.

3. How the Framework Works (The Three Steps)

The paper suggests a simple three-step process that can be added to any existing rewriting detector:

Create the Mirror: For every text you want to check, generate a rewrite (the Knockoff).
Compare the Twins: Measure how different the original is from the rewrite.
- If they are very different, it's likely a Human (because the AI struggled to mimic them).
- If they are very similar, it's likely AI (because the AI just rewrote its own work).
The "Fairness" Filter: This is the paper's big innovation. Instead of just picking a random "cutoff line" to decide who is guilty, the system looks at the whole group of essays at once. It asks: "If I flag these people as AI, how many innocent people (humans) will I accidentally catch?"

The system automatically adjusts the "cutoff line" to guarantee that, for example, no more than 20% of the people you accuse are actually innocent. This is called False Discovery Rate (FDR) control.

4. Why This is a Big Deal

No New Training Needed: You don't need to build a new, expensive AI model. You can take any existing tool that rewrites text and "plug in" this statistical framework to make it fair and reliable.
Works Everywhere: The authors tested this on 19 different topics (from sports to religious texts) and four different AI models. It worked consistently.
The "Symmetry" Check: The math behind this relies on a rule called "symmetry." It basically means: "If the text is AI, the difference between the original and the rewrite should be equally likely to be positive or negative." The paper shows that while real-world AI isn't perfect, it's close enough that this trick works reliably, especially if you do a small "calibration" step to fix minor biases.

5. The Catch (Limitations)

The paper is honest about its limits:

It's a Calibration Layer, Not a Detector: This framework doesn't find the AI text by itself; it just makes sure the detector you already have doesn't make too many mistakes.
Garbage In, Garbage Out: If your underlying rewriting tool is terrible and can't tell the difference between human and AI text at all, this framework can't magically fix it. It can only control the rate of mistakes, not create power out of thin air.
Imperfect Symmetry: In the real world, AI isn't perfectly symmetrical. The authors had to add a "mean-correction" step (like leveling a wobbly table) to make the math work perfectly. If they don't do this, the system might be slightly too strict or too lenient.

Summary

Think of this paper as a quality control inspector for AI detectors. It takes a detector that might be "reckless" (accusing too many people) and puts a safety harness on it. It ensures that when you say, "This text is AI," you can be mathematically confident that you aren't falsely accusing too many humans, all without needing to retrain the detector or change how it works.

Technical Summary: A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering

Problem Statement

The detection of Large Language Model (LLM)-generated text is a critical challenge, particularly when screening large corpora (e.g., academic assignments, reviews, web pages) rather than classifying isolated documents. Existing approaches generally fall into two categories: logit-based methods (analyzing token probabilities) and rewrite-based methods (comparing original text against an LLM-generated rewrite).

A significant limitation of current deployment is the lack of rigorous statistical guarantees regarding False Discovery Rate (FDR). A detector with high AUROC may still produce an unacceptably high number of false accusations when applied at scale. Furthermore, many existing methods require retraining whenever the underlying rewrite model changes, making them brittle and costly. There is a need for a framework that provides finite-sample FDR guarantees without requiring retraining of the base detector.

Methodology

The authors propose a distribution-free statistical framework that reformulates rewrite-based detection as a multiple hypothesis testing problem using the Knockoff Filter (Candès et al., 2018; Barber and Candès, 2015). The core insight is that rewrite-based detection implicitly constructs "knockoff" samples, enabling the application of knockoff filtering theory without training a new classifier.

The framework operates in three steps:

Generate Knockoffs: For a collection of texts $T_1, \dots, T_n$ , an LLM generates a corresponding rewrite $R_i$ for each text.
Compute Statistics: A comparison statistic $s_i = f(T_i, R_i)$ is computed. This statistic measures the distance or asymmetry between the original text and its rewrite. The framework requires the statistic to be antisymmetric (i.e., $f(T_i, R_i) = -f(R_i, T_i)$ ).
Thresholding and Selection: The framework applies a knockoff filtering rule to select texts classified as human-written while controlling the FDR at a user-specified level $q$ . The threshold $\tau$ is determined by:
$\tau = \min \left\{ c \in \mathbb{R}^+ : \frac{\#\{s_i \leq -c\} + 1}{\max\{\#\{s_j \geq c\}, 1\}} \leq q \right\}$
Texts with $s_i > \tau$ are classified as human-written.

Theoretical Foundation

The validity of the framework relies on Assumption 1 (Knockoff Symmetry): Under the null hypothesis (AI-generated text), the distribution of the statistic must be symmetric around zero. Specifically, $P(f(T_i, R_i) < -a | H_i=0) = P(f(T_i, R_i) > a | H_i=0)$ .

The paper establishes that this symmetry holds if the rewrite generation process satisfies a specific exchangeability condition: $P_{LLM}(T_i)K(R_i | T_i) = P_{LLM}(R_i)K(T_i | R_i)$ . While real-world rewriters may not satisfy this condition exactly, the framework includes an empirical diagnostic (the frac+ metric and KS-test) to monitor symmetry. When symmetry is approximate, a mean-correction step (cross-domain transfer) is applied to shift the null distribution before thresholding.

Key Contributions

Implicit Knockoff Construction: The authors demonstrate that rewrite-based detection naturally induces a knockoff structure, allowing the reformulation of detection as a multiple hypothesis testing problem.
Distribution-Free Procedure: They introduce a calibration procedure that converts arbitrary rewrite-based statistics into detectors with provable finite-sample FDR control under a knockoff symmetry condition, requiring no retraining of the base detector.
Theoretical and Empirical Validation: The paper provides theoretical proofs for FDR control under the symmetry assumption and empirical evidence showing that the framework achieves reliable FDR control and meaningful detection power across diverse domains and models.

Experimental Results

The framework was evaluated using three detection statistics (ImBD, L2D, and Likelihood) across 19 domains and four source models (GPT-3.5T, GPT-4o, Gemini, Llama-70B).

FDR Control: The proposed method successfully controlled the empirical FDR close to the target levels ( $q \in \{0.2, 0.3, 0.5\}$ $q \in {0.2, 0.3, 0.5}$ ).
- L2D and IMBD: These methods showed strong performance. At $q=0.2$ , observed FDR ranged from 0.16 to 0.28 (slightly above nominal for some pairs), while detection power was high (e.g., L2D achieved 0.833–0.941 power).
- Likelihood: Even without specific training for detection, the likelihood method achieved substantial power (0.220–0.529) with minimal FDR inflation.
Symmetry Verification: Empirical diagnostics (frac+ close to 0.5 and high KS-test p-values) confirmed that the symmetry assumption holds approximately for AI-generated texts in many cases, particularly after cross-domain mean correction.
Cross-Domain Transfer: Borrowing the null mean from held-out source domains substantially improved detection power, though it introduced mild FDR inflation in some cases, highlighting a trade-off between power and strict FDR control.

Significance and Claims

The paper claims that its primary contribution is a statistical calibration layer rather than a new detector. By separating the design of detection statistics from the control of false discoveries, the framework allows existing rewrite-based detectors to inherit finite-sample FDR guarantees.

The authors emphasize that this approach is:

Deployment-ready: It provides a guarantee on the expected fraction of false accusations among texts classified as human-written.
Robust: It functions effectively across different detectors and models without retraining.
Modest in Scope: The authors acknowledge that the method relies on the approximate validity of the knockoff symmetry condition. If the underlying rewrite statistic is uninformative or the symmetry is severely violated, the framework cannot compensate. Therefore, the method is presented as a calibration tool for rewrite-based detectors, not a standalone solution that replaces the need for high-quality base statistics.