💬 NLP

Who Wrote the Book? Detecting and Attributing LLM Ghostwriters

This paper introduces GhostWriteBench, a dataset of long-form texts generated by frontier LLMs for testing authorship attribution, and proposes TRACE, an interpretable and lightweight fingerprinting method that achieves state-of-the-art performance in detecting and attributing LLM ghostwriters across diverse and out-of-distribution scenarios.

Original authors: Anudeex Shetty, Qiongkai Xu, Olga Ohrimenko, Jey Han Lau

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anudeex Shetty, Qiongkai Xu, Olga Ohrimenko, Jey Han Lau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a massive library where someone has secretly replaced thousands of books with perfect forgeries. These aren't just any forgeries; they were written by "ghostwriters" who are actually super-smart AI robots. The problem? These AI robots are getting so good that the books look, feel, and read exactly like they were written by humans.

This paper is like a team of digital detectives stepping in to solve the mystery: "Who actually wrote this book?"

Here is the story of their investigation, broken down into simple parts:

1. The Problem: The "Fake Book" Epidemic

In the past, if you wanted to catch a fake, you might look for typos or weird grammar. But modern AI (Large Language Models, or LLMs) writes so smoothly that those tricks don't work anymore.

Worse, most previous detective tools were like flashlights that only work in the dark. They could tell you "This is AI" vs. "This is Human," but they couldn't tell you which specific AI wrote it. If an AI wrote a book about space, and then you showed it a book about cooking, the old tools would get confused and fail. They also mostly looked at short sentences, not full-length novels.

2. The New Evidence Locker: GHOSTWRITEBENCH

To catch these digital forgers, the researchers built a massive new evidence locker called GHOSTWRITEBENCH.

The Size: Instead of looking at short tweets or paragraphs, they gathered over 300 full-length books (each about 50,000 words long). That's like reading a whole library of novels!
The Variety: They used 10 different "top-tier" AI models (the current champions of AI writing) to write these books.
The Trap: They set up a tricky test. They asked the detectives to identify the author even when the AI was writing about a topic it had never seen before (like an AI trained on sci-fi suddenly writing a history book) or when the AI was one they had never met before.

3. The Detective Tool: TRACE

The researchers didn't just build a database; they invented a new super-tool called TRACE.

Think of every author (human or AI) as having a unique digital fingerprint.

Old Detectors tried to guess the author by looking at what words were used (like counting how many times the word "the" appeared).
TRACE looks at how the author thinks.

The Analogy: Imagine two people walking through a forest.

Person A always steps on the left foot, then the right, then pauses.
Person B skips, then hops, then steps twice.
Even if they both wear the same shoes and walk the same path, their rhythm is different.

TRACE listens to the "rhythm" of the text. It doesn't care about the words themselves; it cares about the transition between them. It asks: "When this AI chose the word 'cat', what was the probability it was thinking of 'dog' next? And how did that probability change for the next word?"

It creates a heat map (a colorful 2D picture) of these transitions. Just like a human fingerprint has ridges and loops, every AI model creates a unique, messy, colorful pattern that is nearly impossible to fake.

4. Why TRACE is a Game-Changer

The researchers tested TRACE against all the other "flashlights" and found it was the only one that could handle the tricky situations:

The "Unseen Author" Test: They showed TRACE a book written by an AI it had never met before. While other tools got lost, TRACE said, "I don't know exactly who this is, but I know it's not the ones I've seen before." It's great at spotting the unknown.
The "Low Data" Test: Usually, detectives need a huge library of samples to learn a suspect's style. TRACE only needed a few pages (3 to 5 books) to learn the "rhythm" of an AI and then spot it in a whole novel.
The "Black Box" Test: TRACE doesn't need to see the AI's internal code (which is often secret). It just reads the finished book, like a detective reading a letter to find the writer's style.

5. The Verdict

The paper concludes that TRACE is the new gold standard. It's lightweight (doesn't need a supercomputer to run), interpretable (we can actually see the patterns it found), and robust (it works even when the AI tries to hide by changing topics or genres).

In short: If AI ghostwriters are trying to sneak into the library and steal the author's credit, TRACE is the security camera that doesn't just look at the face, but listens to the unique heartbeat of the writing to catch them red-handed.

1. Problem Statement

The paper addresses the emerging threat of LLM ghostwriting, where frontier Large Language Models (LLMs) generate low-quality, deceptive books to mimic human authors. Current detection and attribution methods face three critical limitations:

Task Framing: Most existing datasets focus on binary classification (Human vs. AI) rather than Authorship Attribution (AA), which requires identifying the specific LLM model that generated the text.
Generalization: Existing methods struggle with Out-of-Distribution (OOD) scenarios, failing to generalize across unseen domains (genres) or unseen LLM authors.
Text Length: Most benchmarks focus on short texts (<500 words), whereas ghostwriting primarily affects **long-form documents** (books, often >50k words), which pose unique challenges for transformer-based models due to context window limits.

2. Methodology

A. GHOSTWRITEBENCH Dataset

The authors introduce GHOSTWRITEBENCH, a novel dataset designed specifically for long-form LLM authorship attribution.

Scale & Content: Contains 325 books (both fiction and non-fiction) generated by 10 frontier LLMs (e.g., GPT-4.1, GPT-5.1, Claude Sonnet, Kimi K2, Qwen3-Max) and 100 human-authored books from Project Gutenberg.
Generation Pipeline: Mimics human writing via a hierarchical process: outlining, summarization, and incremental long-text expansion, using human books as few-shot references to ensure narrative coherence.
OOD Dimensions:
- OOD-Domain: Tests generalization to unseen genres (e.g., training on History, testing on Science).
- OOD-Author: Tests generalization to unseen LLMs (leave-one-out evaluation where one model is held out for testing).
Quality Control: The generated books undergo rigorous cleaning (removing markers) and quality analysis (Perplexity, Self-BLEU, N-gram diversity) to ensure they are comparable to human texts, making the attribution task difficult.

B. TRACE (Transition-based Representation for Authorship Classification and Evaluation)

The authors propose TRACE, a lightweight, interpretable, and white-box-free fingerprinting method.

Core Concept: Instead of analyzing static text features, TRACE models the transitions of token-level patterns (specifically Word Rank and Entropy) as estimated by a separate, lightweight evaluator language model (GPT-2).
Fingerprint Construction:
1. Token Scoring: For a text sequence, an evaluator model computes the rank or entropy of each token based on its context.
2. Transition Modeling: The method captures the transition between consecutive token scores (e.g., $rank_{i-1} \to rank_i$ ).
3. Compression:
  - Rank-based: Counts of rank transitions are binned into equal-probability clusters to create a compressed $K \times K$ matrix.
  - Entropy-based: Continuous entropy transitions are smoothed using Kernel Density Estimation (KDE) to create a $G \times G$ density map.
Attribution: A test text's fingerprint is compared against reference fingerprints of known LLMs using similarity metrics (Jensen-Shannon distance for ranks; Norm-Mean or JS distance for entropy). The closest match determines the author.

3. Key Contributions

GHOSTWRITEBENCH: The first benchmark specifically for long-form LLM authorship attribution that explicitly evaluates generalization across domains and unseen authors.
TRACE: A novel, model-agnostic attribution technique that:
- Requires no white-box access (works on closed-source models).
- Is lightweight (uses a small evaluator model).
- Is interpretable (produces 2D heatmaps reflecting writing styles).
- Requires minimal training data (few-shot capability).
Comprehensive Evaluation: Extensive experiments covering in-distribution, OOD, and low-resource scenarios.

4. Experimental Results

Experiments were conducted on GHOSTWRITEBENCH comparing TRACE against baselines (N-GRAM, BERT-AA, DETECTIVE, GLTR, etc.).

Performance:
- In-Distribution (ID): TRACE achieves state-of-the-art performance, particularly the Entropy-Norm variant (Macro-F1: 0.96 in high-resource settings).
- OOD-Domain: TRACE significantly outperforms baselines, maintaining robustness when testing on unseen genres (e.g., 0.82 F1 vs. 0.90 for baselines in high-resource, but baselines drop sharply in low-resource).
- OOD-Author: TRACE demonstrates superior generalization to unseen LLMs. While baselines often confuse models within the same family (e.g., GPT-4 vs. GPT-5), TRACE maintains distinct fingerprints.
- Low-Resource: TRACE excels in scenarios with limited training data (3–5 books per author), where supervised baselines fail.
Human vs. LLM: In an OOD-Author variant testing unseen human authors, the entropy-based TRACE variants achieved perfect separation (1.00 F1), indicating human and LLM fingerprints are fundamentally distinct.
Ablation Studies:
- Evaluator Model: TRACE is robust to the choice of the evaluator model (GPT-2, OLMo, Gemma), showing consistent performance.
- Fingerprint Size: Optimal performance is found at specific grid sizes (K=50 for ranks, G=50 for entropy).
- Token Count: Entropy-based fingerprints converge at ~10k tokens, while rank-based require ~50k, suggesting TRACE works well even for shorter documents.

5. Significance and Conclusion

This paper marks a significant shift in AI text forensics by moving from short-text binary detection to long-form, multi-class attribution.

Practical Impact: It provides a tool to detect "sham" books and hold specific LLMs accountable for generated content, addressing a growing issue in the publishing industry.
Methodological Advancement: TRACE proves that token-level transition patterns are a more robust and generalizable signal for authorship than static features or heavy supervised fine-tuning.
Future Directions: The authors note limitations regarding language diversity (English only) and the need to study resilience against obfuscation attacks and multi-authorship scenarios.

In summary, the paper establishes a new standard for evaluating LLM attribution in realistic, long-form, and open-set scenarios, offering a lightweight and highly effective solution (TRACE) that outperforms current state-of-the-art methods.