💬 NLP

Discourse Features Enhance Detection of Document-Level Machine-Generated Content

This paper addresses the limitations of existing machine-generated content detectors in handling long and paraphrased texts by introducing new datasets and a novel model, DTransformer, which leverages discourse analysis to achieve significant performance improvements over state-of-the-art approaches.

Original authors: Yupei Li, Manuel Milling, Lucia Specia, Björn W. Schuller

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yupei Li, Manuel Milling, Lucia Specia, Björn W. Schuller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to spot a student who copied an essay from a friend. In the past, it was easy: the copy-paster left obvious typos or used words the student never knew. But today, with powerful AI tools (like the ones behind "Chatbots"), the "copy-pasters" are getting very good at rewriting the essay. They change the words, shuffle the sentences, and make it sound just as smooth as the original. It's like a master forger who doesn't just copy a painting; they repaint it with a different brush, making it look brand new.

This paper is about building a better "detective" to catch these AI forgers, especially when they are dealing with long documents like essays or stories.

The Problem: The "Surface-Level" Trap

Current detectors are like security guards who only check a person's face. They look at the surface: "Does this sentence sound weird? Is the vocabulary too fancy?"

The Flaw: AI is great at mimicking human faces. If you ask an AI to rewrite a paragraph, it keeps the meaning the same but changes the "face" (the specific words).
The Result: The old detectors get fooled. They see a smooth, readable text and say, "This looks human!" even though it was written by a machine. This is especially true for long texts where the AI might lose track of the big picture, but the surface words still look perfect.

The Solution: Looking at the "Skeleton"

The authors of this paper realized that while AI can change the "skin" (the words), it struggles to keep the "skeleton" (the structure) exactly like a human's.

Think of a human writer as an architect who builds a house with a clear blueprint. They know exactly how the kitchen connects to the living room, and how the story flows from the beginning to the end.

Human Writing: Has a natural "flow" or "discourse." Sentences connect logically, like a chain. One idea leads to the next in a way that feels organic.
AI Rewriting: Often acts like a robot that rearranges furniture. It might move the sofa to the kitchen and the TV to the bedroom. The rooms are still there, but the flow of the house feels slightly off. The connections between sentences might be a bit clunky or follow a different pattern than a human would naturally choose.

The New Detective: "DTransformer"

The authors built a new model called DTransformer. Instead of just looking at the words (the skin), this model looks at the blueprint (the structure).

How it works: It breaks the text down into sentences and asks, "How does this sentence relate to the one before it?"
- Does it add a detail? (Like adding a brick to a wall).
- Does it give a cause and effect? (Like "Because it rained, the grass is wet").
- Does it give an example? (Like "For instance...").
The Training: They taught this model using a special map called PDTB (Penn Discourse Treebank). Think of this map as a guidebook that teaches the model to recognize the "glue" that holds sentences together.
The Result: The model learned that humans and machines use this "glue" differently. Even if the words are changed, the way the ideas are connected gives the AI away.

The New Test Cases (The Datasets)

To prove their detective was good, the authors created two new "training grounds" (datasets) where the AI forgers were very tricky:

paraLFQA: Short paragraphs where the AI completely rewrote the structure but kept the meaning.
paraWP: Very long stories where the AI had to keep the same length and story but rewrite it.

They also tested their model against famous commercial detectors (like GPTZero and Copyleaks).

The Scoreboard

The results were like a heavyweight boxing match:

Old Detectors: They got confused. On the tricky long texts, they were barely better than guessing (around 50% accuracy). They couldn't see the structural differences.
DTransformer: It stepped into the ring and knocked them out.
- On the long story dataset, it got 99% accuracy.
- On the tricky rewritten paragraphs, it improved accuracy by 15.5% over the best existing methods.

The Big Takeaway

The paper concludes that machines are like excellent "word processors" but poor "architects." They can change the paint and the furniture, but they struggle to maintain the natural, logical flow of a long story that a human creates.

By teaching computers to look for these structural "blueprints" (discourse features) rather than just the words, we can catch AI-generated content much more effectively, even when it has been heavily rewritten to look human.

In short: Don't just listen to what is being said; look at how the ideas are connected. That's where the AI gives itself away.

Technical Summary: Discourse Features Enhance Detection of Document-Level Machine-Generated Content

Problem Statement
The rapid proliferation of Large Language Models (LLMs) has led to a surge in Machine-Generated Content (MGC), creating significant risks regarding academic plagiarism and misinformation. A critical challenge in detection is distinguishing MGC from Human-Produced Content (HPC), particularly when LLMs are used to paraphrase existing human texts rather than generate them from scratch. Current detection methods predominantly rely on surface-level linguistic features (e.g., perplexity, n-grams) or standard Transformer-based Natural Language Understanding (NLU) models. These approaches face two primary limitations:

Document-Level Complexity: Standard Transformer models (e.g., RoBERTa) struggle to retain information and capture complex structural relationships in long documents due to memory constraints and the nature of self-attention mechanisms.
Paraphrasing Evasion: When MGC is generated by paraphrasing HPC, surface-level differences diminish. Existing detectors often fail to identify implicit structural features, such as discourse relations, which may differ significantly between human and machine writing even when semantic content remains similar.

Methodology
To address these limitations, the authors propose a novel framework comprising new datasets and a specialized model architecture:

Dataset Creation:
- paraLFQA: A corpus of approximately 2,000 paragraph pairs derived from the Long-Form Question Answering (LFQA) dataset. Human-written answers serve as HPC, while MGC is generated using the DIPPER model (a discourse paraphrasing tool) to create sentences with high semantic similarity but altered discourse organization.
- paraWP: A large-scale document-level dataset (over 166,000 training pairs) based on Writing Prompts (WP). HPC stories are paraphrased by GPT-3.5-turbo using specific prompts to maintain narrative consistency while altering the text.
- Existing Benchmarks: The study also utilizes Plagbench, M4 (Multi-domain, Multi-lingual Black-Box Machine-Generated Text Detection), and a manually revised subset of paraWP to test robustness against surface-level word twists.
Model Architecture (DTransformer):
The authors introduce DTransformer, a discourse-aware Transformer-based model designed to integrate structural features with semantic features. The architecture operates as follows:
- Sentence Splitting: Documents are split into individual sentences using NLTK.
- Dual Encoding:
  - Semantic Path: Sentences are processed through a standard Transformer Encoder to extract content features, represented by [CLS] tokens.
  - Discourse Path: Consecutive sentence pairs are processed by a BERT model pre-trained on Penn Discourse Treebank (PDTB) annotated texts. This module outputs "discourse codes" representing the relation between the current sentence and its predecessor (e.g., Expansion, Contingency).
- Integration: The semantic [CLS] tokens and the sequence of discourse codes are fed into a Decoder. The decoder utilizes a modified cross-attention mechanism where the semantic tokens generate queries ( $Q$ ) and the discourse codes generate keys ( $K$ ) and values ( $V$ ).
- Classification: The final layer applies a linear network and softmax to output the probability of the text being MGC or HPC.

Key Contributions

New Datasets: The introduction of paraLFQA and paraWP, which specifically target document-level paraphrased MGC, filling a gap in publicly available resources for this specific detection task.
DTransformer Model: A novel architecture that explicitly incorporates discourse analysis (via PDTB preprocessing) to encode structural features, addressing the inability of standard models to capture document-level organization.
Empirical Validation: Comprehensive evaluation demonstrating that integrating discourse features significantly improves detection accuracy, particularly for long-form and paraphrased texts.

Results
The DTransformer model was evaluated against baseline models (RoBERTa Large) and state-of-the-art (SOTA) commercial tools (GPTZero, Copyleaks, Winston AI) across multiple datasets. Key findings include:

Performance Gains: DTransformer achieved substantial absolute improvements over SOTA approaches:
- 15.5% on paraLFQA (using PDTB 3.0).
- 4% on paraWP.
- 1.5% on the M4 dataset.
Robustness: The model maintained high accuracy (99% on paraWP) even against surface-level word twist attacks, whereas commercial tools like GPTZero and Copyleaks showed significant performance drops or failure (e.g., GPTZero scored 0% on the revised paraWP subset).
Ablation Study: Experiments confirmed that the performance gain is primarily attributable to the inclusion of discourse features. A hierarchical model (HiBERT) without explicit discourse encoding performed near random chance (0.53 accuracy) on the paraWP dataset.
PDTB Version: Models utilizing PDTB 3.0 annotations consistently outperformed those using PDTB 2.0, attributed to the more comprehensive relation types in version 3.0.

Significance
The paper argues that the theoretical basis for this approach lies in the fundamental difference between human and machine text generation at the document level. While humans maintain a stable discourse structure and a clear central theme in long texts, LLMs tend to process text in sequential chunks, often leading to a loss of contextual coherence and unstable discourse organization. By explicitly modeling these structural relationships, DTransformer overcomes the limitations of surface-level analysis. The authors conclude that their approach provides a robust method for detecting document-level paraphrased MGC, offering a significant advancement over current detectors that rely solely on semantic or surface-level linguistic features.