💬 NLP

'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

This paper addresses the lack of consensus on harmful AI text usage by introducing AITDNA, a new benchmark of human-machine co-constructed texts with detailed interaction histories, to demonstrate that current detectors often fail as broad solutions and perform well only for specific definitions of AI-generated content.

Original authors: Nils Dycke, Marina Sakharova, Nico Daheim, Iryna Gurevych

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nils Dycke, Marina Sakharova, Nico Daheim, Iryna Gurevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to grade a stack of essays. You suspect some students used a "magic writing robot" (AI) to help them, but you don't know how they used it. Did the robot write the whole thing? Did it just fix a few sentences? Did the student ask the robot to write a specific paragraph about a topic they weren't allowed to discuss?

This paper argues that right now, everyone is playing a different game with different rules, and that's why the "AI detectors" we have today are often confused and unreliable.

Here is the breakdown of the paper's main points, using simple analogies:

1. The Problem: Everyone is Using a Different Map

The authors say that researchers building AI detectors are all trying to solve the same problem, but they are defining "the problem" in completely different ways.

The "All-or-Nothing" Map: Some researchers think a text is "AI" only if a robot wrote the entire essay from start to finish.
The "Sentence-by-Sentence" Map: Others think a text is "AI" if any single sentence was written by a robot.
The "Polishing" Map: Some think it's only cheating if the robot wrote the ideas, even if a human wrote the words.

The Analogy: Imagine trying to find a specific type of fish in a lake. One group of fishermen is looking for any fish. Another group is only looking for blue fish. A third group is only looking for fish caught with a net. If they all report back, "We found fish!" but they are using different definitions, their reports don't match. The paper says we need to agree on what "fish" (or "AI text") actually means before we can catch it.

2. The New Tool: A "Black Box" Writing Lab

To fix this confusion, the researchers built a new dataset called AITDNA.

The Old Way: Most previous datasets were like "fake news." Researchers took a human essay and told a robot to rewrite it, or took a robot essay and asked a human to fix it. It was a staged, artificial scenario.
The New Way (AITDNA): The researchers set up a real-world "writing lab." They hired 99 people to write essays while sitting next to a robot. They recorded everything: every time the human typed, every time they asked the robot for help, every time they deleted a robot's suggestion, and every prompt they typed.

The Analogy: Previous datasets were like watching a movie where actors pretend to cook a meal. You know it's fake because they aren't actually chopping vegetables. The new dataset is like putting a camera in a real kitchen and watching a chef and a sous-chef actually cook together, recording every chop, stir, and taste test. This gives a true picture of how humans and AI actually work together.

3. The Discovery: Detectors are "Specialists," Not "Generalists"

The researchers tested various AI detectors on their new, realistic data. They found a surprising truth: There is no single "super detector" that works for everything.

The "Big Picture" Detector: Some detectors are great at spotting if a whole essay was written by a robot. They are like a security guard who can tell if a whole building is empty or full.
The "Micro" Detector: Other detectors are better at finding just one sentence written by a robot. They are like a detective looking for a single fingerprint.
The Failure: When you ask a "Big Picture" detector to find a single sentence, it fails. When you ask a "Micro" detector to judge a whole essay, it gets confused.

The Analogy: It's like trying to use a telescope to read a street sign. The telescope is amazing for seeing distant stars (detecting whole AI essays), but it's terrible for reading the small text on a sign (detecting a single AI sentence). The paper found that detectors are usually very good at the specific job they were trained for, but terrible at any other job.

4. The New Rules: "Content" and "Intent"

The paper also suggests we need new ways to define "AI text" that match real-life rules.

Current Rule: "If a robot touched it, it's AI."
Proposed New Rules:
- Content-Based: "It's only AI if the robot wrote the ideas (like a literature review), even if a human wrote the words."
- Intent-Based: "It's only AI if the robot was asked to write something forbidden (like cheating on a test), even if it was just polishing a sentence."

The Analogy: Imagine a school rule: "You can't use a calculator."

Old Definition: If you used a calculator for any math, you failed.
New Definition: If you used a calculator to solve the problem, you failed. But if you used a calculator just to check your addition, that's fine. The paper argues we need detectors that understand the intent (the "why") and the content (the "what"), not just the "who."

5. The Conclusion: Stop Guessing, Start Specifying

The main takeaway is that we cannot compare different AI detectors unless we agree on the rules of the game.

If you want to catch a student who let a robot write their whole essay, use a "Document-Level" detector.
If you want to catch a student who used a robot to write a specific paragraph, use a "Sentence-Level" detector.
If you want to catch a student who used a robot to write a forbidden topic, use a "Content-Based" detector.

The Final Metaphor: The paper is like a mechanic telling you, "You can't use a hammer to fix a screw, and you can't use a screwdriver to fix a nail. Stop complaining that your tools aren't working; you just need to pick the right tool for the specific job you are trying to do."

The researchers released their new "real-world" data and their list of definitions so that everyone can finally speak the same language and build better, more honest tools.

Technical Summary: 'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

Problem Statement

While AI-generated text detection (AITD) is widely recognized as a critical technology for mitigating societal risks—such as weakened scientific quality control, academic dishonesty, and misinformation campaigns—there is no consensus in the literature on what constitutes "harmful" AI use. Existing datasets and evaluation approaches often rely on implicit, simplistic assumptions that misalign with real-world human-AI co-creation patterns. Specifically, current benchmarks frequently treat texts as either entirely human or entirely AI-generated (en-bloc), or they assume fixed boundaries between human and AI passages. These synthetic constructions fail to capture the nuanced, iterative nature of human-AI collaboration, where authors and Large Language Models (LLMs) interact through prompts, revisions, and continuations. Consequently, existing notions of AITD are often under-specified, making findings across different datasets incompatible and hindering the development of detectors that generalize to realistic use cases.

Methodology

The authors address these gaps through a three-pronged methodology:

Conceptual Framework: The paper establishes a shared vocabulary to define AITD notions along three dimensions:
- Normative Standard: Distinguishes acceptable from unacceptable AI use (e.g., total prohibition vs. permissible polishing).
- Genesis Assumptions: Constraints on how text is created (e.g., en-bloc generation vs. mixed spans).
- Attacker Model: The degree to which genesis assumptions are violated (e.g., replacing tokens post-hoc).
  Based on this, the authors categorize existing notions into Genesis-based (Document-level, Boundary-level, Sentence-level) and Population-based (Membership-based). They further propose two new notions to cover practical gaps: Content-based AITD (flagging AI sentences violating content policies) and Authorship-ID-based AITD (flagging AI text that does not match a specific human author's style).
Dataset Construction (AITDNA): To test these notions against realistic data, the authors collected AITDNA, a new benchmark of 352 human-machine co-constructed texts from 99 participants.
- Collection Process: Participants used a custom interface based on CARE to co-write texts in four conditions: human-only, and three co-creation modes (Argumentative, Creative, Explanatory) using various LLMs (including Llama-4, Qwen2.5, DeepSeek-V3.2, GPT-5.2, and Gemini-3).
- Granularity: The interface recorded detailed interaction traces, including user prompts, LLM responses, and specific edit histories (insertions/deletions). This allows for the precise derivation of "genesis" (the origin of every token: Human, AI, or Mixed).
- Scope: The dataset covers multiple genres and LLM configurations, resulting in an average of ~50% AI tokens per text with 23.33 boundaries between human and AI spans, reflecting high intermixing.
Evaluation Protocol: The authors benchmarked eight detectors (six open-source, two proprietary) across the new AITDNA dataset and five existing datasets (CoAuthor, SenDetEx, BD, Mixset, DetectRL).
- Controlled Variables: They systematically varied notion parameters, specifically the threshold $\tau$ (the minimum ratio of AI tokens required to label a span as AI) and the segmentation granularity (document, boundary, sentence).
- Metrics: Performance was measured using AUROC, F1-Score, and False Positive Rate (FPR).

Key Contributions

C#1: AITD Notion Framework: The authors derive a formal framework for specifying AITD problems, identifying five common notions and introducing two new ones (Content-based and Authorship-ID-based) to better align with real-world policies.
C#2: The AITNA Dataset: They release AITDNA, the first dataset capturing detailed human-AI interaction histories (prompts, edits, and genesis) in a realistic co-writing setting, serving as a notion-agnostic reference point.
C#3: Empirical Analysis of Assumptions: The study systematically demonstrates that existing datasets encode unrealistic assumptions (e.g., low mixing ratios, en-bloc generation) that do not reflect natural co-creation. They show that detector performance is highly sensitive to the underlying notion and its parameters.

Results

Misalignment of Existing Data: Analysis reveals that synthetic datasets (like DetectRL and Mixset) often assume "en-bloc" generation where AI rewrites entire documents, resulting in low boundary counts and high AI token ratios (~~65%). In contrast, AITDNA shows a more balanced mix (~~51% AI tokens) with significantly higher boundary counts (23.33), indicating that synthetic data fails to model the granular intermixing of real co-writing.
Notion Difficulty: Detection performance varies significantly by notion. Document-level AITD yields the highest performance, while fine-grained notions (Sentence-level, Intent-based, Content-based) are substantially harder. Commercial detectors (GPTZero, Pangram) performed poorly on content/intent-based tasks, suggesting they overfit to surface-level features rather than semantic content or prompt intent.
Parameter Sensitivity: The choice of the threshold parameter $\tau$ drastically alters evaluation outcomes. For instance, setting $\tau = 0.5$ (majority AI) versus $\tau = 0.1$ (any AI) leads to F1-score variations of up to 0.8 across datasets. This indicates that without explicit, consistent parameter specification, cross-dataset comparisons are invalid.
No Universal Detector: No single detector consistently outperforms others across all datasets and notions. Performance rankings shift depending on the dataset's underlying assumptions and the specific notion being tested. For example, moBERT excelled on DetectRL (likely due to training data overlap) but not on AITDNA.

Significance and Claims

The paper argues that the field of AITD must move beyond synthetic, notion-agnostic benchmarks that obscure the complexity of human-AI collaboration. The authors claim that:

Standardization is Critical: Evaluation results are only comparable if the underlying notions (segmentation, normative standards, and parameters like $\tau$ ) are explicitly defined and aligned.
Realism Matters: Detectors trained or evaluated on synthetic, low-mixing datasets may fail in real-world scenarios where human and AI text are deeply interwoven.
Policy Alignment: Future detectors must be designed to support nuanced policies (e.g., distinguishing between permissible polishing and prohibited full-generation) rather than binary "human vs. AI" labels.

The authors conclude that while reliable detection remains feasible, the current lack of standardized definitions and realistic data hinders progress. They call for future work to focus on robust detectors that explicitly account for these varied assumptions and to collect dedicated datasets for under-explored notions like authorship identification.