A Comprehensive Dataset for Human vs. AI Generated Text Detection

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has enabled the generation of highly coherent, contextually relevant text that is increasingly indistinguishable from human writing. This capability poses significant risks regarding:

• Content Authenticity: Difficulty in verifying the source of news and information.
• Misinformation: The potential for AI to generate and scale fake news, particularly in critical domains like elections and public health.
• Detection Gap: Existing research is hindered by a lack of large-scale, diverse, and well-annotated datasets that compare human-authored journalistic content against synthetic outputs from multiple state-of-the-art LLMs under realistic conditions. Most existing datasets rely on synthetic prompts, student essays, or generic web content rather than high-quality professional journalism.

2. Methodology and Dataset Creation

The authors constructed a comprehensive dataset to bridge the gap between real-world journalism and AI generation.

Data Source

• Base Material: Over 2.1 million authentic articles from the New York Times (NYT), spanning from January 1, 2000, to the present.
• Selection: Articles were sampled to ensure diversity in topics and timeframes.

Generation Process

1. Prompt Extraction: The abstract of each selected NYT article was extracted to serve as the prompt.
2. Human Ground Truth: The full text of the original article ("Human Story") was fetched via the article's URL.
3. Synthetic Generation: The extracted abstracts were fed into six state-of-the-art LLMs to generate corresponding synthetic narratives. The models used were:
   • Gemma-2-9b
   • Mistral-7B
   • Qwen-2-72B
   • LLaMA-8B
   • Yi-Large
   • GPT-4-o
4. Structure: The final dataset is tabular, containing 58,502 total samples. Each row includes the original prompt (abstract), the human story, and the six AI-generated variations.

Baseline Methodology (Raidar-inspired)

The authors established a baseline using a rewriting-based approach inspired by the "Raidar" method:

• Concept: LLMs tend to make fewer edits when rewriting text generated by themselves (or similar models) compared to human-written text, which they revise more extensively to fit their own style.
• Procedure:
  1. A fixed rewriting model (e.g., GPT-3.5-Turbo) is prompted to "concise" the input text while keeping all information.
  2. The Levenshtein edit distance is calculated between the original input and the rewritten output.
  3. Classification:
     • Human vs. AI: If the edit distance is below a threshold, the text is classified as AI-generated; otherwise, human.
     • Model Attribution: The input is rewritten by multiple candidate models. The model producing the rewrite with the minimum edit distance is predicted as the generator.

3. Key Contributions

• Dataset Scale and Diversity: A unique dataset of 58,000+ samples combining real-world professional journalism with synthetic counterparts from six distinct, cutting-edge LLMs.
• Real-World Context: Unlike previous datasets based on student essays or synthetic prompts, this dataset uses actual NYT abstracts as prompts, ensuring the generated content is grounded in factual, complex journalistic narratives.
• Dual-Task Benchmark: The dataset supports two critical research tasks:
  1. Binary Classification: Distinguishing human-written text from AI-generated text.
  2. Model Attribution: Identifying which specific LLM generated a piece of text.
• Open Access: The dataset is publicly available on Hugging Face to facilitate reproducibility and further research.

4. Results

The authors evaluated the baseline rewriting method on the two defined tasks:

Analysis of Results:

• The 58.35% accuracy for Task A indicates that distinguishing human from AI text is non-trivial, even with a rewriting-based heuristic, as modern LLMs are becoming increasingly human-like.
• The 8.92% accuracy for Task B (random chance would be ~16.6% for 6 models) highlights the extreme difficulty of attributing text to a specific model, suggesting that current models share significant stylistic and structural overlaps.
• These low baseline scores underscore the need for more sophisticated detection methods beyond simple rewriting heuristics.

5. Significance and Future Work

• Trust and Transparency: The dataset provides a rigorous benchmark for developing tools to verify the authenticity of news, crucial for maintaining trust in media and democratic processes.
• Research Catalyst: It enables the study of cross-model generalization (whether detectors trained on one model work on others) and the evolution of detection techniques as LLMs improve.
• Future Directions:
  • Expanding the dataset to other languages and modalities (multimodal).
  • Investigating advanced detection methods (e.g., watermarking, probability curvature, deep learning classifiers).
  • Studying the impact of in-context learning and fine-tuning on making AI text even more human-like, thereby raising the bar for detection.

In conclusion, this paper addresses a critical infrastructure gap in AI safety research by providing a high-quality, real-world dataset that challenges the community to develop more robust detection and attribution mechanisms for the era of generative AI.