WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia

Technical Summary: WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia

Problem Statement
Wikipedia serves as a foundational dataset for training large language models (LLMs), yet the platform faces growing risks from the proliferation of low-quality machine-generated text (MGT). While existing research has developed detectors for MGT, these benchmarks predominantly evaluate performance on generic, open-ended generation tasks (e.g., "Write an article about X"). The authors argue that this approach misaligns with real-world Wikipedia editing practices, which are typically task-specific, incremental, and context-conditioned. Consequently, detectors trained on generic data may fail to generalize to realistic editorial scenarios, leaving the platform vulnerable to hallucinations, bias, and content integrity issues. Furthermore, prior work often relies on single prompts to generate entire articles, a practice explicitly discouraged by the Wikipedia community.

Methodology
To address this misalignment, the authors introduce WETBench, a multilingual, multi-generator, and task-specific benchmark designed to simulate actual editor workflows. The methodology involves three core components:

1. Dataset Construction: The authors constructed two new corpora covering English, Portuguese, and Vietnamese:

   • WikiPS: A large-scale collection of high-quality Wikipedia paragraphs and lead–infobox–body triplets, filtered to ensure no contamination by MGT (using pre-ChatGPT revisions).
   • mWNC: A multilingual extension of the Wikipedia Neutrality Corpus (WNC), including sentence-level and paragraph-level pairs of biased and neutralized text.

2. Task Definition and Prompt Engineering: The benchmark defines three editing tasks grounded in editor use cases:

   • Paragraph Writing: Generating a new section paragraph from scratch. The authors evaluated prompts ranging from minimal instructions to content-conditioned prompts and Retrieval-Augmented Generation (Naive RAG). Naive RAG was selected as the optimal strategy for generating MGT.
   • Summarisation: Generating a lead section based on article body and infobox. One-shot prompting was determined to be the most effective strategy.
   • Text Style Transfer (TST): Neutralizing biased sentences or paragraphs to align with Wikipedia's Neutral Point of View (NPOV) policy. Five-shot prompting yielded the best results.

3. Experimental Setup:

   • Generators: MGT was generated using four models (GPT-4o mini, Gemini 2.0 Flash, Qwen2.5-7B-Instruct, Mistral-7B-Instruct) across the three languages.
   • Detectors: The benchmark evaluated eight state-of-the-art (SOTA) detectors across three families: training-based (XLM-RoBERTa, mDeBERTa), zero-shot white-box (Binoculars, LLR, FastDetectGPT-WB), and zero-shot black-box (Revise-Detect, GECScore, FastDetectGPT-BB).
   • Scale: The final benchmark comprises 101,940 texts (human-written and machine-generated) across tasks, languages, and generators.

Key Contributions

• WETBench Benchmark: The release of a comprehensive benchmark containing 101,940 task-specific texts, covering three languages with varying resource levels, four generators, and eight detectors.
• New Corpora: The creation and release of WikiPS (paragraphs and lead-content pairs) and mWNC (multilingual neutrality corpus with paragraph-level pairs), which are available for broader AI research beyond MGT detection.
• Empirical Evaluation: A systematic assessment of SOTA detectors in realistic, task-specific editing scenarios, contrasting sharply with prior open-ended generation benchmarks.

Results
The evaluation reveals significant challenges for current detection methods in realistic settings:

• Overall Performance: Detectors struggle significantly compared to results on generic benchmarks. Training-based detectors achieved an average accuracy of 78%, while zero-shot detectors averaged only 58%.
• Task Variance: Detection accuracy varied by task. Summarisation was the easiest to detect (supervised detectors reached ~89% accuracy), followed by Paragraph Writing. Text Style Transfer (TST) proved the most difficult, with accuracies often hovering near random chance for zero-shot methods and only slightly higher for supervised methods.
• Method Variance: Supervised (training-based) detectors consistently outperformed zero-shot methods across all tasks and languages. Among zero-shot methods, black-box detectors (specifically GECScore) generally performed better than white-box methods, though performance varied by language and generator.
• Model and Language Effects: Smaller generators (e.g., Mistral, Qwen) left more detectable traces for training-based models, while larger models (e.g., GPT-4o) were harder to detect. Performance also degraded in low-resource languages (Vietnamese), where some models (e.g., Mistral) failed to follow instructions, creating artifacts that training-based detectors could exploit but zero-shot methods could not.

Significance and Claims
The paper claims that WETBench provides a necessary reality check for the field of MGT detection. The authors assert that existing detectors, while effective on generic tasks, lack the generalisability required for real-world, editor-driven contexts on Wikipedia. The results underscore that:

1. Task Specificity Matters: The textual characteristics of task-specific MGT (e.g., neutralizing a sentence) closely resemble human writing, making them harder to distinguish than open-ended generation.
2. Current Limitations: There is substantial uncertainty regarding the reliability of current automated detection methods in safeguarding Wikipedia's knowledge integrity.
3. Future Direction: The authors advocate for evaluating detectors on data that reflects practical editor workflows rather than synthetic, open-ended generation. They position WETBench as a tool to inform the Wikimedia community and the broader AI research community about the current feasibility and limitations of MGT detection, rather than claiming to have solved the detection problem entirely.

The paper concludes with a modest outlook, noting that while their findings highlight current limitations, the benchmark is designed to be extensible with additional tasks, languages, and generators as the field evolves.