Technical Summary: ARB – A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation

1. Problem Statement

Current AI-text detection benchmarks primarily evaluate detectors on a binary classification task: distinguishing human-written text from text generated directly by Large Language Models (LLMs). While prior research has established that paraphrasing and rewriting can degrade detector performance, a critical gap remains: it is unclear whether performance measured on conventional "Human vs. Direct-LLM" benchmarks predicts detector behavior when human-authored content is rewritten by an LLM.

Standard benchmarks often conflate two distinct factors:

1. Content Origin: Whether the ideas and semantic content originate from a human or an LLM.
2. Linguistic Surface: Whether the final wording is human-written, directly generated by an LLM, or mediated by an LLM through rewriting.

In realistic workflows (e.g., a student drafting an essay and using an LLM to polish it), the content is human-origin, but the surface is LLM-mediated. Existing benchmarks fail to isolate these factors, potentially overestimating the robustness of detectors against human-origin, LLM-polished text.

2. Methodology: The ARB Benchmark

The authors introduce ARB (Authorship-Rewriting Benchmark), a matched four-regime dataset designed to separate content origin from linguistic surface.

Dataset Construction

• Source Material: 1,800 human source texts (600 each from XSum, WritingPrompts, and OpenWebText), stratified by length (150–500 words).
• Generators: Four open-weight instruction-tuned models (Llama-3.2-3B, Qwen2.5-7B, Mistral-7B, Gemma-2-9B).
• Four Matched Regimes: For each source item and generator, four variants are created:
  1. HUMAN: The original human text.
  2. FREE-LLM: Direct generation by the LLM based on a topic extracted from the source (conventional baseline).
  3. H2L (Human-to-LLM): The original human text rewritten by the LLM (simulating human-origin content polished by AI).
  4. LLM2L (LLM-to-LLM): The FREE-LLM output rewritten by the same LLM (simulating a second pass on AI-generated content).

This design creates matched quartets where the only variables are the content origin and the rewriting regime, allowing for paired comparisons within dataset-generator blocks.

Evaluation Protocol

• Detectors: Five detectors representing different families were evaluated:
  • Supervised Encoders: BERT-Defense, RoBERTa-Defense.
  • Robust Supervised: RADAR.
  • Zero-shot Likelihood/Curvature: FastDetectGPT.
  • Zero-shot Contrastive: Binoculars-falcon-7b.
• Metrics:
  • Primary: TPR@1%FPR (True Positive Rate at 1% False Positive Rate). This conservative metric is chosen because false accusations of AI authorship carry significant ethical and practical costs.
  • Secondary: AUROC (Area Under the Receiver Operating Characteristic Curve) to assess global ranking separability.
• Analysis: Performance is measured via block-level paired deltas (comparing H2L and LLM2L against the FREE-LLM baseline) and macro-averaged across dataset-generator blocks.

3. Key Results

The evaluation reveals a significant "benchmark-transfer failure," where performance on direct generation does not translate to human-origin rewriting.

Baseline Performance (Human vs. FREE-LLM)

• Strong Detectors: FastDetectGPT and Binoculars-falcon-7b achieved high recall (91.2% and 93.5% respectively) at 1% FPR. RADAR reached 66.8%.
• Weak Detectors: BERT-Defense and RoBERTa-Defense performed near chance levels (TPR < 2%) despite moderate AUROC scores.

Robustness to Rewriting

• H2L (Human-Origin Rewriting): Strong detectors suffered catastrophic drops in recall when human text was rewritten by an LLM.
  • FastDetectGPT: Dropped from 91.2% to 30.8% (a loss of ~60 percentage points).
  • Binoculars-falcon-7b: Dropped from 93.5% to 15.1% (a loss of ~78 percentage points).
  • RADAR: Dropped from 66.8% to 12.2%.
• LLM2L (LLM-Origin Rewriting): When LLM-generated text was rewritten by the same model, the performance drop was significantly smaller.
  • FastDetectGPT: Dropped to 78.3% (loss of ~13 points).
  • Binoculars-falcon-7b: Dropped to 83.0% (loss of ~10 points).
  • RADAR: Remained nearly stable at 64.6%.

The H2L–LLM2L Gap

The operational gap between H2L and LLM2L is substantial. For the best detectors, LLM2L remains much more detectable than H2L (e.g., a 68-point gap for Binoculars). Textual diagnostics indicate that H2L rewrites involve stronger surface changes (higher edit distance, lower lexical overlap) than LLM2L rewrites, suggesting the degradation is driven by a combination of source origin and transformation strength.

4. Contributions and Significance

The paper claims the following contributions:

1. Conceptual Framing: It reframes paraphrase robustness as a benchmark-transfer problem. It demonstrates that the standard "Human vs. Direct-LLM" benchmark is not a reliable proxy for detecting human-origin text that has been LLM-mediated.
2. Design Innovation: ARB introduces a matched four-regime design that isolates human-origin LLM rewriting (H2L) from LLM-origin second-pass rewriting (LLM2L) using a shared human reference and same-generator controls.
3. Empirical Evidence: The study provides evidence that detectors robust to direct generation (like FastDetectGPT and Binoculars) are not robust to human-origin rewriting. Conversely, the same detectors remain largely robust to LLM-origin rewriting.
4. Diagnostic Insight: The results suggest that content origin is a structural factor in detection sensitivity under LLM-mediated surfaces, distinct from simple surface-level transformation. This aligns with findings from other works (e.g., RAIDAR, MAGRET) that exploit the asymmetry in how LLMs edit human vs. machine text.

5. Implications for Practice

The authors argue that procurement and audit decisions relying solely on standard Human-vs-LLM benchmarks risk substantially overestimating detector robustness for the increasingly common workflow of human drafts revised by AI.

• Evaluation: Detectors should be evaluated specifically on H2L-style conditions before deployment in settings involving human-authored, AI-polished text.
• Reporting: Benchmarks should report performance stratified by regime (H2L vs. LLM2L) and operating point (TPR@1%FPR) rather than relying on aggregate AUROC scores.
• Deployment: Detector scores should be treated as uncertain signals rather than definitive evidence of authorship, particularly when the text may have undergone human-origin rewriting.

The paper concludes that future work should focus on causal isolation of source origin (matching transformation strength) and broader validation across languages and domains.