💬 NLP

Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection

This paper introduces OpAI-Bench, a novel benchmark that evaluates AI-text detection across multiple granularities by simulating progressive human-to-AI co-editing workflows, revealing that detectability follows non-monotonic patterns influenced by edit operations, domains, and revision history rather than simply the proportion of AI-generated content.

Original authors: Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tianjun Yao, Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Hao Li, Salman Khan, Zhiqiang Shen

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tianjun Yao, Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Hao Li, Salman Khan, Zhiqiang Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a chef prepare a dish. In the old days, if you wanted to know if a meal was "human-made" or "machine-made," you just tasted the final plate. If it tasted like a robot, you labeled it "AI." If it tasted like a human, you labeled it "human."

But today, cooking is different. A human chef starts the dish, then an AI assistant comes in to chop some vegetables, then the human adds spices, then the AI stirs the sauce, and finally, the human garnishes it. The final dish is a collaboration.

The paper you provided, OpAI-Bench, argues that our current tools for detecting AI writing are like a food critic who only tastes the final dish and tries to guess the entire cooking history based on that one bite. They often get it wrong because they don't understand the process of how the dish was made.

Here is a breakdown of what the paper does, using simple analogies:

1. The Problem: The "Snapshot" vs. The "Movie"

Current AI detectors are like taking a snapshot of a finished painting. They look at the final image and say, "This is 100% human" or "This is 100% AI."

But in the real world, writing is a movie. A human writes a draft, then an AI "polishes" a sentence, then the human "expands" a paragraph, then the AI "compresses" a section. The paper says: We need to watch the whole movie, not just the final frame. Existing tests don't show us the steps in between, so they miss how AI signals appear, disappear, or get hidden during the editing process.

2. The Solution: OpAI-Bench (The "Time-Lapse Camera")

The authors built a new testing ground called OpAI-Bench. Think of this as a time-lapse camera set up to film a human writing a document while an AI edits it step-by-step.

The Setup: They start with a purely human-written document (Version 0).
The Process: They create 9 versions of that document. In each step, the AI edits a specific percentage of the text (from 15% up to 100%).
The Twist: They don't just change the amount of AI text; they change how the AI changes it. They use five different "editing tools":
- Polish: Making it sound smoother (like waxing a car).
- Paraphrase: Rewording it without changing the meaning (like translating a song into another language).
- Style Rewrite: Changing the tone (like turning a formal report into a casual blog post).
- Compress: Shortening it (like summarizing a long story).
- Expand: Adding more details (like adding a subplot to a movie).

Crucially, they keep a detailed map (provenance) showing exactly which words, sentences, and paragraphs were touched by the AI at every single step.

3. The Big Surprise: The "Sweet Spot" of Confusion

The researchers tested many different AI detectors on this new benchmark. They expected that as more AI edits were added, the detectors would get better and better at spotting the AI.

They were wrong.

They found a strange, non-linear pattern. Imagine a rollercoaster:

At the start (Human only): Detectors are confident it's human.
At the end (Fully AI): Detectors are confident it's AI.
In the middle (The Mix): This is where it gets weird. When the document is a mix of human and AI (especially around the 40-50% mark where the AI is "compressing" text), the detectors get confused. They perform worse here than they do on the fully AI version.

The Analogy: It's like trying to spot a fake coin. If a coin is 100% plastic, you know it's fake. If it's 100% gold, you know it's real. But if someone melts 50% plastic into 50% gold and reshapes it, it might look more real than the pure plastic, or it might look so strange that your detector gets dizzy and gives up. The paper calls this a "non-monotonic" pattern—meaning "more AI" doesn't always equal "easier to detect."

4. The "Compression" Trap

One specific finding stood out: Compression (shortening text) was the hardest type of edit for detectors to catch. Even when the AI only changed a small amount of text, if that change involved compressing the content, the detectors often failed to notice it. It's as if the AI was a magician making things disappear, and the detectors were blind to the vanishing act.

5. Why This Matters (According to the Paper)

The paper concludes that we cannot just ask, "Is this text AI?" anymore. We need to ask:

How was it edited?
What kind of edits were made?
When did the edits happen in the history of the document?

The current detectors are like security guards who only check the final ID card. OpAI-Bench shows us that we need guards who can watch the whole security camera footage to see if someone sneaked in, changed their clothes, and then walked out looking different.

In short: The paper introduces a new way to test AI detectors that mimics real-life, step-by-step editing. It proves that detecting AI is much harder when humans and AI work together in the middle of the process, and that current tools are often blind to these complex, mixed-authorship situations.

Technical Summary: OpAI-Bench

Problem Statement

As AI writing assistants become integral to drafting and revision workflows, real-world documents increasingly result from progressive human–AI co-editing rather than being purely human-written or fully AI-generated. Existing AI-text detection benchmarks primarily evaluate static final outputs, failing to capture how AI authorship signals emerge, accumulate, or disappear during the revision process. Consequently, current benchmarks cannot answer critical questions regarding:

At what stage AI involvement becomes reliably detectable.
Which specific edit operations introduce the strongest detection signals.
Whether mixed-authorship texts are easier or harder to detect than endpoint cases (purely human or fully AI-edited).

Current evaluation frameworks often collapse diverse operations (e.g., polishing vs. paraphrasing) into a single "AI-written" label and lack intermediate revision states, obscuring the non-monotonic nature of detection difficulty.

Methodology: OpAI-Bench Construction

The authors introduce OpAI-Bench, an operation-guided benchmark designed to study progressive human-to-AI text transformation across multiple granularities (document, sentence, token, and span).

1. Data Construction Pipeline

Source Material: The benchmark is built from 15,722 human-written documents across four domains: student essays, news articles, government reports, and scientific abstracts.
Cumulative Trajectories: Unlike "naive" approaches that generate independent versions from a source, OpAI-Bench constructs cumulative revision trajectories. Starting from a human source document ( $V_0$ ), nine sequentially revised versions ( $V_1$ to $V_8$ ) are generated.
Deterministic Selection: To ensure reproducibility and remove positional bias, sentences within each document are shuffled using a deterministic seed based on the document ID.
Progressive Coverage: Each version $V_t$ applies edits to a specific, increasing percentage of sentences (from 15% to 100%) based on a fixed schedule. Once a sentence is selected for editing, it remains in the set for all subsequent versions, allowing for re-editing under new operations.
Edit Operations: Five representative AI edit operations are applied: Polish, Paraphrase, Style Rewrite, Compress, and Expand. Prompts enforce constraints such as preserving factual content and returning single sentences.
Granular Provenance: The benchmark preserves authorship labels at four levels:
- Document: Binary label (Human vs. AI-involved).
- Sentence: Binary label indicating if the sentence has entered the cumulative AI-edited set.
- Token/Span: Fine-grained localization of AI-edited regions using character offsets.

2. Evaluation Protocol

The benchmark evaluates detectors across three categories:

Zero-shot Detectors: Existing models (e.g., RADAR, Fast-DetectGPT, AdaLoc) evaluated without exposure to OpAI-Bench data.
LLM-as-Detector: Frontier models (GPT-5.4, Gemini 3 Flash, Claude Haiku 4.5) used as zero-shot classifiers via prompting.
Fine-tuned Detectors: Models trained on the OpAI-Bench training split using sentence-level provenance labels from specific generators.

Evaluation metrics include Accuracy and F1-AI across all nine versions ( $V_0$ – $V_8$ ) to track performance evolution throughout the revision trajectory.

Key Contributions

OpAI-Bench Benchmark: The first operation-guided benchmark that preserves intermediate revision states and multi-granularity provenance, enabling the study of detection as a trajectory rather than a binary endpoint decision.
Controlled Revision Trajectories: Construction of cumulative versioned datasets with predefined AI coverage levels and five distinct edit operations, covering four diverse domains.
Comprehensive Detector Benchmarking: Evaluation of 8 document-level, 7 sentence-level, and 2 fine-grained token/span-level detectors across varying coverage, operations, and revision histories.

Key Results

Experiments reveal that AI-text detectability is non-monotonic and governed by factors beyond the mere proportion of AI-edited content:

Non-Monotonic Detection Patterns: Mixed-authorship intermediate versions are often harder to detect than both fully human ( $V_0$ ) and heavily AI-edited ( $V_8$ ) endpoints. Specifically, a critical region of difficulty is observed around Version 4 (v4), where intermediate AI coverage coincides with the compression operation.
Operation Sensitivity: Detectability varies significantly by edit type. For instance, compression is generally harder to detect than expansion or paraphrasing at comparable coverage levels.
Domain and Generator Variance: Detector performance is highly dependent on the domain (e.g., news vs. essays) and the underlying generator. Document-level detectors show substantial variation; some (like RADAR) perform well on essays but fail on reports, while others (like Fast-DetectGPT) detect early edits in reports but degrade as revision progresses.
Granularity Differences:
- Document-level: Performance does not consistently improve with increased AI coverage; it fluctuates based on domain and operation.
- Sentence-level: Supervision improves stability but does not eliminate sensitivity to revision stages. Fine-tuned models generally recover better at higher AI coverage than zero-shot models.
- Token/Span-level: Fine-grained methods show decreasing accuracy but increasing F1-AI as AI-labeled regions become denser, reflecting class imbalance in early versions.
Cumulative vs. Independent: While independent editing (rewriting from source) produces smoother trajectories, the non-monotonic drop around the compression step persists, indicating that the difficulty is intrinsic to the operation and coverage level, though cumulative history can amplify instability.

Significance and Claims

The paper argues that reliable AI-text detection requires moving beyond binary endpoint classification toward trajectory-aware and operation-aware evaluation.

Limitations of Static Benchmarks: Existing benchmarks miss critical failure modes, particularly the "detection valley" in mixed-authorship texts where AI signals are neither absent nor dominant.
Need for New Evaluation: The authors claim that detection performance is not solely determined by the amount of AI content but is heavily influenced by how the text was edited (operation), where it was edited (domain), and the history of revisions.
Practical Implication: OpAI-Bench provides a controlled testbed to analyze when and how AI-assisted writing becomes detectable under realistic, progressive editing scenarios, highlighting the need for detectors capable of reasoning about partial and incremental AI involvement.

The authors conclude that current detectors are insufficient for the evolving landscape of human-AI co-editing and that future research must account for the dynamic nature of text transformation.