💬 NLP

Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision

This paper introduces Re3, a holistic framework and the accompanying Re3-Sci dataset of annotated scientific paper revisions, reviews, and edit summaries, to enable the modeling of collaborative document revision and to evaluate the capabilities of large language models in automating edit analysis and facilitating text-based collaboration.

Original authors: Qian Ruan, Ilia Kuznetsov, Iryna Gurevych

Published 2026-01-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qian Ruan, Ilia Kuznetsov, Iryna Gurevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a group of friends trying to write a story together. One person drafts a paragraph, another reads it and says, "This part is confusing, and that fact is wrong." The first person then rewrites the paragraph, and a third person writes a note explaining why they made those changes.

This paper, titled Re3, is about building a map to understand exactly how this messy, back-and-forth process works. The authors realized that while computers are getting good at reading text, they are terrible at understanding the conversation behind the changes. They wanted to create a system that connects three things:

The Review: The feedback (e.g., "Fix this grammar").
The Revision: The actual changes made to the text.
The Response: The author's explanation of what they did.

The Problem: Missing the Big Picture

Before this paper, researchers studied these pieces separately. They looked at how sentences change, or how reviews are written, but they didn't have a way to see the whole picture at once. It's like trying to understand a movie by only watching the trailers, the script, and the director's notes separately, without ever seeing the film itself.

The authors argue that to really understand how people collaborate, you need to see the entire document and how every single change fits into the bigger story.

The Solution: Re3 and the "Re3-Sci" Dataset

To fix this, the team created Re3, a new framework (a set of rules and categories) for analyzing these documents. They then built a massive dataset called Re3-Sci to test it.

Think of Re3-Sci as a giant, organized library of 314 scientific papers that went through the peer-review process. For every paper, they have:

The original draft.
The final version.
The reviewers' comments.
The authors' replies.

But they didn't just collect the files; they acted like super-detectives. They manually labeled 11,600 specific changes (edits) in these papers. For every change, they asked:

What did they do? (Did they add a sentence? Delete a paragraph? Merge two ideas?)
Why did they do it? (Was it to fix a typo? To add a new fact? To change a scientific claim?)
Where did it happen? (Was it a tiny word change or a whole section rewrite?)

What They Discovered (The "Human" Insights)

By looking at all this data, the authors found some interesting patterns about how humans edit:

The "Bookends" Effect: People tend to do the most editing at the very beginning (Introduction) and the very end (Conclusion) of a paper. The middle is usually more stable.
Fact vs. Flow: Early in the process, people mostly fix grammar and clarity (making sentences flow better). Later on, they focus more on changing facts and scientific claims.
Reviewers are Specific: When a reviewer gives a specific, clear instruction (e.g., "Add a citation here"), authors usually follow it. But if the reviewer just says "This section is weak," authors are less likely to make a specific change.
Not Everything Changes: Even in papers that get "heavily revised," most of the text stays the same. Only about 18% of the sentences are actually changed, and even fewer are changed for deep scientific reasons.

Can Computers Do This? (The AI Experiment)

The authors also asked: "Can modern AI (Large Language Models) understand this process?"

They tested the AI on four tasks:

Guessing the Intent: Can the AI look at a change and guess if it was for grammar or for a new fact?
- Result: The AI got about 70% right. It's pretty good, but it still struggles to tell the difference between a "fact" change and a "claim" change.
Summarizing Changes: Can the AI read all the changes and write a short summary for the author to send back to the reviewers?
- Result: The AI could write a summary, but it sometimes missed details or got the facts slightly wrong. It tended to group changes by "what was done" (e.g., "We deleted 5 things") rather than "where it happened" (e.g., "In the Introduction, we changed...").
Matching Reviews to Changes: Can the AI link a reviewer's comment to the specific sentence the author changed?
- Result: The AI was very good at this, almost perfect.
Finding Review Requests: Can the AI spot which sentences in a review are actually instructions for the author?
- Result: The AI was very good at spotting these instructions.

The Bottom Line

This paper doesn't claim that AI is ready to replace human editors or that it can write scientific papers for us yet. Instead, it provides the first complete map of how collaborative writing actually happens.

They built the tools, the data, and the rules to help computers learn the "language" of collaboration. It's like giving a computer a dictionary and a grammar book for the complex dance of writing, reviewing, and revising, so that in the future, AI can actually help humans work together better, rather than just guessing what they are doing.

Technical Summary: Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision

Problem Statement

Collaborative review and revision of textual documents are central to knowledge work, involving iterative cycles of drafting, feedback (reviews), execution of revisions, and author responses. Despite the potential for Natural Language Processing (NLP) to assist in this domain, a holistic framework capable of modeling the complex relationships between reviews, revisions, and author responses is lacking.

Prior NLP research has largely focused on isolated components: relationships between original and revised documents, reviews and original documents, or reviews and responses. Existing datasets often neglect the broader document context, focusing instead on sentence-level edits, or fail to capture the underlying intent of revisions. Furthermore, there is a gap in methodologies and datasets for "full-scope" modeling, which encompasses edits across various granularity levels (from subsentences to sections) and includes qualitative labels for both the action (how) and intent (why) of the revision. This limitation hinders the development of NLP applications that can effectively automate edit analysis or facilitate text-based collaboration.

Methodology

The Re3 Framework

The authors introduce Re3, a holistic framework that conceptualizes the review-revision-response procedure as a set of interactions among four document types: the original document ( $D_t$ ), the revised document ( $D_{t+1}$ ), the review ( $C$ ), and the author response ( $A$ ).

Graph Representation: Documents are represented as graphs preserving structure and granularity. Text elements range from subsentences to sections. Cross-document relations (e.g., a review request prompting a specific edit) are treated as edges between these document graphs.
Edit Alignment: The framework links elements from the new and old versions ( $e_{ij}$ $e_{ij}$ ), associating each alignment with an Edit Label ( $EL_{ij}$ $E L_{ij}$ ) comprising three dimensions:
1. Granularity: The scope of the text element (subsentence, sentence, paragraph, section).
2. Action: The method of revision (Add, Delete, Modify, Merge, Split, Fusion).
3. Intent: The underlying purpose (Grammar, Clarity, Fact/Evidence, Claim, Other).

Dataset Construction: Re3-Sci

To instantiate this framework, the authors created Re3-Sci, a large-scale, human-annotated corpus in the scholarly domain.

Data Source: 314 document pairs (revised scientific papers) and their corresponding peer reviews were randomly selected from the F1000RD and NLPeer datasets.
Annotation Process:
- Pre-alignment: A lightweight algorithm using form and semantic similarity (Levenshtein, fuzzy matching, SBERT) pre-identifies sentence alignment candidates, achieving 0.95 accuracy before human validation.
- Environment: A cross-document annotation environment was built using INCEpTION to allow annotators to view full document context, crucial for accurate intent labeling.
- Annotators: Six master's students with C1-level English proficiency performed the annotation.
- Quality Control: An iterative quality management process involving training, pilot studies, and intermediate meetings improved Inter-Annotator Agreement (IAA).
Statistics: The dataset contains 11,648 sentence-level edits, 5,064 paragraph-level edits, and 2,008 section-level edits. It also includes 560 extracted review requests and 1,364 linkages between revisions and author response summaries.
Reliability: The dataset achieved a substantial IAA of 0.78 (Krippendorff's $\alpha$ ) for labeling and a perfect IAA of 1.0 for alignment.

Experimental Evaluation

The authors utilized Re3-Sci to assess the capabilities of state-of-the-art Large Language Models (LLMs), specifically Llama2-70B and GPT-4, on four tasks:

Edit Intent Classification: Predicting the intent label for a sentence edit.
Document Edit Summarization: Generating a coherent summary of all edits in a document.
Revision Alignment: Determining if two sentences from different versions constitute a revision pair.
Review Request Extraction: Identifying review sentences that prompt revisions.

The experiments employed In-Context Learning (ICL) with various demonstration selection methods (dynamic vs. static) and Chain-of-Thought (CoT) prompting strategies.

Key Results

Dataset Analysis Insights

Analysis of Re3-Sci revealed several patterns in human collaborative revision:

Action/Intent Correlation: Additions and deletions are primarily driven by changes in factual content or claims, whereas modifications are predominantly used for grammar and clarity improvements.
Distribution: Revisions are concentrated in the initial and final parts of papers. The beginning sees more modifications, while the end sees more additions/deletions.
Clustering: Edits are not evenly distributed; they cluster significantly within specific paragraphs (average Crest Factor of 3.79).
Review Impact: Explicit edit suggestions from reviewers are implemented more frequently (over 50%) than implicit suggestions or general weakness comments.

LLM Performance

Edit Intent Classification: Llama2-70B achieved a macro-average F1 score of 0.69 on the full test set. The best performance was achieved using a combination of dynamic demonstrations (selected via difference-based similarity) and static examples with CoT rationale. The model struggled with distinguishing between "Fact/Evidence" and "Claim" changes.
Revision Alignment: Llama2-70B achieved 0.97 accuracy, significantly improving upon the pre-alignment algorithm's recall for alignment (0.92 vs. 0.99) and precision for non-alignment (0.89 vs. 0.99).
Document Edit Summarization: GPT-4 generated coherent summaries but exhibited issues with factuality (4% of sentences contained factual errors) and comprehensiveness (failing to address 21% of edits). Humans outperformed GPT-4 in factuality and specificity, while GPT-4 produced slightly more compact summaries.
Review Request Extraction: The model achieved high recall (0.98) for positive samples but lower precision (0.74), indicating difficulty in distinguishing actionable requests from general comments.

Significance and Contributions

The paper claims four primary contributions:

A Holistic Framework: Re3 provides the first formal framework for jointly modeling reviews, revisions, and responses, complete with a label taxonomy and robust annotation methodology.
A High-Quality Dataset: Re3-Sci is the first large-scale corpus with full-scope annotations of edit alignments, actions, and intents across multiple granularity levels in academic writing.
Empirical Insights: The study provides the first large-scale analysis of human editing behavior in scholarly peer review, revealing relationships between edit actions, intents, and document structure.
Automation Baselines: The work establishes baselines for four NLP tasks, demonstrating that while LLMs show promise (particularly in alignment and summarization), significant challenges remain in understanding complex edit intents and ensuring factual accuracy in generated summaries.

The authors conclude that their work paves the path for systematic, full-scope study of text-based collaboration in NLP, offering a foundation for future research into automated revision assistance, edit generation, and the anchoring of edits to review requests. They explicitly note limitations, including the focus on English-language scientific publications and the use of human-generated summaries rather than author responses due to data availability.