Technical Summary: SCHOLAWRITE: A Dataset of End-to-End Scholarly Writing Process

Problem

Scientific writing is a cognitively demanding, non-linear process characterized by recursive cycles of planning, drafting, and revising. Despite the integration of Large Language Models (LLMs) into research workflows, current writing assistants largely operate autoregressively, generating text linearly without understanding the underlying cognitive shifts of the writer. Existing datasets typically capture only final manuscript versions or short-term revision logs, failing to represent the longitudinal, intention-driven nature of scholarly writing. Consequently, there is a lack of empirical data to model how human scientists actually write, limiting the development of AI assistants that can genuinely align with and support the human cognitive writing process.

Methodology

To address this gap, the authors present SCHOLAWRITE, the first dataset capturing the end-to-end scholarly writing process from initial drafts to final manuscripts.

Data Collection

• Participants: Ten graduate students in computer science at a U.S. R1 university.
• Duration: Four months (November 2023 – February 2024).
• Tooling: A custom Chrome extension was developed to unobtrusively record real-time keystrokes from the Overleaf LaTeX editor. The system captures every keystroke, compares text versions using a diff algorithm, and logs metadata (timestamp, file name, author ID) while filtering out non-consented projects to ensure privacy.
• Scope: The dataset covers five Overleaf projects culminating in arXiv preprints, containing approximately 62,000 keystroke-based writing actions and over 118,000 words added or deleted.

Annotation and Taxonomy

• Process: Two annotators (graduate students with NLP expertise) reviewed the keystroke data using a specialized interactive interface. They applied a cognitive taxonomy grounded in writing process theory (Flower & Hayes, 1981) and prior empirical studies.
• Taxonomy: A 15-category taxonomy of writing intentions was developed, grouped into three high-level categories:
  1. Planning: Idea Generation, Idea Organization, Section Planning.
  2. Implementation: Text Production, Object Insertion, Citation Integration, Cross-reference, Macro Insertion.
  3. Revision: Fluency, Coherence, Clarity, Structural, Linguistic Style, Scientific Accuracy, Visual Formatting.
• Reliability: The annotation process achieved a weighted F1 inter-annotator agreement of 0.71 on a 1,000-keystroke subset.

Analysis and Evaluation

The authors utilized SCHOLAWRITE for three primary purposes:

1. Behavioral Analysis: Examining the distribution, duration, and transitions of writing intentions to characterize the non-linear nature of scholarly writing.
2. Benchmarking: Evaluating state-of-the-art LLMs (GPT-5, Qwen2.5-14B) on next-intention prediction and output alignment tasks.
3. Model Training: Fine-tuning models (BERT, RoBERTa, Llama-3.1-8B) on the dataset to assess improvements in aligning with human writing workflows.

Key Contributions

1. The SCHOLAWRITE Dataset: A publicly available, fine-grained corpus linking every keystroke to a cognitive writing intention, covering the full lifecycle of five scholarly manuscripts.
2. Empirical Insights into Writing Behavior:
   • Scholarly writing is highly non-linear and multi-intentional; over 57% of writing sessions involve three or more distinct intentions.
   • Writing is dominated by Text Production (57.4%) and Revision (specifically Clarity and Coherence), while Planning activities occupy a smaller share of time but often involve longer, more focused sessions.
   • Writing patterns shift across phases: early stages focus on structuring and idea generation, while later stages are dominated by revision and formatting.
3. Benchmarking and Training Results:
   • Current LLM Limitations: Zero-shot models struggle to predict next intentions (weighted F1 < 0.5) and fail to sustain cognitively complex revisions, particularly in Planning and Implementation.
   • Impact of Fine-tuning: Models fine-tuned on SCHOLAWRITE (e.g., BERT, RoBERTa, Llama-8B) showed significant improvements in next-intention prediction and output alignment.
   • Iterative Writing: Fine-tuned models adopted more human-like iterative patterns, frequently alternating between Planning, Implementation, and Revision, whereas baseline models tended to remain in a single stage.

Results

• Behavioral Analysis: The data reveals that "rapid, iterative drafting" and "longer, focused planning/visual composition" are complementary cognitive modes. Transitions between intentions are dense and bidirectional, with Revision intentions (Clarity, Fluency, Coherence) forming strong recursive loops.
• Prediction Performance: Fine-tuned BERT and RoBERTa achieved weighted F1 scores of 0.64 for next-intention prediction, a substantial improvement over base models (0.04–0.12).
• Alignment: Fine-tuned Llama-8B models demonstrated higher semantic similarity (BERTScore-F1) and operational similarity (Levenshtein distance) to human edits compared to zero-shot baselines.
• Iterative Generation: In 100-iteration self-writing tasks, fine-tuned models covered a broader range of writing intentions and produced more lexically diverse and topic-consistent outputs than baselines, though they still lagged in surface-level fluency and logical coherence.

Significance

The paper argues that capturing the cognitive dynamics of writing is essential for building writing assistants that support rather than substitute human cognition. SCHOLAWRITE provides a foundational resource for:

• Studying the fine-grained cognitive patterns of scholarly writing.
• Evaluating the alignment of LLMs with human writing intentions and workflows.
• Training models that can adapt to evolving, multi-intent writing behaviors.

The authors conclude that while current models still face challenges in text quality, exposure to process-level supervision via SCHOLAWRITE enables models to better emulate the process of human scholarly writing. They acknowledge limitations, including the dataset's focus on the computer science domain (LaTeX), the small sample size due to privacy concerns, and the exclusion of pre-writing ideation outside the editor, suggesting future work should expand to broader disciplines and multimodal collaboration.