Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
This paper introduces a pipeline that transforms structured scientific papers into multi-turn generation trajectories for continued pre-training and benchmarking, demonstrating that this approach significantly enhances academic writing capabilities while preserving general reasoning and long-document comprehension.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Large language models, the powerful computer programs that can write stories, solve problems, and answer questions, have reached a point where they are running out of fresh, high-quality human text to learn from. For years, researchers have fed these systems vast amounts of books, websites, and articles. Now, as the best human writing becomes scarce, scientists are turning to synthetic data—text created by computers themselves. A common approach has been to take existing short web articles and rewrite them into cleaner, more textbook-like formats. However, this method changes the original content and treats documents as simple streams of words, ignoring how a human author actually planned and structured a piece of writing. A newer idea suggests that instead of just rewriting the final text, we should try to reconstruct the hidden thinking process that led to it. The challenge has been that most existing text is too short to have a complex structure, leaving little room to recover the author's original plan.
A team of researchers has now taken this concept and applied it to scientific papers, which are written with a clear, uniform structure and dense, compressed ideas. They developed a method to "unfold" a finished scientific paper back into the multi-step journey of its creation. Imagine a finished house; instead of just looking at the walls, this process reconstructs the architect's initial sketch, the list of materials, the notes taken before laying each brick, and the final inspection that led to the blueprint. The researchers built a system that takes a real scientific paper and works backward to generate the writing request that started it, the global plan for its sections, and the specific thoughts an author would have had before writing each paragraph. Crucially, the actual text of the paper remains exactly as the original authors wrote it; the system only invents the surrounding thinking process that explains how that text came to be.
The team applied this unfolding process to 1.8 million scientific papers from the arXiv repository, a massive collection of pre-print research. By running this reconstruction on papers from 2006 through early 2026, they transformed 30 billion words of raw paper text into a new training dataset containing roughly 60 billion words of "trajectories." Each trajectory is a long, multi-turn conversation where a computer acts as a teacher, first asking for a paper, then outlining a plan, then deliberating on how to write each section before finally producing the real text. This process effectively doubled the size of the training data while also stretching the length of the documents the models had to process, moving the average training document from about 11,000 words to nearly 29,000 words. The researchers found that using this reconstructed thinking process helped the models learn to write much better than simply feeding them the same papers as plain text.
The results showed that this method improved the models' writing abilities across the board, including in academic and engineering contexts, without hurting their ability to reason or solve logic puzzles. Even when the models were later fine-tuned on other public writing datasets, the models that had first learned from these "unfolded" trajectories still performed better than those that had not. The study also demonstrated that this same reverse-engineering technique could be used to create new types of instruction data and even a new testing ground for academic writing. By taking a paper that the model had never seen before, the system could generate a specific question about it, along with a detailed checklist and scoring guide to evaluate the answer. This created a benchmark called PAW-Bench, which contains nearly 3,000 tasks anchored in real research papers.
One of the most surprising findings was that the size of the computer model used to generate the thinking process mattered less than the method itself. The researchers used three different models to create the thinking data: a small one, a medium one, and a large one. Contrary to the intuition that a smarter, larger model would produce better training data, the smallest model actually yielded the best results for writing and reasoning. The data generated by the larger models was too uniform and repetitive, while the data from the smaller model contained more variety and "surprises" that forced the learning model to work harder and learn more effectively. This suggests that the value lies in the structure of the reconstruction—the act of unfolding the writing process—rather than in the intelligence of the machine doing the unfolding.
The researchers also observed that this method helped the models understand long documents better. Because the reconstructed trajectories were significantly longer than the original papers, the models became more comfortable reading and reasoning over thousands of words at once. This improvement in long-context ability was not seen in models that were trained on the same papers but left as plain text. The study concludes that by treating real human writing as the final answer and reconstructing the steps that led to it, researchers can create a powerful new form of training data. This approach allows artificial intelligence to learn not just what to write, but how to think about writing, bridging the gap between raw information and the structured, deliberate process of human creation.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.