💻 computer science

From Papers to Progress: Rethinking Knowledge Accumulation in Software Engineering

Based on an analysis of 280 researchers' perspectives, this paper identifies structural barriers to knowledge accumulation in software engineering—such as isolated papers and misaligned incentives—and proposes four technology-agnostic principles for redesigning research artifacts to foster cumulative scientific progress.

Original authors: Jason Cusati, Chris Brown

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Cusati, Chris Brown

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the field of Software Engineering as a massive, bustling construction site. Every day, hundreds of new architects (researchers) show up, draw up blueprints (papers), and build new tools. The site is growing fast, and there is plenty of activity.

But here is the problem: The building isn't getting taller.

Even though everyone is working hard, the community is struggling to stack their work on top of each other to create something bigger. Instead, they are just building a million separate, tiny sheds that don't connect.

This paper, "From Papers to Progress," asks: Why can't we build a skyscraper when we have so many bricks?

The authors surveyed 280 experienced experts and found that the problem isn't that people aren't trying hard enough. The problem is the tools and rules they are using to share their work.

Here is the breakdown of the problem and the solution, using some everyday analogies.

🚧 The Four Broken Tools (The Problem)

The authors say the current way we do research has four major structural failures:

1. The "Island" Papers

The Analogy: Imagine every architect writes their blueprint on a piece of paper and locks it inside a glass box. You can see the paper, but you can't touch it, you can't highlight it, and you can't easily compare it to the blueprint next door.
The Reality: Research papers are written as long stories (prose). If you want to know if "Method A" is better than "Method B," you have to read two whole books, guess what the authors meant, and hope they used the same words. The data is buried in the text, making it impossible to automatically combine results.

2. The "Black Box" History

The Analogy: Imagine a chef publishes a recipe for a delicious cake. But the recipe only says "Mix ingredients." It doesn't say which brand of flour, how hot the oven was, or why they chose those ingredients. If you try to bake it, it might fail, and you won't know why.
The Reality: Papers show the final result but hide the messy journey. They don't explain why a researcher chose a specific dataset or how they handled a mistake. When others try to repeat the work, they are flying blind.

3. The "Frozen" Claims

The Analogy: Imagine a news report from 2010 says, "The sky is blue." In 2024, new science proves the sky is actually "blue with a hint of violet," but the old newspaper is still sitting on the shelf saying "blue." No one has updated the headline.
The Reality: Science changes. New evidence contradicts old ideas. But in software engineering, once a paper is published, it stays frozen. If a new study proves an old claim wrong, the old paper doesn't get a "Correction" stamp. It just sits there, confusing future researchers who have to manually figure out which facts are still true.

4. The "New Toy" Incentive

The Analogy: Imagine a school where the only prize is for inventing a new toy. If you spend your time fixing a broken toy or organizing the toy box so everyone can find things, you get zero points.
The Reality: The system rewards "New!" and "First!" It punishes people who try to check if old work is true (replication) or who try to organize existing knowledge. So, everyone chases the next shiny new idea, leaving the foundation messy and fragmented.

🛠️ The Four New Rules (The Solution)

To fix this, the authors suggest we stop treating research like a "finished book" and start treating it like a living, breathing digital ecosystem. They propose four principles:

1. Make it Structured (Not Just Stories)

The Idea: Instead of hiding facts in paragraphs, we should tag them like data.
The Analogy: Stop writing a novel about your cake. Instead, fill out a digital form: "Flour: 2 cups," "Sugar: 1 cup." This way, a computer (or a human) can instantly search for "All recipes using 2 cups of flour" and compare them instantly.

2. Show Your Work (Provenance)

The Idea: Keep a full, transparent log of every decision made.
The Analogy: Like a "Version History" in Google Docs. You should be able to click on a result and see the exact path: "We chose this data because..." and "We changed this setting because..." This builds trust and helps others learn from mistakes.

3. Make it "Living" (Not Frozen)

The Idea: Research artifacts should evolve after publication.
The Analogy: Think of a Wikipedia page, not a printed encyclopedia. If new data comes in, the "page" updates. If a claim is proven wrong, the record is corrected. The knowledge grows and changes, rather than staying stuck in the past.

4. Govern with Humans (Community Care)

The Idea: We need people to manage these new systems, not just code.
The Analogy: A library needs librarians. We need a community to decide who gets credit for organizing the shelves, how to handle disputes, and how to make sure the system is fair. We need to reward the "librarians" (those who consolidate knowledge) just as much as the "authors" (those who write new books).

🏁 The Bottom Line

The paper argues that Software Engineering is great at producing knowledge, but terrible at accumulating it.

We are building a library where every book is written in a different language, the pages are glued shut, and the librarians are fired if they try to organize the shelves.

To move from "Papers" to "Progress," we need to change our tools and our rewards. We need to build a system where knowledge is structured, transparent, updatable, and cared for by the community. Only then can we stop building isolated sheds and start building a skyscraper.

1. Problem Statement

Despite rapid growth in output and participation within Software Engineering (SE) research, the field faces a critical bottleneck: the inability to effectively accumulate, integrate, and reuse knowledge.

The Core Issue: While individual researchers are productive, the community struggles to synthesize results, track evolving claims, or build cumulative understanding over time.
Structural Barriers: The current research ecosystem treats papers as isolated units where knowledge is embedded in unstructured prose. This leads to:
- Loss of context and provenance as knowledge moves through the publication pipeline.
- Claims evolving without systematic tracking, leading to conflicting results in literature.
- Incentive structures that prioritize novelty over consolidation, replication, and infrastructure building.
Consequence: The field risks creating a "vast graveyard of undead theories" where new work does not genuinely extend or correct prior work due to fragmentation.

2. Methodology

The authors employ a community-informed qualitative analysis to diagnose the root causes of these barriers.

Data Source: Responses from the ICSE 2026 Future of Software Engineering (FOSE) pre-survey.
Sample Size & Demographics: $N=280$ $N = 280$ globally distributed researchers.
- Experience: Highly experienced (57% have 10+ years of experience).
- Productivity: Highly productive (44% published 11+ papers in the last 3 years).
- Geography: Europe (50%), North America (25%), Asia (15%), Others (10%).
Analysis Approach: Two researchers performed a lightweight qualitative analysis of open-ended survey comments. They identified recurring themes regarding knowledge building, synthesis difficulties, and systemic incentives.
Logic: The authors argue that because the community is experienced and productive yet still struggles with accumulation, the barriers must be structural (systemic design flaws) rather than individual (lack of effort or skill).

3. Key Contributions: Diagnosis of Structural Breakdowns

The paper identifies four interrelated structural breakdowns in how SE research artifacts are produced and connected:

Papers as Isolated Knowledge Units:
- Claims and evidence are buried in narrative prose, making extraction and comparison difficult.
- Artifacts (code/data) are often static snapshots; updates require new papers, leaving original claims unverified against new data.
- Synthesis requires manual reading of entire documents, which does not scale.
Loss of Context and Provenance:
- Critical methodological decisions (dataset splits, baselines, edge case handling) are often omitted or lack rationale in final publications.
- As work is cited, nuanced findings become "unqualified facts," and the reasoning path from hypothesis to result is lost.
- Later researchers must reverse-engineer decisions, often discovering that subtle choices significantly impacted results.
Untracked Claim Evolution:
- Scientific claims are refined, contradicted, or superseded, but the publication system lacks mechanisms to track these relationships.
- Conflicting claims (e.g., Technique A > B in one paper, B > A in another) exist without systematic flagging.
- Relationships (extension, contradiction, dependency) remain implicit in natural language, requiring extensive manual synthesis to determine the current state of knowledge.
Incentive Structures Favoring Novelty:
- Hiring, funding, and publication venues reward "new" techniques over replication, negative results, or synthesis.
- Researchers face high opportunity costs for investing in infrastructure or replication, as these efforts rarely yield top-tier publications or promotion credit.

4. Proposed Solutions: Four Technology-Agnostic Principles

To address these breakdowns, the authors propose a framework for rethinking research artifacts (papers, datasets, code, benchmarks, etc.). They argue for a shift from static documents to living, structured substrates.

Principle	Technical Requirement	Goal
1. Structured & Interpretable	Claims, evidence, and context must be explicit, machine-readable entities (e.g., semantic annotations, knowledge graphs), not just prose.	Enable automated extraction, comparison, and resolution of conflicting claims.
2. Inspectable & Provenance-Aware	Artifacts must preserve the full lineage of a claim (raw data $\to$ decisions $\to$ results) and make it accessible for verification.	Allow researchers to trace why a result occurred, verify evidence, and understand conditions.
3. Long-Lived & Reusable	Artifacts must be "living" (versioned, updatable) rather than static snapshots. Narratives remain historical records; structured data evolves.	Ensure knowledge accumulates in shared repositories and can be re-evaluated as datasets or methods improve.
4. Governed with Human Oversight	Community processes must manage quality, conflict resolution, and credit allocation (e.g., for infrastructure work).	Align individual incentives with collective goals; ensure ethical stewardship and trust in the infrastructure.

5. Results and Implications

The analysis does not provide a specific software tool but offers a diagnostic framework and design principles for the community.

Shift in Practice: Research should move toward documenting processes (via computational notebooks, version control) alongside outcomes.
Shift in Publication: Venues must treat structured artifacts as first-class contributions. Replication and consolidation work must be recognized and rewarded alongside novelty.
Shift in Infrastructure: The community needs shared platforms (knowledge graphs, benchmark repositories) maintained by governance structures that handle curation and dispute resolution.
Role of FOSE: The "Future of Software Engineering" track is positioned as a venue to experiment with these alternative artifact designs (e.g., living documents, structured claims) before broader adoption.

6. Significance

Paradigm Shift: The paper challenges the fundamental assumption that the "paper" is the primary unit of scientific progress in SE. It advocates for a shift where the artifact (the structured, evolving data/claim) is the unit of accumulation.
Addressing the Replication Crisis: By making provenance inspectable and claims structured, the field can better address the reproducibility crisis (noted in the survey where respondents cited low replicability).
Sustainable Growth: Without these changes, the field risks fragmentation where knowledge production outpaces knowledge integration. The proposed principles offer a path to ensure the next generation inherits a buildable knowledge base rather than just a literature to read.
Human-in-the-Loop: Crucially, the authors emphasize that technology alone is insufficient; governance and incentive realignment are required to make these technical solutions viable.

In summary, the paper argues that for Software Engineering to mature as a cumulative science, it must transition from a culture of dissemination (publishing isolated papers) to a culture of accumulation (maintaining structured, provenance-aware, and governed knowledge substrates).