On the Abolition of the "ICSE Paper" and the Adoption of the "Registered Proposal" and the "Results Report"

Technical Summary: On the Abolition of the "ICSE Paper" and the Adoption of the "Registered [Proposal]" and the "[Results] Report"

Problem Statement
The paper identifies two critical, interlocking crises plaguing the software engineering (SE) research community: a replicability crisis and a "novelty-vicious cycle."

• Replicability Crisis: There is a widespread inability to reproduce results, particularly evident in fields like machine learning for vulnerability detection where new papers frequently invalidate previous findings. Survey data indicates that researchers often remove code or data post-publication, and complex toolchains make replication difficult.
• Novelty-Vicious Cycle: The current publication model prioritizes "novelty" (new tools, datasets, or techniques) over "rigor" and validation. This creates a feedback loop where:
  • Reviewers reject replication studies or negative results because they are not "new."
  • Established groups dominate the literature, creating an elitist environment where challenging established norms is discouraged.
  • Researchers focus on "toy examples" and incremental improvements to secure publication, resulting in tools that are unusable in industrial practice.
  • The community fails to consolidate knowledge, leading to a shallow body of research with limited industrial relevance.

Methodology
The authors do not present new empirical data or a new software tool. Instead, the paper is a position paper proposing a structural reform of the publication ecosystem. The methodology relies on:

1. Survey Analysis: The authors analyze responses from the "Future of Software Engineering" pre-survey (Storey & van der Hoek, 2026) and discussions from the community Discord channel. They extract specific themes regarding the tension between novelty and rigor, and the marginalization of replication.
2. Conceptual Proposal: Based on these qualitative findings, the authors propose a new two-tier publication model to replace the traditional "ICSE paper" (where idea, methodology, and validation are performed by the same authors).

Key Contributions: The Two-Tier System
The core contribution is the proposal to abolish the standard technical paper format for main tracks and replace it with two distinct, first-class publication types:

1. Registered [Proposal]:

   • Focus: Novelty and Methodological Rigor.
   • Content: Authors submit a new research idea, hypothesis, and a detailed experimental/analysis plan before conducting the work.
   • Review Criteria: Reviewers evaluate the novelty of the idea and the soundness of the proposed methodology. The paper argues that the expectation of results should be removed at this stage; the proposal is judged on its potential to close gaps identified by past negative results or theoretical justifications.
   • Outcome: If accepted, the proposal is presented, and the "race" to validate it begins.

2. [Results] Report:

   • Focus: Rigor and Validation.
   • Content: Any researcher (not just the original proposers) can submit a report detailing the execution of the validation experiment based on a previously accepted Registered Proposal.
   • Format: These are shorter papers (potentially half-page pointers to the proposal) that focus strictly on the execution and results, complemented by artifacts.
   • Inclusivity: Crucially, this tier allows multiple independent teams to validate the same proposal. This injects replication directly into the publication pipeline rather than treating it as an afterthought.
   • Outcome: The system expects that most independent validations will fail to confirm the original proposal. Only those proposals that survive independent, rigorous validation by multiple teams are considered robust enough for industrial adoption.

Results and Claims
As a position paper, the authors do not report experimental results of their own system. Instead, they present the following claims and expectations based on their analysis:

• Decoupling Novelty and Rigor: The proposed system separates the validation of "new ideas" (Proposal) from the validation of "truth" (Results Report), addressing the trade-off where reviewers currently demand both simultaneously.
• Democratization of Validation: By allowing "anyone" to submit a Results Report, the system breaks the monopoly of established groups. It removes the need for "authorization" from famous researchers to challenge existing work.
• Handling Negative Results: The system explicitly values negative or conflicting results as informative data points that measure robustness, rather than discarding them as failures.
• The "Two Independent Implementations" Rule: The authors draw a parallel to the IETF's rule for RFC adoption (requiring two independent interoperable implementations). They argue that industry adoption should only follow when at least two independent groups have successfully validated a Registered Proposal.
• Mitigating the "Lost Hard Disk" Problem: By requiring multiple teams to execute the same protocol, the risk of data loss (e.g., a post-doc losing a hard disk) is exponentially reduced, ensuring the survival of valid data.

Significance
The paper claims its significance lies in offering a "bold" and "disruptive" solution to the stagnation of the SE field. It argues that the current culture of chasing novelty at the expense of consolidation has rendered SE research "shallow" and disconnected from industrial practice. By institutionalizing the separation of proposal and validation, the authors aim to:

• Transform the community from trend-driven to cumulative.
• Create a culture where emerging researchers feel safe to challenge established ideas without fear of gatekeeping.
• Ensure that only solutions proven robust by independent replication reach industrial practice.

The authors acknowledge that their proposal is a hypothesis based on community sentiment and that it may fail to deliver its goals, but they assert it is a necessary step to address the "post-doc lost hard disk" phenomenon and restore trust in empirical software engineering.