Towards A Sustainable Future for Peer Review in Software Engineering
This paper proposes a vision for a sustainable future in Software Engineering peer review that addresses the growing imbalance between submissions and reviewers by implementing scalable strategies to attract and train new reviewers, incentivize community participation, and cautiously integrate AI tools.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the world of software engineering research as a massive, bustling marketplace where scientists trade their new ideas (called "papers") to get them published. To keep this marketplace honest and high-quality, there's a strict rule: every new idea must be inspected by a panel of experts before it can be sold. This inspection process is called peer review.
However, the paper by Parra and colleagues argues that this marketplace is in trouble. The number of ideas being submitted is growing faster than the number of experts available to inspect them. It's like a restaurant where the line of hungry customers is getting longer every day, but the kitchen staff isn't growing fast enough to cook the meals. The result? The chefs (reviewers) are exhausted, working late nights and weekends, and the quality of the food (the research) might start to suffer.
Here is a breakdown of the problem and the authors' proposed solutions, using simple analogies:
The Problem: A Tipping Point
The authors point out that the "kitchen" is overwhelmed.
- The Overload: In the last few years, the number of papers submitted to major conferences has nearly tripled, while the number of reviewers has only doubled. This means each reviewer is stuck with more plates to check.
- The Invisible Labor: Being a reviewer is like being a volunteer firefighter. You have a full-time job as a scientist or teacher, and then you spend your weekends and evenings inspecting other people's work for free. In the current "Publish or Perish" culture (where your career depends on how many papers you publish), this volunteer work doesn't help you get promoted. It's often ignored by universities when deciding who gets a raise or tenure.
- The AI Trap: Because reviewers are so tired, some are turning to Artificial Intelligence (AI) to do the work for them. The authors warn that this is dangerous. It's like asking a robot to taste-test a complex dish; the robot might make up ingredients that don't exist or give vague advice like "this tastes good" without knowing why. In one recent case at a top AI conference, over 15,000 reviews were found to be fully written by AI, which is a major red flag for the quality of the research.
What's Working (The Good News)
The paper acknowledges that some things are already helping:
- The "Hall of Fame": Some conferences give out "Distinguished Reviewer Awards" to the hardest workers. This is like a "Employee of the Month" plaque. It's nice to have, but it usually goes to the same experienced people over and over, so it doesn't really encourage new people to join the team.
- The "Shadow" Program: Some conferences have started a "Junior PC" or "Shadow PC" program. Think of this as a shadowing internship. Newcomers (like PhD students) sit next to an experienced reviewer, watch how they inspect a paper, and then try it themselves. The expert gives them feedback. This is a great way to train new inspectors, but so far, it's only happening at a few conferences, not everywhere.
The Vision: A Sustainable Future
The authors propose a three-part plan to fix the system and make it sustainable for the long haul:
1. A "Driver's License" for Reviewers (Scalable Training)
Currently, anyone can volunteer to review a paper, but they might not know how to do it well. The authors suggest creating an online training course (like a certification program).
- How it works: Newcomers would watch videos from experts, learn the rules of good reviewing, and take a quiz.
- The Reward: Once they pass, they get a digital certificate. They can put this on their profile (like a LinkedIn badge) to show conference organizers, "I am trained and ready to review." This creates a larger pool of qualified, ready-to-go inspectors.
2. AI as a "Co-Pilot," Not the Pilot (Responsible AI Use)
The authors don't want to ban AI, but they want to use it carefully.
- The Rule: AI can help check for typos or summarize data, but it cannot write the final review or replace human judgment.
- The Safety Net: Editors should use tools to detect if a review was written by a robot. If someone tries to sneak in a fake AI review, they should be penalized.
- The Helper: AI could be used after humans have done their work to double-check for things the humans might have missed, but only after the humans have already formed their own opinions.
3. Better Rewards (Incentives)
To get more people to volunteer, the system needs to value their time better. The authors suggest:
- The "Pay-to-Play" Rule: If you want to submit a paper to a conference, you must agree to review papers for that conference too. It's a "you scratch my back, I scratch yours" deal.
- Discounts and Badges: Give reviewers a discount on conference registration fees or a special badge on their name tag so everyone knows they are helping out.
- Newcomer Awards: Create a specific award just for new reviewers, so they feel recognized early in their careers, not just after 20 years of service.
The Bottom Line
The paper concludes that if we don't fix this now, the whole system could collapse. The "kitchen" will be too busy, the "food" (research) will be low quality, and the best experts will burn out. By training more people, using AI responsibly as a helper, and giving reviewers better recognition, the software engineering community can ensure that their marketplace of ideas remains healthy, fair, and growing for the future.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.