🤖 AI

Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment

This paper evaluates reproducibility practices in evolutionary computation over a ten-year period by comparing a manual assessment using a structured checklist against a new LLM-based automated pipeline called RECAP, finding significant gaps in reporting and demonstrating that automated tools can effectively monitor reproducibility.

Original authors: Francesca Da Ros, Tarik Začiragić, Aske Plaat, Thomas Bäck, Niki van Stein

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Francesca Da Ros, Tarik Začiragić, Aske Plaat, Thomas Bäck, Niki van Stein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are following a recipe for a legendary chocolate cake you found online. You follow the instructions perfectly, but the cake turns out to be a flat, salty mess. You realize the recipe forgot to mention the oven temperature, didn't say whether to use salted or unsalted butter, and the "secret ingredient" was a link to a website that no longer exists.

This paper is essentially a "Recipe Audit" for the world of high-tech science.

The Problem: The "Secret Sauce" Crisis

In the world of Evolutionary Computation (a type of AI that learns by "evolving" solutions, much like biological evolution), scientists publish papers claiming they’ve found a "perfect recipe" for solving complex problems.

However, the researchers discovered a major problem: many of these scientific "recipes" are incomplete. They might tell you what they achieved, but they forget to tell you the exact "temperature" (settings) or provide the "kitchen tools" (the actual computer code) needed to recreate the result. If you can't recreate the result, the science isn't truly reliable.

The Study: The Great Science Inspection

A team of researchers decided to act like health inspectors for a decade's worth of scientific papers (from 2016 to 2025). They looked at 168 major papers to see how "reproducible" they actually were. They used a Checklist—much like a pilot’s pre-flight checklist—to see if the authors included:

The Ingredients: The exact data and parameters used.
The Kitchen Setup: What kind of computers they used.
The Cooking Instructions: The actual code and step-by-step logic.

What they found:

The "Incomplete Recipe" Score: On average, papers only scored about 62% on completeness. It’s like a recipe that tells you how to bake a cake but forgets to tell you how long to leave it in the oven.
The Missing Tools: Only about 37% of papers actually provided the "tools" (the code or data) to let someone else try it.
Broken Links: Some scientists provided links to their code, but by the time the inspectors arrived, the links were dead—like a recipe pointing to a restaurant that has already gone out of business.

The Innovation: RECAP (The Robot Inspector)

Manually checking thousands of papers is exhausting and slow. To solve this, the researchers built RECAP, an AI-powered "Robot Inspector."

Think of RECAP as a highly trained digital sous-chef. Instead of a human reading every single word, RECAP uses a Large Language Model (like the technology behind ChatGPT) to scan the papers and even "try out" the code in a digital sandbox to see if it actually works.

The researchers found that the Robot Inspector was remarkably good—it agreed with the human experts about 67% of the time. While it’s not perfect, it’s a massive step toward being able to scan thousands of scientific papers instantly to flag the ones that are "unreliable recipes."

The Big Picture: Why does this matter?

Science is built on a foundation of trust. If one scientist claims to have found a cure or a breakthrough, everyone else needs to be able to test it to make sure it’s real.

By creating better checklists and using AI to catch missing information, this paper is helping to ensure that when scientists publish a "recipe" for the future, it actually works when you try to cook it in your own kitchen.

Technical Summary: Assessing Reproducibility in Evolutionary Computation

1. Problem Statement

In fields driven by computational experiments—such as Evolutionary Computation (EC), Machine Learning (ML), and Artificial Intelligence (AI)—the validity of research conclusions relies heavily on the ability of independent researchers to rerun experiments. Reproducibility in EC is particularly challenging due to the stochastic nature of algorithms, the complexity of parameter tuning, and the dependence on specific hardware, software environments, and benchmark datasets.

Despite the recognized importance of "Open Science," there is a lack of empirical evidence regarding the actual reproducibility levels in the EC community. Furthermore, reviewers often lack standardized, scalable tools to consistently evaluate whether a paper provides sufficient information and artifacts to make replication feasible.

2. Methodology

The authors employ a mixed-method approach, combining manual human assessment with an automated Large Language Model (LLM) pipeline to study a corpus of 168 full papers published in the Evolutionary Combinatorial Optimization and Metaheuristics (ECOM) track of the GECCO conference over a ten-year period (2016–2025).

The Reproducibility Checklist: The researchers developed a structured checklist tailored to EC, organized into five dimensions: (i) methodological clarity, (ii) experimental setup, (iii) results reporting, (iv) artifact evaluation, and (v) paper metadata.
Manual Assessment: A human-led protocol involved reading the papers, inspecting supplementary materials, and—crucially—attempting to set up and run provided code/data within a fixed two-hour time budget per paper.
RECAP (REproducibility Checklist Automation Pipeline): To test scalability, the authors developed RECAP, an LLM-based system (using OpenAI’s GPT-4o/nano models).
- Process: RECAP extracts text from PDFs and processes repository URLs.
- Tool Use: For artifact-related fields, the system clones Git repositories into an isolated Docker sandbox and attempts to execute the code (limited to a five-minute window) to verify functional exercisability.
- Constraint Management: It uses structured JSON schemas to ensure the LLM acts as an information retrieval and extraction tool rather than a reasoning engine, minimizing hallucinations.

3. Key Contributions

A Domain-Specific Checklist: A specialized framework for assessing reproducibility in empirical evolutionary research.
RECAP System: A novel, automated pipeline that uses LLMs and sandboxed code execution to evaluate reproducibility signals at scale.
Longitudinal Empirical Study: A comprehensive ten-year analysis of reporting practices in a major EC conference track.
Comparative Benchmark: A validation of how closely automated LLM-based assessments can align with human expert judgments.

4. Results

Low Completeness: The average completeness score of the analyzed papers is 0.62. While there is a slight upward trend in reporting over the years, significant dispersion remains.
The "Operational Gap": There is a structural imbalance in reporting. While "descriptive" items (e.g., problem descriptions, aggregated results) are frequently reported, "operational" details (e.g., parameter ranges, random seeds, tuning budgets, and execution environments) are systematically omitted.
Artifact Scarcity and Fragility: Only 36.90% of papers provide additional materials (code/data). Furthermore, a significant portion of provided links were broken at the time of assessment, and very few papers used persistent archival repositories (like Zenodo).
Best Paper Trends: While award-winning papers were more likely to provide artifacts (70% vs 34.5% for non-nominated papers), they did not show a statistically significant increase in overall completeness scores.
Automation Feasibility: RECAP achieved a mean accuracy of 76.8% and a Cohen’s $\kappa$ of 0.67 (substantial agreement) with human evaluators. This demonstrates that LLMs can effectively serve as a "first-pass" tool for monitoring reproducibility.

5. Significance

This study highlights a "reproducibility crisis" within the EC community, specifically noting that even when code is provided, it is often poorly documented or difficult to execute. The research moves the conversation from theoretical guidelines to empirical reality.

The introduction of RECAP is particularly significant for the future of scientific publishing; it suggests that conference organizers and journals can use automated, LLM-driven tools to provide consistent, scalable, and objective "reproducibility audits" during the peer-review process, thereby raising the bar for scientific transparency without exponentially increasing the workload of human reviewers.