An Agentic Approach Towards Replication Package Quality Evaluation

Technical Summary: An Agentic Approach Towards Replication Package Quality Evaluation

Problem Statement
Reproducibility in empirical software engineering relies on the availability and quality of research artifacts (source code, datasets, scripts, etc.). However, the evaluation of these replication packages remains a largely manual, time-consuming, and non-scalable process. Existing guidelines and checklists (e.g., from conferences like ICSE or OSDI) define what should be reported but offer limited support for automatically assessing heterogeneous artifact types against these criteria. Consequently, reviewers face substantial overhead, and large-scale reproducibility assessments are hindered by inconsistent evaluation depths and the lack of standardized, machine-verifiable procedures.

Methodology
The authors propose an agentic approach to operationalize open-science guidelines into machine-verifiable criteria for automated artifact assessment. The methodology is structured around three research questions (RQs) concerning the assessment of reproducibility potential, the identification of strengths/limitations, and the perceived usefulness to stakeholders.

1. Criteria Operationalization: The team analyzed 34 sources (including top-tier conference guidelines, journal policies, and community best practices) to extract 380 raw requirements. These were consolidated into 51 hierarchical reproducibility criteria. To enable automated reasoning, 31 of these criteria were enriched with structured auxiliary fields: additional_information, input_needed, detection_strategy, and success_failure_conditions. These were stored in an SQLite knowledge base to support traceability and retrieval.
2. Agentic System Architecture: The system is implemented as a stateful, graph-based workflow using LangGraph, consisting of five phases:
   • Ingestion & Embedding: Converting input papers (PDF to Markdown) and artifacts into vector databases for Retrieval-Augmented Generation (RAG).
   • Artifact Retrieval & Analysis: A sub-agent extracts artifact references (e.g., GitHub, Zenodo). If automatic retrieval fails, a Human-in-the-Loop (HITL) interrupt requests user guidance. Retrieved READMEs are summarized, and irrelevant files (binaries, logs) are pruned via rule-based and LLM-based classifiers.
   • Plan Generation: Worker sub-agents generate inspection steps for user-selected subtopics, synthesized into a hierarchical strategy. This plan undergoes a HITL loop for user refinement and approval.
   • Subtopic Evaluation: An orchestrator dispatches sub-agents to evaluate criteria in isolated contexts, gathering evidence via hybrid retrieval (lexical and semantic). Evaluations require explicit citations (file paths, code snippets) and default to "Fail" if evidence is missing.
   • Report Compilation: Findings are aggregated into a hierarchical Markdown report with execution metadata and trace IDs.
3. Agent Configuration: The system employs five specialized agents (retrieval, plan, orchestrator, executor, reporting) utilizing various GPT-5 variants (e.g., gpt-5-nano, gpt-5-mini, gpt-5.1) with a temperature of 0 to maximize determinism.

Key Contributions
The paper presents three primary contributions:

1. Operationalized Criteria: A unified set of 51 machine-verifiable reproducibility criteria derived from 34 sources, with 31 specifically operationalized for automated, artifact-based evaluation.
2. Agentic Evaluation Framework: A novel multi-agent prototype that autonomously retrieves, structures, and evaluates research artifacts against these criteria, producing evidence-grounded improvement reports.
3. Pilot Empirical Evidence: Preliminary results from a technical evaluation and a survey of seven software engineering researchers demonstrating the approach's potential and current limitations.

Results
The evaluation was conducted on five replication packages and a pilot survey with seven researchers.

• Reliability and Consistency: The system demonstrated high inter-run consistency of 91.4% across 10 independent executions per package. Stability was highest in structured domains (Data/Artifact Availability: 94.8%; Ethical/Governance: 94.0%).
• Agreement with Manual Baseline: Compared to a manual ground-truth assessment, the agent achieved a 75.4% micro-accuracy and 68.2% macro-accuracy.
  • The agent showed high precision (86.4%) and recall (81.5%) for identifying "Passed" criteria.
  • For "Failed" criteria, recall was high (85.8%) but precision lower (56.7%), indicating a bias toward strictness (defaulting to fail when evidence is unclear).
  • Performance varied significantly by topic: 84.6% for Code/Environment and 77.2% for Data/Availability, but only 38.0% for Experimental Rigor.
• Qualitative Analysis: The agent performed well on structural criteria (code, environment, availability) but struggled with qualitative or mixed-method studies and conditional criteria (e.g., distinguishing between training new models vs. re-running existing code).
• User Perception: Survey participants rated the tool's clarity and integration positively (Mean ~3.83/5) and expressed confidence in using it to improve artifacts (Mean ~3.67/5). However, users noted a cognitive load during the HITL planning step, finding the level of required detail overwhelming for new users.

Significance and Claims
The paper claims that agentic research artifact evaluation has the potential to support authors and reviewers by automating selected routine checks, shifting the process from a retrospective, manual burden to a scalable, proactive workflow.

• Automation of Routine Checks: The approach successfully operationalizes broad open-science rules into automated checks, particularly for computational and code-centric research.
• Evidence-Based Transparency: Unlike opaque end-to-end models, the agent grounds evaluations in extracted evidence with explicit file-path citations, allowing for independent auditing of reasoning.
• Scalability: The architecture handles the heterogeneity of replication packages (diverse formats and structures) reasonably well, offering a path toward large-scale reproducibility assessments.
• Limitations: The authors modestly acknowledge that the current approach is biased toward quantitative, code-centric research and struggles with qualitative studies and unconventional artifact layouts. They also note that the reliance on commercial LLMs and the cognitive load of the HITL planning step are areas for future improvement.

The authors position this work as an "emerging results" paper, providing preliminary evidence that such systems are feasible and valuable, while highlighting the need for further refinement in handling complex formats, improving adaptability to different study types, and streamlining the user interface.