Technical Summary: Reprodgen – Replicating Buggy and Correct Data Science Code

Problem Statement

Reproducing reliable data science code from informal developer Q&A sources (e.g., Stack Overflow, GitHub Issues) is a significant challenge. While these forums contain rich discussions on diagnosing and fixing real-world issues, the information is often unstructured, incomplete, and lacks critical execution context. Specifically, posts frequently omit library dependencies, execution environments, functional requirements, and example datasets.

Existing benchmarks for debugging and repair often rely on curated datasets, but constructing such benchmarks directly from informal posts remains difficult. A critical gap exists in the automated reconstruction of executable buggy–patched code pairs. Current state-of-the-art (SOTA) approaches, such as AutoCodeRover and ArchCode, struggle with this task:

• AutoCodeRover assumes a complete repository context; when applied to Stack Overflow posts lacking source files, it fails to retrieve context and produces non-executable patches.
• ArchCode often misinterprets the intent, generating the optimized solution directly rather than reproducing the original buggy behavior, and fails to synthesize necessary mock data.

Without executable buggy code, it is impossible to analyze runtime behavior, validate inferred requirements, or verify whether a generated patch truly resolves the defect. This paper addresses the need for a framework that can semantically faithfully reconstruct both the faulty behavior and the corresponding fix from informal text.

Methodology: The Reprodgen Framework

The authors propose Reprodgen, a Large Language Model (LLM)-based framework designed to automatically generate executable buggy and patched Python programs from Q&A forum posts. The framework employs a Generator–Reviewer workflow, inspired by the Driver–Navigator style of pair programming, to iteratively refine code until it is both executable and semantically consistent with the source discussion.

Core Workflow

1. Q&A Parsing: The system extracts the question, a selected answer (preferably the accepted one), and metadata from the forum post.
2. Structured Thought Generation (Guidance): Before code synthesis, Reprodgen generates three structured intermediate artifacts to guide the process:
   • Code Intent (CI): A high-level semantic specification of the developer's intended functionality, distinguishing between the "Buggy CI" (the goal described in the question) and "Patched CI" (the goal after the fix).
   • Functional Requirements (FR): Structured, user-oriented specifications describing expected behavior, input-output relationships, and edge cases, independent of implementation details.
   • Structured Chain of Thought (SCoT): A step-by-step procedural reasoning description of how inputs are transformed into outputs, modeling the control flow for both the incorrect and correct logic.
   • Note: These artifacts are generated and iteratively reviewed by an independent Reviewer LLM to ensure accuracy and completeness before proceeding to code generation.
3. Code Generation & Execution:
   • The Generator LLM synthesizes runnable code, inferring missing dependencies, synthesizing mock data, and fixing random seeds to ensure reproducibility.
   • A Code Executor runs the generated scripts in isolated Docker containers. It captures standard output, error streams, and exit codes to determine executability.
4. Iterative Refinement:
   • If execution fails or the behavior does not match the source post, the Reviewer LLM evaluates the output against the original Q&A and the generated guidance artifacts.
   • The Reviewer provides feedback (labels: Correct, Partially Correct, Incorrect, Unknown) and rationales.
   • The Generator uses this feedback to refine the code in subsequent iterations (capped at three loops per sub-task).

Evaluation Datasets

The authors constructed two benchmarks to evaluate the framework:

• ReprodgenBench-B: A dataset of 203 Stack Overflow posts (sampled from a curated set of 427) covering seven data science libraries (NumPy, pandas, Matplotlib, scikit-learn, SciPy, DataFrame, PyQt). The ground truth for this dataset was established by two expert human judges validating LLM-synthesized artifacts.
• ReprodgenBench-GI: A dataset of 50 closed GitHub Issues from five libraries, manually reproduced and validated by a human expert to ensure a fully human-validated baseline.

Key Contributions

1. Reprodgen Framework: A novel pair-programming framework that leverages LLMs to generate and review executable buggy and patched code from informal posts, integrating mock data synthesis, dependency inference, and semantic validation.
2. Structured Prompting Approach: A method combining Code Intent, Functional Requirements, and Structured Chain of Thought (SCoT) with iterative assessment feedback to ensure semantic clarity and correctness.
3. ReprodgenBench-V: A publicly released benchmark of 176 real-world data science programs (including the subsets used for evaluation) curated from Stack Overflow and GitHub Issues, promoting reproducibility in automated debugging research.
4. Community Leaderboard: A platform for transparent model comparison based on Execution Rate, Success Rate, and semantic similarity.

Experimental Results

The framework was evaluated across multiple open-source LLMs (Qwen 2.5, Gemma 3, Phi-4, DeepSeek-Coder, Code Llama) on both benchmarks.

• Effectiveness (RQ1):

  • Qwen 2.5 and Gemma 3 demonstrated the strongest overall performance. On ReprodgenBench-B, Qwen 2.5 achieved a Success Rate (SR) of 58.12% for buggy code and 51.72% for patched code.
  • Buggy vs. Patched: Buggy code replication was consistently more reliable than patched code replication. This is attributed to the fact that buggy behavior is often explicitly described or implied in the posts, whereas patch generation requires inferring the correct fix and matching expected post-fix behavior.
  • Dataset Differences: GitHub Issues (ReprodgenBench-GI) showed higher executability rates than Stack Overflow posts, likely due to more structured bug reporting templates providing clearer context.

• Reliability of Reviewers (RQ2):

  • Gemma 3 and Qwen 3 were identified as the most reliable reviewer LLMs, achieving high coverage-weighted F1-scores (0.88 and 0.86, respectively).
  • A performance gap was observed between textual subtasks (CI, FR, SCoT) and code-level evaluation. Reviewers performed well on extracting intent and requirements but struggled significantly with evaluating the correctness of patched code, which requires reasoning over subtle behavioral changes.

• Efficiency (RQ3):

  • Gemma 3 offered the best speed, with average generation times of ~17 seconds for buggy code and ~13 seconds for patched code.
  • Qwen 2.5 required slightly more time but achieved the fewest execution refinement iterations for patched code, indicating stable convergence.
  • Refinement loops were found to be more critical for patched code than for buggy code, where the original logic is often directly available.

Significance and Claims

The paper claims that Reprodgen bridges a critical research gap by unifying mock data synthesis, dependency inference, semantic validation, and executability. Unlike prior systems that either generate functional code without reproducing the bug or produce non-executable patches, Reprodgen reliably generates executable, buggy-patched pairs from informal text.

The authors emphasize that their approach enables:

• Faithful Reproduction: The ability to reproduce the exact faulty behavior described in informal discussions, which is a prerequisite for reliable debugging and automated patch generation.
• Benchmarking: The creation of a high-confidence benchmark (ReprodgenBench) that allows for the systematic evaluation of LLMs in realistic debugging scenarios.
• Insight into LLM Limitations: The study reveals that while LLMs excel at textual understanding and intent extraction, they still struggle with code-level analysis, dependency inference, and the generation of semantically correct patches without iterative refinement.

The paper concludes that Reprodgen provides a reproducible and extensible foundation for future research in LLM-based bug reproduction and repair for data science programs, while acknowledging that challenges remain in handling ambiguous contexts and ensuring perfect semantic alignment without human intervention.