Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent

The paper introduces RepGen, an intelligent agent-based approach that leverages large language models to automatically reproduce deep learning bugs with an 80.19% success rate, significantly outperforming existing methods and reducing developer effort.

The Gist

Imagine you are a mechanic trying to fix a very strange car. This isn't a normal car; it's a Deep Learning (DL) car. The weird thing about this car is that it doesn't always break the same way. Sometimes it sputters because of the weather (hardware), sometimes because of the fuel mix (data), and sometimes just because the engine decided to be random today (non-determinism).

When a driver calls you and says, "My car makes a weird noise when I turn left," you need to reproduce that noise in your garage to figure out how to fix it. If you can't make the noise happen again, you can't fix it.

For Deep Learning software, this is a nightmare. A study mentioned in the paper says that human developers can only reliably recreate these "noises" (bugs) about 3% of the time. It's like trying to catch a ghost that only appears when the moon is full, the wind is from the north, and you're wearing a red hat.

Enter "RepGen": The Intelligent Detective

The authors of this paper built a tool called RepGen (Reproduction Generator). Think of RepGen as a super-smart, tireless detective who has a magical library and a crystal ball. Instead of guessing, RepGen follows a strict, four-step process to recreate the bug every time.

Here is how RepGen works, using simple analogies:

1. The "Learning-Enhanced Context" (The Detective's File)

When a human gets a bug report, it's often messy. "It crashed!" they say, but they don't say which file, what data, or which version of the software they were using.

• RepGen's Move: Instead of just reading the report, RepGen goes into the project's entire codebase (the garage) and pulls out everything related to that specific problem. It finds the training loops, the data scripts, and the specific libraries. It builds a massive, organized "case file" that connects the dots between the bug report and the actual code.
• Analogy: If a human is looking for a needle in a haystack, RepGen brings the whole haystack, sorts it by color, and hands you the exact section where the needle is hiding.

2. The "Plan" (The Recipe)

Once the detective has the file, they don't just start guessing. They write a recipe.

• RepGen's Move: It breaks the complex task of "recreating the bug" into small, manageable steps. It figures out: "First, install this library. Second, load this specific dataset. Third, run the model with these specific settings."
• Analogy: Instead of saying "Make a cake," RepGen writes a step-by-step recipe: "Preheat oven to 350, mix flour, add eggs..." ensuring no step is missed.

3. The "Generate-Validate-Refine" Loop (The Trial and Error)

This is the magic part. RepGen uses an AI (a Large Language Model) to write the code that tries to recreate the bug. But it doesn't just write it once and hope for the best.

• RepGen's Move:
  • Generate: It writes the code.
  • Validate: It runs the code. Does it crash? Does it give the right error? If not, it asks the AI, "Why didn't it work?"
  • Refine: The AI fixes the code based on the feedback and tries again.
• Analogy: Imagine a chef tasting a soup. If it's too salty, they add water. If it's bland, they add salt. They keep tasting and adjusting until the soup tastes exactly like the customer's complaint. RepGen does this with code, checking for missing imports, wrong settings, or logic errors until the bug "appears" on the screen.

The Results: A Game Changer

The researchers tested this detective on 106 real-world bugs from popular software projects.

• The Old Way: Humans (or simple AI) could only fix about 3% of these bugs.
• The Best Previous AI: Could get about 60% right.
• RepGen: Succeeded 80% of the time.

They also did a study with 27 real developers.

• With RepGen: Developers fixed bugs 23% more often and finished the job in less than half the time (saving about 57% of their time!).
• The Feeling: The developers felt much less stressed and "mentally tired" because RepGen did the heavy lifting of figuring out the missing pieces.

Why Was This Hard Before?

The paper explains that old tools failed because they were looking for the wrong things:

• They looked for GUIs: Old tools tried to click buttons on a screen to reproduce bugs. But Deep Learning bugs happen in the "engine room" (math and data), not on the dashboard.
• They missed the "Silent" bugs: Some bugs don't crash the program; they just make the AI give bad answers (like a GPS sending you to the wrong city). Old tools only looked for crashes. RepGen looks for any wrong behavior.
• They lacked context: Old tools didn't know how the code pieces fit together. RepGen builds the whole puzzle before trying to solve it.

The Bottom Line

RepGen is like a master mechanic who doesn't just guess why a car is making noise. It reads the manual, checks the engine history, writes a perfect test plan, and runs the test over and over, tweaking it until the noise happens again. This allows developers to finally fix the Deep Learning bugs that have been driving them crazy, saving time and reducing frustration.

The paper concludes that while AI is getting smarter, the real secret sauce isn't just a bigger brain—it's giving that brain the right context and a good plan to follow.

Technical Summary

1. Problem Statement

Deep Learning (DL) applications are prone to bugs that are significantly harder to reproduce than traditional software bugs due to three main factors:

• Non-determinism: Randomness in model training (e.g., weight initialization) makes bug manifestation inconsistent.
• Opacity: DL models involve high-dimensional tensor operations and lack interpretability, making root causes difficult to trace.
• Complex Dependencies: Bugs often stem from intricate interactions between hardware (GPUs), frameworks (PyTorch, TensorFlow), data pipelines, and hyperparameters.

Current manual reproduction efforts are inefficient, with studies showing only ~3% of DL bugs can be reliably reproduced manually. Existing automated techniques fail because they:

1. Lack DL-specific contextual understanding (focusing only on code, ignoring training loops, data, and hardware).
2. Rely on GUI-based interactions (record-and-replay), which are irrelevant to headless DL training processes.
3. Struggle with "silent bugs" (e.g., high loss, poor accuracy) that do not cause crashes or exceptions.

2. Methodology: RepGen

The authors propose RepGen, an automated, intelligent agent-based framework designed to systematically reproduce DL bugs. The workflow consists of four main stages:

A. Construction of Learning-Enhanced Context

Instead of feeding the entire codebase to an LLM, RepGen constructs a targeted context:

• Hybrid Retrieval: Uses a combination of BM25 (lexical matching for exact API/error matches) and Approximate Nearest Neighbour (ANN) search (semantic matching for conceptual relationships) to retrieve relevant code snippets from the repository.
• Dependency Resolution: Parses Abstract Syntax Trees (AST) to identify and include necessary imports, helper functions, and module dependencies.
• Training Loop Extraction: Specifically identifies and extracts model training loops (e.g., model.fit(), optimizer.step()) using framework-specific heuristics, distinguishing them from general utility code.
• Ranking: A cross-encoder re-ranks retrieved snippets and training loops based on their relevance to the specific bug report.

B. Bug Report Restructuring

An LLM restructures the raw bug report into a standardized format containing:

• Core Problem Statement: The central issue.
• Observed Behaviour: Stack traces, error messages, and performance metrics.
• Expected Behaviour: Intended system behavior.
• Reproduction Steps: A generated sequence of steps covering environment setup, hardware requirements, and data specifications.

C. Plan Generation

Using the restructured report and the enhanced context, the system generates a comprehensive reproduction plan. This plan breaks the task into modular, verifiable stages, incorporating output assertions, resource monitoring, and error verification steps to guide the code generation process.

D. Iterative Generate-Validate-Refine Agent

A specialized LLM agent (powered by Qwen2.5-Coder-7B) executes the plan through an iterative loop:

1. Code Generation: Generates initial code based on the context and plan.
2. Multi-Stage Feedback:
   • Structural Feedback: AST-based validation to catch syntax errors.
   • Static Analysis: PyLint checks for missing imports, type inconsistencies, and vulnerabilities.
   • Relevance Feedback: Evaluates semantic alignment between the generated code and the bug report to prevent hallucinations.
   • Runtime Feedback (Novelty): The LLM approximates the program state (e.g., "high loss") and maps it to a taxonomy of DL bug symptoms. It compares these symptoms against the bug report. If the similarity score is low, the agent refines the code.
3. Termination: The loop continues until a reproducible code snippet is found or a maximum number of attempts (5) is reached, at which point the system switches to the next available context.

3. Key Contributions

1. RepGen Framework: The first unified, intelligent approach to systematically reproduce DL bugs, addressing non-determinism and silent failures.
2. Agentic Workflow: A novel pipeline combining learning-enhanced context construction, targeted planning, and a feedback-driven agent that iteratively refines code.
3. Comprehensive Evaluation:
   • Evaluation on 106 real-world DL bugs from 16 GitHub projects.
   • Comparison against 8 state-of-the-art LLM baselines (including GPT-4.1, Llama-3, DeepSeek-R1, and Qwen variants).
   • An ablation study analyzing 13 major components.
   • A controlled developer study with 27 participants.
4. Benchmark Dataset: A curated dataset of 106 reproducible bugs with associated artifacts (code, configs) released for future research.

4. Results

• Reproduction Rate: RepGen achieved an 80.19% success rate, outperforming the best baseline (DeepSeek-R1-685B with few-shot prompting at 60.38%) by 19.81%.
• Developer Study Impact:
  • Success Rate: Developers using RepGen's generated code reproduced 23.35% more bugs than those working manually.
  • Time Efficiency: Time to reproduce bugs was reduced by 56.8% (from ~25.7 mins to ~11.1 mins).
  • Cognitive Load: Participants reported a 43.2% reduction in frustration and a 25.6% reduction in mental demand.
• Ablation Findings: The Relevance Feedback component had the highest impact; removing it dropped success rates to ~17.9%. Training Loop Ranking was also critical, with its removal dropping success to ~20.75%.
• Failure Analysis: RepGen struggled primarily with API/Dependency bugs (due to model knowledge cutoffs), Environment-dependent bugs (multi-GPU/distributed setups), and Data-dependent bugs (missing external files). Increasing model size (e.g., using GPT-4.1) helped slightly but did not solve the fundamental issue of missing external context.

5. Significance

• Paradigm Shift: Moves beyond simple "prompt engineering" to a structured, agent-based approach that mimics the systematic debugging process of human experts.
• Reliability: By addressing the "silent bug" problem and non-determinism through runtime symptom approximation, RepGen makes DL systems more trustworthy.
• Practical Utility: The tool significantly lowers the barrier for developers to diagnose and fix DL issues, potentially accelerating the deployment of robust AI systems in critical domains like healthcare and finance.
• Future Direction: Highlights that for DL bug reproduction, context quality (access to specific environment/data details) is more critical than simply increasing model parameter size.