DePro: Understanding the Role of LLMs in Debugging Competitive Programming Code

1. Problem Statement

Debugging is a resource-intensive phase of the software development lifecycle, often consuming nearly 50% of development time. While Large Language Models (LLMs) have shown promise in code generation and algorithmic problem solving, their effectiveness in debugging—specifically correcting existing faulty code rather than generating new solutions—remains under-evaluated.

The authors identify a gap in current research:

• Existing LLM debugging tools often rely on zero-shot prompting (asking the LLM to fix code without specific failure context).
• Competitive programming offers a rigorous testbed due to its strict efficiency requirements, diverse problem domains, and clear pass/fail criteria based on test cases.
• There is a lack of systematic evaluation on how LLMs handle iterative debugging compared to human programmers, particularly regarding the strategies they employ and the number of attempts required.

2. Methodology

The research is conducted in two phases: an initial Empirical/Manual Study to understand LLM behavior, followed by the development and evaluation of the DePro framework.

Phase 1: Empirical & Manual Study

• Dataset: 100 user submissions (10 problems × 10 users each) from Codeforces, focusing on problems where users struggled to debug their solutions.
• Process: The authors prompted ChatGPT-5 (Reasoning model) to debug user code using only the problem statement and the code itself (zero-shot).
• Findings:
  • Efficiency: LLMs required fewer attempts on average (1.8) compared to humans (3.68).
  • Limitations: LLMs struggled with problems requiring unconventional heuristics or multiple valid solutions. They tended to apply localized fixes based on patterns rather than strategic reasoning.
  • Key Insight: LLM performance improved significantly when provided with failing test cases (input + expected output vs. actual output).

Phase 2: DePro Framework

Based on the insight that specific failure data enhances debugging, the authors proposed DePro, an automated, test-case-driven debugging approach.

Workflow:

1. Brute-Force Reference Generation: DePro first prompts the LLM to generate a correct, brute-force reference solution for the problem. This serves as the "ground truth."
2. Stress Testing: The system executes both the user's faulty code and the brute-force reference on a wide range of randomly generated inputs, including edge cases and boundary values.
3. Failure Detection: The system identifies the first input where the user's code output diverges from the reference solution.
4. Iterative Debugging Loop:
   • The LLM is prompted with the failing input, the expected output (from the reference), and the user's code.
   • The LLM suggests a fix.
   • The modified code is re-tested against the stress test suite.
   • This loop continues for up to 8 iterations or until the code passes all tests.

3. Key Contributions

• DePro Framework: A novel, automated debugging pipeline that integrates LLMs with brute-force reference generation and stress testing to create a feedback loop driven by failing test cases.
• Empirical Analysis of LLM Debugging Strategies: A manual study revealing that while LLMs are good at reproducing standard patterns and localized fixes, they lack the strategic flexibility of human programmers for complex or non-canonical problems.
• Demonstration of Test-Case Driven Efficacy: The paper proves that providing specific failing test cases to an LLM significantly outperforms zero-shot debugging approaches.

4. Results

The DePro framework was evaluated on 13 faulty user submissions across 6 different Codeforces problems.

• Success Rate: DePro successfully produced correct solutions for all 13 problems.
• Attempt Reduction: Compared to zero-shot LLM debugging, DePro reduced the number of debugging attempts by an average of 64% (ranging from 50% to 75% reduction).
• Time Efficiency: DePro reduced the average debugging time by 7.6 minutes per problem compared to human manual debugging. The average time to debug a problem using DePro was 6.5 minutes.
• Comparison: DePro consistently outperformed both human programmers and zero-shot LLM approaches in terms of both the number of attempts and total time required.

5. Significance and Future Scope

• Practical Application: The study suggests that integrating execution feedback (failing test cases) is crucial for effective LLM-assisted debugging. This approach can be adapted for software maintenance, automated program repair, and educational tools.
• Limitations: The current approach relies on the ability to generate a correct brute-force solution and effective test cases. This may not scale to problems with massive input spaces or extremely complex logic where a brute-force reference is infeasible.
• Future Directions: The authors propose integrating runtime debuggers (e.g., GDB) and symbolic execution tools (e.g., KLEE) to provide richer execution feedback. They also envision DePro as a teaching assistant or an integrated feature in professional IDEs for real-time analysis and refactoring.

In conclusion, DePro demonstrates that by shifting from generic prompting to a structured, test-case-driven iterative loop, LLMs can become highly effective debugging assistants, significantly reducing the time and effort required to resolve software defects in competitive programming contexts.