Simulated Interactive Debugging

1. Problem Statement

Despite the rise of AI-assisted development, there is a critical gap in debugging education within Computer Science (CS) curricula.

• Current State: Most students learn debugging in an ad-hoc, unstructured manner, relying heavily on "print/log" statements or trial-and-error. They lack systematic training in fault localization and program comprehension.
• Limitations of Existing Tools: While Automated Program Repair (APR) and Intelligent Tutoring Systems (ITS) exist, they primarily focus on generating patches or grading assignments. They rarely guide students through the interactive process of debugging itself.
• The Need: There is a need for a system that acts as a human tutor: it should not simply reveal the solution but guide students step-by-step to understand the defect, identify the fault location, and repair the code themselves.

2. Methodology: Simulated Interactive Debugging

The authors propose a concept called Simulated Interactive Debugging, implemented as a Visual Studio Code (VS Code) extension. The system leverages the fact that in an educational context, the "correct" solution and fault locations are known (via reference implementations and test suites), allowing for a controlled simulation.

Core Architecture & Technologies

• Fault Localization: Uses Spectrum-Based Fault Localization (SBFL) (specifically the FauxPy library) to identify suspicious code lines based on test execution results.
• Large Language Models (LLMs): Integrates OpenAI's gpt-3.5-turbo to power an interactive chatbot. The LLM generates natural language hints, explanations, and Socratic questions based on the context provided by the fault localization engine.
• IDE Integration: The tool is embedded directly into the IDE to provide a seamless workflow.

Key Features

1. Automated Breakpoint Setting: The system automatically sets breakpoints at lines identified as highly suspicious by SBFL, lowering the barrier for students unfamiliar with interactive debuggers.
2. Interactive Chatbot: A guided dialogue where the AI explains failing tests, offers progressive hints (without giving away the full solution), and asks Socratic questions to prompt student reasoning.
3. Variable Highlighting: The system identifies and highlights relevant variables during execution to aid in state comprehension.
4. Test Assessment: A secondary mode for lecturers to evaluate the strength of their test suites (e.g., via mutation testing) and for students to validate custom test cases against reference implementations.

Workflow

1. Input: Student submits buggy code and runs tests.
2. Analysis: The system detects failures, runs SBFL to locate faults, and generates a reference patch.
3. Guidance:
   • The AI explains the failure.
   • It suggests setting breakpoints at specific lines.
   • The student steps through the code.
   • If stuck, the student asks the chatbot for hints, which provide incremental guidance rather than direct answers.

3. Key Contributions

1. Conceptual Framework: Introduction of "Simulated Interactive Debugging," a paradigm that bridges the gap between automated repair and human tutoring by simulating an interactive debugging session.
2. Implementation: A functional prototype as a VS Code extension supporting Python, combining SBFL and LLMs to provide automated breakpoints and conversational guidance.
3. Empirical Validation: A pilot user study demonstrating the feasibility and usability of the approach in a real-world educational setting.

4. Results (Pilot User Study)

The authors conducted a controlled experiment with 8 undergraduate CS students (beginners with <2 years of programming experience and little to no interactive debugging experience).

• Task: Students fixed two medium-difficulty LeetCode-style bugs (maxOperations and longestOnes) within 40 minutes using the tool.
• Success Rate: All 8 participants successfully solved both tasks within the time limit.
• Feature Effectiveness (User Ratings):
  • Most Effective: Automatic Breakpoint Setting (5/8 agreed it was effective). Students found it crucial for getting started.
  • Second: Interactive Debugging & Chatting (4/8).
  • Third: Breakpoint Explanations (4/8).
  • Least Effective: Progressive hints and test error explanations were rated lower (2/8), with some students finding hints too vague.
• Qualitative Feedback:
  • Students appreciated the systematic guidance, contrasting it with their usual "trial and error" or "print statement" methods.
  • The tool helped them "brainstorm" without fully relying on AI to solve the problem for them.
  • Usability: The System Usability Scale (SUS) score averaged 65/100, indicating a good baseline for future iterations.
• Skill Transfer: After the study, 6/8 participants correctly identified where to set breakpoints in a new code snippet without assistance, suggesting the tool helped build intuition.

5. Significance and Future Work

• Educational Impact: This work addresses a critical skill gap in CS education. It moves debugging instruction from "learning by doing" (often inefficient) to "guided learning," fostering deeper program comprehension.
• AI Integration: It demonstrates a responsible way to integrate LLMs into education—using them as scaffolding tools that guide reasoning rather than just providing answers.
• Future Directions:
  • State Machine Guidance: Implementing a state-machine-driven approach to provide more active, structured guidance (similar to Bouzenia et al.).
  • Advanced APR: Integrating state-of-the-art APR techniques for more complex fault localization.
  • Test Generation: Adding features to automatically generate test cases for students struggling with input design.
  • Larger Studies: Expanding the user study to a larger cohort to validate generalizability.

In conclusion, the paper presents a promising approach to modernizing debugging education by combining traditional fault localization with the conversational capabilities of LLMs, effectively simulating the role of a human tutor within an IDE.