LitterBox+: An Extensible Framework for LLM-enhanced Scratch Static Code Analysis

Technical Summary: LitterBox+ Framework

Problem Statement
While Large Language Models (LLMs) have become essential for supporting developers in text-based programming, their application to the Scratch block-based environment is hindered by the graphical nature of Scratch code. Although LLMs can assist in fixing and improving code for text-based languages, the block-based notation of Scratch prevents direct integration. Existing approaches for Scratch and LLMs often focus on creative aspects (e.g., generating sprite images or collecting feature ideas) rather than targeted code analysis. Furthermore, static analysis tools like LitterBox can detect bugs based on recurring patterns of misconceptions but typically provide only generic descriptions of issues, leaving learners without specific guidance on how to reproduce errors or implement fixes.

Methodology and System Architecture
The authors propose LitterBox+, an extensible framework that bridges this gap by integrating the static code analysis capabilities of LitterBox with the generative abilities of LLMs. The system architecture operates through the following mechanisms:

1. Code Conversion: The framework converts Scratch block-based code into a textual representation known as scratchblocks. This format is chosen because it is common in community forums and likely present in LLM training data, unlike Scratch's internal JSON representation. The conversion includes special comments with unique IDs for sprites and scripts to facilitate later correlation.
2. Prompt Engineering: When a user requests assistance, the system constructs a prompt combining the specific issue (e.g., "Missing Loop"), the generic description from LitterBox, and the relevant scratchblocks code context.
3. LLM Interaction: The framework supports various LLM providers (e.g., OpenAI, Ollama) via a generic interface. It utilizes langchain4j for managing these interactions. The system is designed to be LLM-agnostic, though the UI refers to models generically as "GPT" for user familiarity.
4. Parsing and Integration: A custom ScratchBlocks Parser was developed to interpret LLM outputs.
   • For explanations, the LLM's natural language response is appended to the original issue description.
   • For fixes, the LLM outputs scratchblocks code. The parser attempts to convert this back into the program's Abstract Syntax Tree (AST). If parsing fails due to syntactic errors (e.g., incorrect block usage or missing markers), a post-processing step applies heuristic rules, followed by up to three iterative attempts to prompt the LLM for a syntactically correct script.
5. User Interfaces:
   • Scratch-GUI Extension: A toggleable panel integrates directly into the Scratch web interface, allowing users to view LitterBox issues, request "Explain!" or "Fix!" actions, and ask general questions about specific sprites or the whole program.
   • CLI and API: A command-line interface and Java API allow for batch processing, programmatic access, and integration into other tools.

Key Contributions

• LitterBox+ Framework: An open-source extension to the existing LitterBox static analysis tool that enables LLM-driven interaction with Scratch programs.
• ScratchBlocks Parser: A novel parser capable of converting LLM-generated scratchblocks text back into the internal AST of a Scratch program, allowing for direct code modification and integration.
• Extensibility: The framework is designed to be easily extended with new prompt templates, LLM providers, and analysis features. It supports both self-hosted and externally hosted models.
• Dual Interface: The provision of both a user-friendly GUI extension for learners and a CLI/API for researchers and advanced users.

Results and Demonstrations
The paper presents a functional implementation of the framework, demonstrated via a screencast. Key operational results include:

• Contextual Explanations: The system successfully generates detailed, program-specific explanations for generic LitterBox issues, including steps to reproduce the bug and its effect on sprite behavior.
• Automated Fixes: The framework can propose and integrate code fixes directly into the Scratch project. The parser handles standard blocks, custom blocks, and the "Pen" extension.
• Error Handling: The system includes robust fallback mechanisms for LLM hallucinations or syntax errors, such as iterative prompting and heuristic post-processing, to ensure that only valid code is integrated.
• User Feedback: Early trials indicate that while LLMs can reason about Scratch code, they occasionally produce misleading explanations or incorrect fixes. Consequently, the UI includes prominent warnings (e.g., red exclamation marks) to alert learners that LLM-generated content may be inaccurate.

Significance
The paper claims that LitterBox+ serves as a foundational, extensible platform for researchers to build creative integrations between Scratch and LLMs. By combining established static analysis with generative AI, the framework aims to provide learners with specific, actionable feedback that generic static analysis tools cannot offer. The authors emphasize that the open-source nature of the project (available via GitHub) is intended to encourage further development of Scratch+LLM integrations beyond the features currently presented, such as code completion and agentic testing capabilities. The work positions itself as a step toward overcoming the graphical barrier that has previously limited the application of LLMs in block-based programming education.