💻 computer science

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

This paper empirically evaluates four large language models and five embedding approaches for Scratch programs, demonstrating that models trained on large open datasets can effectively capture structural and semantic information to support learning analytics tasks like predicting functional correctness in small classroom settings without requiring further fine-tuning.

Original authors: Benedikt Fein, Gordon Fraser

Published 2026-07-22

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Benedikt Fein, Gordon Fraser

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to understand human language. You feed it millions of books, movies, and websites, and eventually, it learns to predict the next word in a sentence with amazing accuracy. This is the magic of Large Language Models (LLMs). Now, imagine you want that same robot to understand source code—the instructions programmers write to tell computers what to do. Since code looks a bit like a strange, structured language, researchers have found that these robots can learn to understand it too, helping developers write software faster or even fixing bugs automatically.

But here's the twist: not all code looks like text. In schools, many kids learn to program using Scratch, a colorful, block-based system where you snap puzzle pieces together on a screen instead of typing lines of text. It's like building with Lego bricks rather than writing a story. The big question for scientists is: Can the same "robot brains" that read text-based code also understand these visual block puzzles? And if they can, which method works best? This matters because if we can teach robots to understand Scratch, we could build tools to help teachers grade homework, spot struggling students, or give instant hints, making learning to code much smoother for the next generation.

The Great Code Translator Challenge

In this study, researchers Benedikt Fein and Gordon Fraser set out to answer a tricky question: Can we teach computers to "understand" Scratch programs well enough to help in a classroom? They didn't just guess; they built a digital playground to test different kinds of "code translators."

Think of a code translator as a way to turn a complex Scratch program into a single, secret number (or a "vector") that captures the program's essence. If two programs are similar, their secret numbers should be close together. If they are different, the numbers should be far apart. The researchers wanted to see which translator was the best at this game.

The Contenders

They lined up five different types of translators to compete:

The "Flat" Readers: These models treat the code like a simple list of words, ignoring the structure. It's like reading a recipe but ignoring the fact that some steps happen inside a "mixing bowl" and others on the "stove."
The "Tree" Readers: These models look at the code's structure, like a family tree of instructions. They understand that some blocks are nested inside others.
The "Flow" Readers: These are the most advanced. They don't just see the structure; they track how information moves through the program, like watching water flow through pipes. They know if a variable changes in one block and is used in another.
The "Big Brains" (LLMs): These are the giant, pre-trained AI models (like the ones that power chatbots) that have already read millions of text documents. The researchers asked them to look at Scratch code and guess what it was doing.

The Test: Naming the Characters

To test these translators, the researchers created a game called "Sprite Naming." In Scratch, every character (or "sprite") has a name. The task was simple: show the model the code inside a sprite, and ask it to guess the sprite's name. It's like showing someone a character's diary and asking, "What is this character's name?"

The results showed a clear hierarchy. The winners were the "Flow" readers (specifically a model called GGNN). By paying attention to how the blocks connect and how data moves between them, these models were much better at understanding the "story" of the code. They guessed the sprite's name with significantly higher accuracy than the others.

The giant "Big Brain" models (LLMs) and the "Flat" readers performed poorly on this specific naming task. They struggled to guess the correct names because Scratch programs rarely use the fancy variable names or custom functions that these big models are used to seeing in text-based code. Instead, Scratch relies heavily on the structure of the blocks and how they connect. However, it is important to note that even the winning "Flow" model didn't get it perfect; it still made mistakes, meaning no model was a flawless oracle. The LLMs weren't the only ones failing to achieve perfect scores, but they were consistently outperformed by the models that understood the program's structure and flow.

The Language Barrier

The researchers also tested if the language the students used mattered. Scratch is used all over the world, in Brazil, China, Poland, and many other places. They found that the models worked best when the code was written in English or other languages using the Latin alphabet (like A, B, C). When the code used other writing systems (like Chinese characters or Russian Cyrillic), the models got confused. This suggests that for these tools to work globally, they need to get better at understanding different languages and symbols, not just English.

From Naming to Grading: Can We Use This in Class?

The real magic happened next. The researchers asked: "If these models are good at naming sprites, can they also help teachers?"

They tested two big ideas:

Checking for Correctness: Can the model look at a student's messy program and tell if it works, even without running it? They found a strong link between the model's "understanding" of the code and whether the code actually passed tests. If the model thought the code was "close" to the solution, the code usually worked. This means these models could act as a "surrogate" to check homework instantly.
Tracking Progress: Can the model show how much a student has improved? They used a special map to plot student progress. As students added features to their games, their "code map" moved closer to the final solution. The models successfully tracked this journey, showing teachers exactly where a student was stuck or how far they had come.

Interestingly, for these broader tasks (like classifying the type of project or tracking progress), the LLMs actually performed quite well, sometimes even matching or beating the specialized structural models. This suggests that while LLMs might miss the tiny structural details needed for naming a specific character, they are very good at understanding the "big picture" of a project.

The Verdict

The study concludes that yes, it is possible to use code embeddings for Scratch, but you have to choose the right tool for the job. For tasks requiring deep structural understanding (like naming a sprite), specialized models that understand the flow and structure of the blocks are the champions. The giant chatbots (LLMs) aren't the best fit for those specific puzzles because they miss the structural details that Scratch relies on.

However, for broader educational tasks like grading or tracking progress, the LLMs are surprisingly effective and can be just as good as the specialized models. Even better, these specialized models, once trained, can be used for other tasks—like grading or tracking progress—without needing to be retrained for every single new assignment. This is a huge win for teachers, as it means they could have powerful, automated tools to help students learn, even in small classrooms where there isn't enough data to train a new AI from scratch.

In short, the researchers found a way to give computers a "sense" of Scratch code that goes beyond just reading words. By understanding the shape and flow of the blocks, these models can help us build a smarter, more supportive future for programming education.

Technical Summary: EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

Problem Statement

The proliferation of code embeddings and Large Language Models (LLMs) has transformed software engineering and educational analytics, particularly for text-based languages like Python and Java. However, their application to visual, block-based programming environments like Scratch remains an open question. Scratch presents unique challenges:

Structural Differences: Scratch code consists of freely placeable visual blocks rather than linear token sequences, and relies heavily on concurrent execution and event-driven messaging between "sprites," which standard text-based embeddings may fail to capture.
Linguistic Diversity: The Scratch community is highly international, with users employing diverse natural languages for identifiers, comments, and UI elements. Standard models often assume English-centric identifiers.
Data Scarcity for Education: Common code embedding benchmarks (e.g., method naming, documentation generation) are infeasible for Scratch due to the lack of comments and documentation in student projects. Furthermore, classroom datasets are typically too small for training task-specific models from scratch.

The core problem is determining which code embedding approach is most suitable for learning analytics in Scratch, specifically whether models can capture structural and semantic information without extensive fine-tuning in data-scarce educational settings.

Methodology

The authors, Benedikt Fein and Gordon Fraser, conducted an empirical evaluation involving four research questions (RQs) to assess the feasibility and performance of various embedding approaches.

1. Data Preprocessing and Representation

The authors utilized LitterBox to parse Scratch projects (JSON format) into Abstract Syntax Trees (ASTs). They implemented EmbeddedKittens, a preprocessing tool, to convert Scratch code into formats suitable for different model architectures:

Flat Token Sequences: Used for NeuralCodeSum (Transformer) and LLMs (GPT-5-mini, DeepSeek Coder v2, Mistral Small 3.1, Gemma 3). The AST was traversed depth-first to create a textual representation (e.g., motion_movesteps(10)). For LLMs, the ScratchBlocks format was used, as it aligns better with their pre-training data.
AST Paths: Used for code2vec and code2seq, representing code as sets of paths between AST leaf nodes.
Statement Subtrees: Used for ASTNN, splitting the AST into per-statement subtrees encoded by a bidirectional GRU.
Control/Data Flow Graphs: Used for GGNN (Gated Graph Neural Network). This approach extended the AST with semantic edges representing data flow (LastRead, LastWrite, ComputedFrom), control flow (GuardedBy), and Scratch-specific event handling (message broadcasting between sprites).

2. Experimental Tasks

RQ1 (Sprite Naming): A baseline task where models predict a sprite's name based on its code. This serves as a code summarization proxy, as Scratch projects lack method names or documentation.
RQ2 (Internationality): Evaluation of model performance across 10 different natural languages (including non-Latin scripts like Chinese, Japanese, and Russian) to assess the impact of linguistic diversity.
RQ3 (Program Classification): Fine-tuning pre-trained sprite embeddings to classify whole programs by category (e.g., game, animation) or by their base project (remix classification).
RQ4 (Correctness Estimation): Using embeddings as a surrogate model to estimate the functional correctness of student programs by measuring the distance to a "model solution" in embedding space, compared against actual test suite results.
RQ5 (Progress Tracking): Applying the "progress-variance-projection" technique to visualize student progress over time using the embeddings, mapping the distance from a starter project to a solution.

3. Models Evaluated

Dedicated Embedding Models: NeuralCodeSum, code2vec, code2seq, ASTNN, GGNN.
Pre-trained LLMs: GPT-5-mini, DeepSeek Coder v2, Mistral Small 3.1, Gemma 3, and Qwen 3 Embedding.

Key Contributions

Systematic Evaluation: The first structured comparison of five different code embedding approaches and four LLMs specifically for the Scratch domain.
Sprite Naming Task: The introduction of a novel "sprite naming" task to serve as a baseline for code summarization in Scratch, overcoming the lack of traditional benchmarks.
Multilingual Analysis: The creation and evaluation of datasets across 10 distinct natural languages, highlighting the performance disparities in non-Latin script environments.
Educational Applicability: Demonstration that pre-trained embeddings can be transferred to educational tasks (correctness estimation, progress tracking) without task-specific fine-tuning, addressing the data scarcity problem in classrooms.
Open Source Tooling: The release of EmbeddedKittens, a preprocessing tool for converting Scratch code into various embedding-ready formats.

Results

RQ1: Code Summarization (Sprite Naming)

Structural Information is Critical: Models incorporating structural and control-flow information significantly outperformed those using flat token sequences. GGNN achieved the best performance (F1 = 0.550), followed by ASTNN (F1 = 0.342).
LLM Limitations: Pre-trained LLMs performed poorly (F1 ≤ 0.081) on the specific task of generating sprite names, likely because Scratch lacks the custom procedures and variable names that LLMs typically rely on for inference in text-based languages.
Metric Discrepancy: While GGNN achieved high BLEU scores, its ROUGE-2 scores were lower, suggesting it often generated correct subtokens but in incorrect orders or with repetitions.

RQ2: Internationality

Language Bias: Models performed best on English and other Latin-based languages. Performance dropped significantly for non-Latin scripts (e.g., Korean, Russian, Chinese).
Tokenization Issues: The authors attribute poor performance on non-Latin scripts to the subtokenization strategy (splitting on whitespace/camel case), which is ineffective for scripts without these delimiters.
Data Scarcity: Low performance in languages with fewer Scratch users (e.g., Korean) was linked to the scarcity of training data in those specific languages.

RQ3: Program Classification

Transferability: Pre-trained sprite embeddings (specifically GGNN and Qwen 3 Embedding) could be effectively fine-tuned for program classification.
LLM Performance: Surprisingly, Qwen 3 Embedding (an LLM-based approach) outperformed the dedicated GGNN model in program category classification (F1 = 0.562 vs. 0.475), suggesting that for high-level categorization, semantic similarity captured by LLMs is sufficient, even if structural details are less critical.
Robustness: Both GGNN and LLM-based embeddings formed robust clusters, correctly identifying remixes of base projects with high accuracy (>97%).

RQ4 & RQ5: Correctness and Progress

Surrogate Validity: A statistically significant linear correlation was found between embedding-based fitness (distance to solution) and actual test suite fitness. This confirms that pre-trained embeddings capture semantic and structural information relevant to correctness.
Progress Visualization: The embeddings successfully mapped student progress in a 2D space. GGNN and Qwen-whole (LLM processing the whole program) showed strong correlations with actual progress ( $R^2 \approx 0.7$ ), whereas per-sprite aggregation with LLMs was less effective.
No Fine-Tuning Required: Crucially, these results were achieved using models pre-trained only on the sprite naming task, without further fine-tuning for correctness or progress, validating their utility in low-data classroom settings.

Significance and Claims

The paper claims that code embeddings are feasible and effective for Scratch, provided the models account for the language's structural complexity.

Complementary Roles: The authors propose a complementary usage model:
- Dedicated Structural Models (e.g., GGNN): Best suited for tasks requiring deep structural insights, such as bug detection or next-step hint generation, and can be deployed locally with low resource overhead.
- Pre-trained LLMs: Best suited for high-level tasks involving semantic similarity, such as project recommendation or progress estimation, leveraging their broad pre-training but requiring more computational resources.
Educational Impact: The study demonstrates that pre-trained embeddings can serve as "surrogate models" for learning analytics in environments where collecting large, labeled datasets for task-specific training is impossible. This enables tools like automated grading, progress tracking, and exercise difficulty estimation in typical classroom settings.
Future Directions: The authors modestly suggest that future work should explore improved tokenization for non-Latin scripts, the integration of visual information (block positions), and the adaptation of these concepts to other event-driven systems.

The paper concludes that while the transfer of code embeddings to Scratch is successful, the choice of model depends heavily on the specific educational application and the available computational resources.