💻 computer science

NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming

This paper introduces NL2Scratch, a large-scale executable benchmark and the Semantic Alignment Consistency (SAC) metric to address the limitations of conventional NL2Code evaluation in block-based programming, revealing that current LLMs often achieve high lexical similarity while failing to capture the necessary semantic alignment for generating correct Scratch programs.

Original authors: Heejin Do, Alexandre Ballenghien, Yang Wu, April Yi Wang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Heejin Do, Alexandre Ballenghien, Yang Wu, April Yi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to draw a picture based on your spoken instructions. If you say, "Draw a red circle," the robot needs to understand exactly what "red" and "circle" mean and how to put them together.

This paper is about a new test called NL2Scratch designed to see how well AI can do this, but with a twist: instead of writing text code (like Python), the AI has to build programs using Scratch blocks. Scratch is the colorful, drag-and-drop programming language kids use to make games and animations.

Here is the breakdown of the paper's story, using simple analogies:

1. The Problem: The "Looks Good" Trap

For years, researchers have tested AI on text-based coding. They check if the AI's answer looks similar to the correct answer (like checking if two sentences share the same words).

But Scratch is different. It's like a Lego set.

In text coding, if you miss a comma, the whole sentence might break.
In Scratch, you can have the right types of blocks (a "move" block, a "repeat" block) and the right words inside them, but if you put them in the wrong order or connect them to the wrong trigger, the game won't work.

The paper argues that current AI tests are like judging a Lego castle just by counting the bricks. If the AI uses the right number of red and blue bricks, the test says "Great job!" even if the castle falls over because the bricks are stacked upside down.

2. The Solution: A New Test and a New Ruler

The authors built NL2Scratch, a massive library of 311,000 examples.

The Source: They took real Scratch projects made by humans.
The Translation: They used AI to write natural English descriptions for these projects (e.g., "When the green flag is clicked, make the cat move 10 steps").
The Filter: They made sure every single example actually works in the Scratch engine.

The New Ruler (SAC):
Instead of just counting matching words, they invented a new measuring stick called Semantic Alignment Consistency (SAC).

Think of it like a checklist.
Does the description mention the right trigger (like the green flag)?
Does it mention the right action (like moving)?
Does it have the right numbers (like 10 steps)?
SAC checks every single item on this list. If the AI gets the "trigger" right but the "number" wrong, the checklist shows a red X, even if the rest of the sentence looks perfect.

3. The Big Discovery: AI is Good at Mimicry, Bad at Meaning

The researchers tested several smart AI models (like GPT-4 and open-source models) on this new test. Here is what they found:

The Illusion of Success: When using old-school tests (counting words), the AI looked amazing. It got 93% to 97% of the words right. It sounded like it knew exactly what to do.
The Reality Check: When they used the new SAC checklist, the AI's score dropped dramatically. Only about 18% of the AI's answers were perfectly correct in meaning.
The "Long Game" Problem: The longer and more complex the Scratch project was, the worse the AI got at getting the details right, even though it still sounded very similar to the correct answer.

Where did the AI fail?
The AI was great at the "big picture" (knowing it needs a "loop" or a "variable"). But it kept messing up the operational details:

It would say "move forward" when it should say "move backward."
It would say "repeat 10 times" when it should say "repeat 5 times."
It would get the condition wrong (e.g., "if touching the wall" vs. "if touching the cat").

It's like a student who memorized the vocabulary list perfectly but failed the math problem because they added instead of subtracted.

4. The Fix: A "Second Opinion"

The paper also showed that you can use this new checklist (SAC) to fix the AI's mistakes after it generates an answer.

Imagine the AI writes 10 different versions of the program.
Instead of just picking the first one, you run all 10 through the SAC checklist.
You pick the one that matches the checklist best.
Result: This simple step improved the AI's accuracy significantly without needing to retrain the AI or teach it anything new.

Summary

The paper concludes that for block-based programming (like Scratch), looking similar isn't enough. An AI can sound like a programmer without actually being one. To truly evaluate AI in this field, we need to check the "gears and wheels" (the specific actions and numbers), not just the "paint job" (the words). They have built the tools to do exactly that.

Technical Summary: NL2Scratch

Problem Statement

Natural-language-to-code (NL2Code) research has predominantly focused on sequential, text-based programming languages (e.g., Python, SQL, Java), relying on evaluation metrics such as token-level lexical overlap (BLEU, CodeBLEU) or unit-test execution. However, block-based programming environments like Scratch, widely used in K-12 education, operate under fundamentally different paradigms: programs are event-driven, visually compositional, and distributed across multiple concurrent scripts.

In this context, conventional evaluation protocols are insufficient. A Scratch program with high surface-level token similarity to a reference may still fail to execute or behave incorrectly due to misaligned event triggers, incorrect control flow, or mismatched arguments. Furthermore, existing AI research in Scratch education has largely focused on conversational tutoring, debugging, or program understanding, lacking a benchmark for executable natural-language-to-Scratch generation that validates both structural correctness and semantic fidelity.

Methodology

1. NL2Scratch Benchmark Construction

The authors introduce NL2Scratch, an executable benchmark comprising 311,648 parser-valid Natural Language (NL)–program pairs. The construction pipeline involves three stages:

Scratchblocks Extraction: Real Scratch projects (from the Aivaloglou et al. corpus) are converted into scratchblocks, a normalized textual notation that preserves event triggers, block ordering, and control-flow hierarchy. Scripts are validated using the Scratch VM parser to ensure structural validity.
Natural Language Generation: For each valid program, a deterministic description is generated via a rule-based converter. This is subsequently rewritten by an LLM to improve fluency and "student-like" naturalness while strictly preserving executable semantics (events, actions, conditions, numeric arguments).
Semantic Verification: A two-stage filtering process ensures data quality. First, parser-based checks confirm structural validity. Second, Semantic Alignment Consistency (SAC) is applied to verify that the rewritten NL description semantically matches the underlying program.

2. Semantic Alignment Consistency (SAC)

To address the limitations of surface-overlap metrics, the authors propose SAC, an interpretable, slot-level verification framework.

Slot Extraction: SAC extracts 11 behavioral slots from both the NL description and the Scratch program (represented as scratchblocks). These include:
- Event triggers (e.g., green flag, key press).
- Actions (e.g., motion, variable updates).
- Control structures (loops, conditionals).
- Operational details (numeric arguments, variable names, costume/backdrop names, broadcast messages).
Scoring: Agreement is computed per slot. Scalar attributes require exact matches; set-valued attributes use Jaccard overlap. The final SAC score is the average across all comparable non-empty slots.
Thresholding: Pairs with $SAC \geq 0.85$ are considered high-confidence aligned.

3. Diagnostic Benchmark

From the full dataset, the authors constructed a controlled 800-example diagnostic benchmark. This subset was curated to maximize diversity across Scratch-specific semantic attributes (event types, loop types, numeric arguments, etc.) and to ensure readability distributions match real-world learner prompts. This set serves as the primary evaluation ground for model performance analysis.

4. Experimental Setup

The study evaluates a range of models, including proprietary (GPT-4.1, GPT-4o-mini) and open-weight (FLAN-T5, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct) LLMs.

Training Strategies: Models are tested under zero-shot, few-shot (retrieval-based in-context learning), and supervised fine-tuning (SFT) conditions.
Metrics: Performance is measured using conventional metrics (Exact Match, Token-level F1, Parse Success) and the proposed SAC metrics ( $SAC_{nl}$ for alignment with input NL, $SAC_{ref}$ for alignment with the reference program).
Inference-Time Selection: The authors investigate using SAC as a reranking signal to select the most semantically faithful candidate from a set of generated programs.

Key Results

1. The Gap Between Lexical Similarity and Semantic Alignment

The experiments reveal a significant discrepancy between conventional metrics and semantic correctness.

Models frequently achieve high token-level F1 scores (often >0.93) and high parse success rates.
However, perfect semantic alignment ( $SAC = 100\%$ ) is achieved on fewer than 18.9% of examples even for the strongest models (e.g., GPT-4.1 with few-shot prompting).
This indicates that high lexical similarity does not guarantee that the generated program faithfully reproduces the intended behavior.

2. Error Concentration in Operational Slots

Slot-level analysis identifies specific failure modes:

Structural Scaffolding: Models perform well on structural and entity-centric slots, such as event types, variable names, and costume names, achieving near-ceiling alignment.
Operational Details: Errors concentrate on slots that directly determine program behavior, specifically action types, condition types, and numeric arguments. These errors often render the program functionally incorrect despite passing parser checks and achieving high F1 scores.

3. Impact of Program Complexity

The gap between lexical similarity and semantic alignment widens as program complexity increases.

For longer programs, Exact Match (EM) and perfect semantic alignment degrade sharply.
Token-level F1 remains relatively stable, suggesting that local token recovery is easier than preserving global executable structure and complex inter-block dependencies.

4. Utility of SAC for Reranking

Using SAC as an inference-time selection signal significantly improves generation quality without retraining.

SAC-based reranking increased perfect $SAC_{ref}$ alignment by up to 32.6 percentage points for open-weight models.
This demonstrates that explicit semantic verification is a more effective selection signal than likelihood-based or parser-only ranking.

Significance and Claims

The paper claims that block-based programming constitutes a distinct NL2Code regime where traditional evaluation metrics are insufficient.

First Executable Benchmark: NL2Scratch is presented as the first benchmark providing large-scale, executable NL–Scratch pairs with aligned quality verification derived from real projects.
Revealing Hidden Failure Modes: The study demonstrates that current robust LLMs, even when fine-tuned, exhibit a "notable gap" between surface similarity and structural accuracy. These failures are largely invisible under conventional metrics but critical for educational applications where behavioral correctness is paramount.
Evaluation Framework: SAC is proposed not just as a dataset filter but as a necessary diagnostic tool for evaluating semantic agreement in event-driven, block-based systems.

The authors conclude that future work in this domain must move beyond lexical overlap and parser validity toward structure-aware generation and evaluation methods that explicitly verify operational semantics.