BoilerSketch: A TA-Supervised, Diagram-First GenAI Practice for Structured Diagrams in CS1/Early CS2

Technical Summary: BoilerSketch

Problem Statement
Large introductory computing courses (CS1 and early CS2) face a support bottleneck during labs and office hours. While students require timely, individualized assistance, many of their questions—particularly those involving recursion traces, tree traversals, pointer aliasing, and dynamic programming tables—are best answered with visual diagrams rather than text. Existing AI tutoring tools in computing education are predominantly text-forward or code-forward; they generate exercises, explain code, or debug through dialogue but lack the ability to produce accurate, pedagogically useful visuals. Conversely, classic program visualization systems offer strong visual representations but are rarely conversational, multimodal, or embedded within a supervised help workflow. This creates a gap between systems that visualize program behavior and systems that discuss it, leaving a need for a "diagram-first" AI support tool that is explicitly constrained to conceptual explanation and situated under human supervision.

Methodology and System Design
The authors present BoilerSketch, a TA-supervised, diagram-first GenAI practice designed for early computing support. The system is not an autonomous tutor but a bounded instructional practice comprising four core components:

1. Dual-Pane Tablet Interface: Students interact via a tablet supporting both natural language chat and a pen-enabled whiteboard. Students can submit text questions, sketch structures or traces, or upload images of hand-drawn diagrams. The system supports iterative, bidirectional visual-textual dialogue.
2. Prompt-as-Policy Architecture: The system employs a two-layer prompting strategy to enforce pedagogical boundaries.
   • Base Instructional Prompt: Defines the model's role as a concise teaching assistant, forbidding executable code, direct debugging, and course complaints. It mandates a focus on conceptual unpacking and independent problem solving.
   • Visualization Subprompt: Invoked only when a visual explanation is warranted, this prompt constrains the model to generate structured Mermaid diagram syntax rather than free-form images.
3. Structured Rendering Pipeline: To ensure reliability and inspectability, the system does not generate raw images. Instead, it extracts the Mermaid code block from the model's response, validates it, and renders it server-side before displaying it to the student. This ensures the output is stable, predictable, and easily reviewable by staff.
4. Human-in-the-Loop Supervision: Teaching Assistants (TAs) act as accountable supervisors. They monitor multiple sessions and intervene when responses require correction, deeper probing, or escalation to live help. The AI provides a "first-pass" artifact, which the TA can accept, extend, or reject.

Evaluation Method
The paper reports an expert evaluation conducted in Spring 2025 with 21 instructional staff (TAs and an instructional specialist) from Purdue University's CS251 (Data Structures and Algorithms).

• Procedure: Participants used an Android tablet with BoilerSketch for a 45-minute hands-on session. They interacted with the system as if they were students, submitting representative conceptual prompts (e.g., recursion traces, graph traversals) and self-generated questions.
• Metrics: Following the session, participants completed a post-use survey rating the system's perceived helpfulness for conceptual understanding and its usefulness for typical support tasks using ordinal scales. Qualitative feedback and illustrative interactions were also collected to identify boundary conditions.

Results
The evaluation yielded mixed but generally positive results regarding perceived utility:

• Helpfulness: 66.7% (14/21) of staff rated the system at least "moderately helpful" for conceptual understanding. 14.3% rated it "moderately not helpful," and 19.0% were neutral.
• Usefulness: 66.7% (14/21) rated the system at least "moderately useful" for typical support tasks.
• Best-Fit Scenarios: Staff identified the system's strongest value in routine, diagram-centric explanations (e.g., visualizing tree traversals, dynamic programming tables, and graph algorithms like Dijkstra's).
• Limitations: Staff noted significant limitations in the system's ability to handle advanced topics requiring deep depth or to diagnose specific, subtle student misconceptions that might be masked in a broadly plausible AI response. 33.3% of participants rated the system as neutral or not helpful/useful for certain scenarios, highlighting the risk of the AI missing the specific root of a student's confusion.

Key Contributions and Claims
The paper makes three primary contributions as an "innovative practice" paper:

1. Instructional Practice: It presents a replicable framework for supervised, diagram-centric AI support in early computing courses, moving beyond text-first AI interactions.
2. Design Pattern: It articulates a deployable design pattern where prompt-level guardrails, structured diagram rendering (Mermaid), and TA escalation rules jointly bound AI assistance to conceptual explanation rather than answer production.
3. Expert Validation: It provides expert evaluation data identifying the "best-fit" uses (routine conceptual visualization) and boundary conditions (nuanced diagnosis, advanced topics) for this approach.

Significance and Modesty of Claims
The authors position BoilerSketch not as a replacement for human instruction but as a credible first-response layer for routine, diagram-amenable questions. The significance lies in demonstrating that AI support in early computing becomes more instructionally credible when it is:

• Narrowly scoped to conceptual explanation.
• Structurally constrained via structured output (Mermaid) rather than free-form generation.
• Embedded within clear escalation protocols to human staff.

The paper explicitly modestly claims that it does not establish student learning gains, comprehensive correctness across all topics, or reductions in office-hour wait times. Instead, it offers a concrete model for expanding conceptual support while preserving instructional control and academic integrity. The authors argue that the "prompt-as-policy" approach allows course values (such as academic integrity and conceptual focus) to be translated directly into system behavior, making the prompt itself a pedagogical design artifact.