💻 computer science

SPARK: Real-Time Monitoring of Multi-Faceted Programming Exercises

This paper introduces SPARK, a real-time monitoring dashboard that helps instructors track student progress on complex, multi-faceted programming exercises by enabling flexible checkpoint grouping, automated test suggestions, and visualization of intermediate outputs, validated through a dataset of keystroke data and an evaluation with 16 instructors.

Original authors: Yinuo Yang, Ashley Ge Zhang, Steve Oney, April Yi Wang

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yinuo Yang, Ashley Ge Zhang, Steve Oney, April Yi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a programming class as a busy kitchen where a chef (the instructor) is trying to teach 20 cooks (students) how to build a complex, custom cake. The problem? The recipe isn't a straight line. Some cooks start with the frosting, others with the sponge, and some are mixing ingredients in a different order. The chef can't be in 20 places at once to see who is stuck, who is following the recipe, or who is accidentally putting salt in the sugar.

This is the problem SPARK solves. It's a digital "control tower" dashboard designed to help teachers watch over students building complex web programs in real-time, even when everyone is working at their own pace and in their own unique way.

Here is how SPARK works, broken down into simple concepts:

1. The "Checkpoint" Map (The Recipe with Milestones)

In traditional coding classes, a teacher might just look at the final code at the end. But with complex projects, that's too late.

The Analogy: Think of SPARK as a GPS for the class. Before the class starts, the teacher sets up "checkpoints" (like rest stops on a road trip).
How it works: The teacher tells SPARK, "Step 1 is building the HTML skeleton," and "Step 2 is adding the CSS colors." SPARK then uses a smart assistant (AI) to write a little "test script" that automatically checks if a student has actually finished that step.
The Benefit: Instead of guessing, the teacher knows exactly who has reached the "HTML" stop and who is still stuck on the "CSS" stop, regardless of the order they are working in.

2. The "Heat Map" Dashboard (Seeing the Crowd)

Once the class is coding, SPARK shows a visual map of the whole room.

The Analogy: Imagine a foggy room where every student is a glowing dot. If many students are stuck at the same spot, the fog gets darker and thicker there.
How it works: The dashboard shows a line of progress. Dots represent students. If a student is 50% done, their dot is halfway down the line. If 10 students are all stuck at the same checkpoint, that area of the line looks dark and crowded.
The Benefit: The teacher can instantly spot, "Oh, half the class is stuck on Step 3," without having to read 20 different code files. They can then stop the class to help everyone with that specific issue.

3. The "X-Ray" Goggles (Looking Inside the Code)

Sometimes, a student's code looks fine on the outside, but the inside is broken.

The Analogy: Imagine you are looking at a car engine. You can see the hood is closed, but you can't tell if the spark plugs are working. SPARK gives the teacher "X-ray goggles."
How it works: The teacher can click on a specific part of a student's project (like a button on a webpage) and see exactly what is happening inside that button for every student at once. They can see if the button is the right size, if it's missing a color, or if it's broken.
The Benefit: The teacher doesn't have to guess why a student's button isn't working; they can see the exact error across the whole class instantly.

4. The "Time-Travel" Replay (Watching the Movie)

Coding isn't just about the final result; it's about the journey.

The Analogy: SPARK is like a DVR for the class. If the teacher steps away for a moment, they can press "rewind" later to see exactly what happened while they were gone.
How it works: The system records every keystroke. The teacher can slide a bar to watch the class code evolve minute-by-minute, seeing who was typing furiously and who was stuck staring at the screen.
The Benefit: This helps the teacher understand how students are thinking and where they got confused, even if they missed the moment it happened.

What the Study Found

The researchers tested SPARK with 16 experienced coding teachers. They compared using SPARK to using a basic, standard tool (the "Baseline").

Better Detective Work: Teachers using SPARK were much better at spotting exactly what students were struggling with. They didn't just guess; they had proof.
More Confidence: Teachers felt much more confident in their understanding of the class. They weren't flying blind.
Less Overwhelm: Instead of feeling like they had to read 20 different books to find the answer, SPARK summarized the information so they could see the big picture and the small details easily.

The Bottom Line

SPARK is a tool that turns a chaotic, confusing coding classroom into a clear, organized map. It helps teachers see the "forest" (the whole class's progress) and the "trees" (individual student struggles) simultaneously, allowing them to give better help exactly when it's needed.

Technical Summary: SPARK – Real-Time Monitoring of Multi-Faceted Programming Exercises

Problem Statement

Programming instructors frequently utilize in-class coding exercises to reinforce concepts; however, effectively monitoring student progress in real-time is increasingly difficult, particularly for multi-faceted problems. These problems are characterized by non-sequential workflows, nested substeps with complex interdependencies, and a lack of predictable completion orders (e.g., web development tasks involving HTML, CSS, and JavaScript).

Existing monitoring solutions face several limitations:

Lack of Control: Code clustering and progress visualization tools often summarize data without allowing instructors to define specific grouping criteria or aspects to monitor.
Information Overload: Real-time monitoring tools often provide raw data streams that overwhelm instructors in large classes without summarizing progress meaningfully.
Scope Mismatch: Many prior works focus on short, single-file snippets, whereas modern exercises often span multiple files and modules.
Insufficient Depth: Current tools rarely allow instructors to inspect intermediate outputs or variations in student solutions to understand diverse problem-solving approaches.

Methodology: The SPARK System

To address these challenges, the authors introduce SPARK, a coding exercise monitoring dashboard tailored for multi-faceted programming exercises, with a specific implementation focus on web programming. The system is built upon a pipeline that integrates customizable checkpoints, automated testing, and real-time visualization.

System Architecture and Components

SPARK consists of five primary panels designed to support the instructor's workflow:

Reference Panel: Instructors input reference code answers. The system automatically generates a preview of the expected webpage and a corresponding DOM tree.
Checkpoints Panel: This is the core configuration interface. Instructors can create nested-task checkpoints.
- Process: Instructors define a task description and use an AI-assisted feature (OpenAI API) to generate testing code.
- Verification: Instructors can verify the generated testing code against the reference code to ensure accuracy before deployment.
- Function: The testing code evaluates specific element behaviors and simulates interactions (e.g., clicking a button) to assess functional correctness.
Progress Visualization Panel: This panel visualizes student progress across checkpoints.
- Visualization: Each student is represented by a dot moving from 0% to 100% completion across a timeline.
- Density: Overlapping areas are shaded to indicate student density (darker shades = more students at that progress level).
- Interaction: Instructors can hover to highlight specific trajectories, use a brush tool to select groups, or lock a specific student's view to trace their history over time.
Components Inspector Panel: This allows for deep-dive analysis of student outputs.
- DOM Property Inspection: Instructors can select specific element selectors and view variations in properties (e.g., width, height) across students.
- Visual Previews: The system clusters and displays screenshots of rendered elements (using Resemble.js) to visually compare how different students implemented the same requirement.
- Simulation: The inspector runs the checkpoint's testing code to simulate interactions (e.g., adding a todo item) before displaying results.
My Classroom Panel: Displays live student code boxes and classroom statistics.
- Modes: Supports Real-Time Mode (live keystroke updates) and Timestamp Mode (minute-by-minute snapshots).
- Replay: Instructors can use a slider to review the chronological history of student code changes.

Data Collection and Implementation

Data Capture: A custom VS Code extension captures keystroke-level data (edit content, location, timestamp) and transmits it to SPARK.
Execution Engine: SPARK uses Puppeteer to simulate browser interactions, run test cases on student code, and generate DOM trees and visual previews.
Dataset: The authors constructed a dataset of 40-minute keystroke coding data from N=22 learners solving two web programming exercises.

Evaluation

The authors conducted a within-subjects user study with 16 programming instructors (instructors, TAs, and experienced graduate students) to evaluate SPARK's effectiveness.

Protocol: Participants monitored simulated real-time classroom sessions using two conditions:
1. Baseline: A simplified version of SPARK containing only reference code and raw code views (no metrics, no checkpoints, no visualizations).
2. SPARK: The full system with all features.
Tasks: Participants answered quiz questions regarding student performance, identified challenges, and assessed code correctness during and after the sessions.
Analysis: The study utilized mixed-effects linear regression models, quiz accuracy analysis, and thematic analysis of semi-structured interviews.

Key Results

The evaluation demonstrated that SPARK significantly outperformed the Baseline system:

Improved Accuracy in Identifying Challenges: Participants using SPARK answered quiz questions with significantly higher accuracy ( $p < .001$ ) compared to the Baseline. While Baseline users often relied on superficial glances at code, SPARK users utilized the Component Inspector to gain deeper insights into code behavior.
Increased Instructor Confidence: Participants reported significantly higher confidence in their assessments of student progress with SPARK. The number of "unsure" answers decreased significantly ( $p < .001$ ).
Efficiency and Detail: SPARK enabled faster and more accurate checkpoint assessments. Participants noted that the system provided detailed, customizable insights into variations in student code states that were otherwise invisible.
Active Engagement: Instructors actively engaged with the system, using the Components Inspector an average of 3.4 times per session. They valued the ability to filter and inspect specific elements rather than reviewing raw code for every student.
Usability: In a dedicated session (S3) where instructors created their own checkpoints, features like "AI Generate Test" and "Verify Checkpoint" were rated highly for usefulness and ease of use.

Significance and Claims

The paper claims that SPARK addresses a critical gap in programming education by providing a flexible, customizable framework for monitoring complex, non-linear programming tasks.

Pedagogical Impact: By visualizing progress across varied workflow sequences and allowing inspection of intermediate states, SPARK helps instructors understand student mental models and identify specific bottlenecks (e.g., students struggling with a specific CSS layout vs. JavaScript logic).
Scalability: The system is designed to handle the cognitive load of large classes by aggregating data into meaningful visualizations (density maps) and allowing targeted drilling down into specific issues.
Active vs. Passive Monitoring: Unlike prior tools that offer passive data streams, SPARK empowers instructors to actively query runtime states, test case results, and output variations, facilitating more targeted and effective feedback.
Future Potential: The authors suggest that the checkpoint-based approach can be extended to other domains (e.g., machine learning pipelines) and that AI-assisted preparation workflows could further reduce the setup burden for instructors.

The study concludes that SPARK effectively supports real-time teaching by deepening instructor understanding of student progress and encouraging active engagement in the monitoring process, ultimately helping to prevent student frustration and loss of confidence.