SCALEFeedback: A Large-Scale Dataset of Synthetic Computer Science Assignments for LLM-generated Educational Feedback Research

Technical Summary: SCALEFeedback Dataset for LLM Educational Feedback Research

Problem Statement
The integration of Large Language Models (LLMs) into education offers significant potential for providing automated, timely feedback to students. However, research into generalizable methodologies for generating effective and responsible educational feedback is currently hindered by a critical lack of open-source datasets. Existing research is often limited to specific programming tasks or single courses, and the creation of comprehensive open datasets containing assignment descriptions, rubrics, and diverse student submissions is obstructed by university intellectual property rights and student privacy concerns. Consequently, there is no large-scale, open dataset that comprehensively includes these elements across various courses to support the development of scalable, generalizable feedback systems.

Methodology: Sophisticated Assignment Mimicry (SAM)
To address the scarcity of shareable educational data, the authors introduce SCALEFeedback (Synthetic Computer science Assignments for LLM Educational Feedback Research), a dataset generated using a novel framework called Sophisticated Assignment Mimicry (SAM).

The methodology involves a one-to-one LLM-based imitation process designed to preserve the semantic meaning and data distribution of real-world data while removing personally identifiable information (PII). The process follows a four-step iterative instruction loop:

1. Evaluate: The LLM analyzes the original assignment description (theme, objectives, style, length, rubric) and the corresponding real student submission (extent, detail, correctness, and estimated mark).
2. Generate: The LLM creates a synthetic assignment description and a synthetic student submission, strictly imitating the dimensions evaluated in the previous step.
3. Re-evaluate: The LLM evaluates the newly generated synthetic data against the same dimensions used for the original data.
4. Compare and Loop: The LLM compares the evaluation results of the synthetic data with the original. If the imitation is deemed successful, the data is output; otherwise, the process loops back to step 2.

To ensure privacy, the framework employs an external privacy gate using a high-reasoning LLM (o4-mini). This gate performs a three-step check to verify that no student names or IDs from the real submission appear in the synthetic output. If a match is found, the generation is discarded and retried.

The dataset was constructed from 45,126 real student submissions from Monash University's Faculty of Information Technology (2021 first semester). After filtering for length constraints (under 7,500 words to accommodate older LLM context windows) and proportional weighting, 10,000 synthetic submissions were generated across 155 assignments in 59 computer science courses.

Key Contributions
The paper presents four primary contributions:

1. Open Dataset Release: The release of SCALEFeedback, containing 10,000 synthetic student submissions across 155 assignments and 59 courses, formatted for research into generalizable automated feedback methods.
2. Technical Validation: Extensive validation demonstrating that the synthetic data closely resembles real data in terms of submission marks, semantic similarity, linguistic metrics (LIWC), and readability metrics (Coh-Metrix), while ensuring the removal of student-identifiable information.
3. Utility as a Proxy: Evidence showing that the dataset serves as a reliable proxy for LLM-based feedback research. Feedback generated for synthetic submissions is comparable in quality to feedback generated for corresponding real submissions.
4. Privacy-Preserving Synthesis Process: Documentation of a one-to-one LLM-based synthesis process that maintains the distributional characteristics of sensitive educational data while enabling open sharing.

Results

• Resemblance: The SAM method achieved high semantic similarity (BERTScore F1 > 0.85) and strong positive correlation in length (PCC > 0.85) compared to the original data. In contrast, a "Naïve mimicry" baseline showed significantly lower correlation in length (PCC ≈ 0.5) and higher error rates.
• Linguistic and Readability: Synthetic descriptions and submissions exhibited cosine similarities greater than 0.94 with real data across 83 LIWC linguistic features and 68 Coh-Metrix readability indices.
• Mark Distribution: While the synthetic submissions showed a moderate positive correlation with original marks (PCC = 0.624) compared to the naïve baseline (PCC = 0.421), the authors note that the synthetic data tends to cluster around the mean, showing less effectiveness in imitating long-tailed patterns (e.g., extremely high or low marks).
• Utility Evaluation: Using 10 different commercial LLMs to generate feedback, the study found high consistency between feedback for synthetic and real submissions. The average alignment was 0.847, and the F1 score was 0.856, indicating the synthetic data effectively preserves features necessary for triggering specific feedback components.
• Privacy Protection: The combination of in-prompt protection and the external privacy gate achieved a 100% privacy protection rate, significantly outperforming the naïve mimicry method (96.5%) and single-mechanism approaches.

Significance and Limitations
The authors position SCALEFeedback as a valuable contribution to researchers aiming to develop scalable, generalizable methods for automated educational feedback. By providing a standardized, open-source dataset that avoids legal and ethical constraints, it facilitates discovery, reuse, and methodological benchmarking.

However, the paper maintains a modest stance regarding limitations. The authors acknowledge that the synthesis method did not fully capture the behavioral diversity of "outlier" student behaviors, such as those with extremely high word counts or failing grades, as the synthetic data tends to be more centered than real-world data. Consequently, the dataset may not be suitable for training models specifically designed to detect failing students or academic misconduct cases, which are often long-tail phenomena. The paper suggests the SAM framework can be adopted by other institutions to transform sensitive internal datasets into shareable resources.