Technical Summary: Mining AI-Assisted Course Design Workflows at Production Scale

Problem Statement

While AI-assisted course design systems are increasingly deployed to generate outlines, assignments, and assessments, empirical evidence regarding their operational workflows remains limited to benchmarks, prototypes, or classroom studies. Production logs reveal a distinct object of study: the end-to-end workflow where users request designs, receive artifacts, revise them, encounter failures, and provide feedback.

The core challenge addressed is how to mine these production-scale workflows for operational insights—specifically for triage, routing, and process monitoring—without violating privacy or introducing data leakage. The authors posit that such systems can be treated as multi-surface workflow problems where value lies not in the generated text itself, but in the metadata surrounding usage, structural validity, iteration, and delay.

Methodology

Data Source and Privacy Protocol

The study analyzes CourseFactory, a deployed AI-assisted course design system. The dataset spans 2023–2026, comprising:

• 197,858 external AI calls.
• 14,598 generated course versions across 9,028 projects.
• 676,417 generated course-structure items.
• 22,746 task-state events.
• 6,323 users.

To ensure privacy and reproducibility, the authors formalize a privacy-preserving workflow-mining protocol (Algorithm 1). This protocol strictly excludes raw prompts, generated text, direct identifiers, and calendar timestamps. Instead, it utilizes:

• Ordinal ranks for temporal ordering.
• Salted surrogate keys for case identification.
• Aggregate metadata (e.g., token counts, prompt lengths, error flags) rather than semantic content.
• De-identified analytic tables released under a CC-BY-4.0 license.

Workflow Surface Abstraction

The system is modeled as a multi-surface workflow rather than a flat scoring table. The study defines and evaluates four distinct surfaces:

1. Generated-Version Review: Triage of course versions based on structural quality.
2. Generated-Item Elaboration: Routing of course-structure items to instructor assignments.
3. Revision Loops: Identification of high-iteration projects requiring review.
4. Task-State Conformance: Checking observed task transitions against a designed process model.

Evaluation Rigor

To prevent common pitfalls in production machine learning, the study employs:

• Temporal and New-Project Splits: Models are evaluated on chronological hold-out sets and, more strictly, on entirely unseen projects to ensure generalization beyond specific project patterns.
• Leakage Audits: Features are audited to ensure they strictly precede the label (e.g., ensuring assignment timestamps do not precede item creation).
• Imbalanced Classification Metrics: Given the skewed nature of the data (e.g., high positive feedback rates, low assignment rates), Area Under the Precision-Recall Curve (AUPRC) is the primary metric, supplemented by AUROC and precision-at-budget operating points.
• Ablation Studies: The impact of excluding raw text is tested using lexical, n-gram, and latent-semantic representations.

Key Results

1. Generated-Version Structural Quality Triage

• Performance: A metadata-driven model achieves an AUPRC of 0.89 (AUROC 0.67) on new projects.
• Feature Importance: The signal is driven by course configuration metadata (duration, language, study load) rather than AI usage error flags.
• Privacy Impact: Excluding raw text and prompt content results in negligible performance loss; metadata-only models perform nearly identically to full non-text usage models.
• Operational Value: A counterfactual replay of the review queue demonstrates that using the model to prioritize reviews saves approximately 20% of reviewer effort compared to random or first-in-first-out (FIFO) queuing.

2. Generated-Item Assignment Routing

• Performance: Predicting whether a course-structure item will evolve into an instructor assignment achieves an AUPRC of 0.28 (baseline 0.10, a 2.74× lift) and an AUROC of 0.81 on unseen projects.
• Leakage Control: This surface is leakage-free by construction, as assignment creation strictly post-dates item creation.
• Utility: The top 10% of items by score captures 29.3% of future assignments with 29.6% precision.

3. Revision Loop Detection

• Observation: High-iteration projects (those generating 11+ versions) form a distinct tail.
• Correlation: These projects exhibit a mean structural quality of 4.46, significantly lower than single-version projects (5.85).
• Operational Trigger: A simple runtime trigger (e.g., flagging projects after 8–10 versions) effectively identifies the high-risk tail for human review without claiming that retries cause quality loss.

4. Task-State Conformance

• Findings: Against a designed process model with 1,112 allowed transitions, only 50 transitions were observed in the data.
• Conformance Rate: Among the observed multi-state task transitions, 66.8% conform to the designed flow.
• Deviation Analysis: The majority of deviations (61.6%) are concentrated in two specific transition pairs, suggesting operational shortcuts (e.g., executing steps out of order under deadline pressure) rather than random errors.

5. Text Exclusion and Signal Analysis

• Ablation: Experiments using lexical, n-gram, and latent-semantic (TF-IDF/SVD) representations of withheld input prompts showed no measurable signal gain over metadata-only features for structural quality or feedback prediction.
• Caveat: The authors note that large neural sentence embeddings (e.g., Sentence-BERT) were not tested and remain an open question regarding their potential to unlock signal behind the privacy boundary.

Significance and Claims

The paper positions itself as a framework-and-dataset contribution rather than a novel model architecture. Its primary significance lies in:

1. First Production-Scale Study: It is the first workflow-mining study of an AI-assisted course design system at production scale, moving beyond prototype or survey-based evidence.
2. Protocol Discipline: It establishes a reusable evaluation protocol that assigns bounded operational roles to different workflow surfaces. It explicitly distinguishes between surfaces suitable for triage (structural quality, item routing), those suitable for monitoring (feedback, task conformance), and those that remain diagnostic (rejected requests) due to leakage risks.
3. Privacy-Feasible Mining: It demonstrates that high-value operational signals (triage and routing) can be extracted using only de-identified metadata, making the study auditable and reproducible without exposing sensitive user prompts or generated content.
4. Realistic Scope: The authors modestly claim that their results support staged deployment for review triage and process monitoring. They explicitly avoid claiming automated grading capabilities or causal relationships (e.g., between retries and quality loss), framing their findings as operational triggers for human intervention rather than autonomous decision-making.

The released dataset, protocol, and evaluation rules aim to provide a template for responsible mining of similar AI-assisted authoring systems, addressing the gap between educational data mining (focused on learners) and production process mining (focused on instructor workflows).