A Controlled Synthetic Benchmark for Educational Aspect-Based Sentiment Analysis

Technical Summary: A Controlled Synthetic Benchmark for Educational Aspect-Based Sentiment Analysis

Problem Statement
Educational Aspect-Based Sentiment Analysis (ABSA) offers a pathway to transform raw student feedback into actionable insights regarding course design, instructional clarity, and learning environments. However, the development of robust ABSA models is severely hindered by the scarcity of public, aspect-labeled datasets. Real student feedback is often private, institution-specific, and costly to annotate due to the complexity of identifying implicit aspects, mixed sentiments, and domain-specific pedagogical terminology. Existing educational text-mining resources are frequently too small, unharmonized, or limited to coarse satisfaction signals rather than fine-grained pedagogical dimensions.

Methodology
The study addresses this data scarcity by introducing a controlled synthetic data generation pipeline paired with a rigorous benchmark evaluation framework.

1. Synthetic Data Generation Protocol:

   • Schema: The corpus is built upon a 20-aspect pedagogical schema categorized into five blocks: instructional quality, assessment and course management, learning demand and readiness, learning environment, and engagement and value.
   • Controlled Separation: The generation process decouples supervision targets (1–3 aspect-sentiment pairs sampled from the 20-aspect inventory) from nuance attributes (contextual variables such as course level, student background, writing style, and realism controls).
   • Prompt Engineering: A realism-constrained instruction merges these sampled targets and attributes. The prompt is refined through a three-cycle "judge-editor" procedure where a human judge attempts to distinguish synthetic reviews from real public OMSCS reviews. Detected synthetic cues (e.g., over-balanced trade-offs, checklist coverage) are used to iteratively stabilize the prompt.
   • Corpus Construction: The final dataset consists of 10,000 synthetic course reviews with explicit train/validation/test splits (8,000/1,000/1,000). The generation utilizes a fixed template with per-row variation to ensure diversity in course contexts and rhetorical styles while maintaining label fidelity.

2. Benchmark Evaluation Pipeline:

   • Task Formulation: The primary task is defined as a two-step ABSA problem: (1) multi-label aspect detection and (2) sentiment prediction for detected aspects.
   • Model Families: The benchmark evaluates several approaches under a strict, shared split:
     • Classical Baselines: TF-IDF with two-step detection and sentiment regression.
     • Transformer Encoders: Two-step models (detection + sentiment) and joint prediction models using BERT, DistilBERT, ALBERT, and RoBERTa.
     • GPT-Based Inference: Zero-shot and few-shot prompting (fixed, diverse, and retrieval-based) using GPT-5.2, evaluated via batch inference without fine-tuning.
   • Metrics: Evaluation focuses on micro-F1 for aspect detection, sentiment Mean Squared Error (MSE) on detected aspects, and complementary diagnostics like balanced accuracy and Matthews correlation coefficient to account for label sparsity.

3. Validation Strategies:

   • Realism Validation: A judge-editor loop using 32 real OMSCS reviews to assess and reduce synthetic detectability.
   • External Transfer: A conservative evaluation on 2,829 mapped student-feedback reviews from Herath et al., restricted to a 9-aspect overlap with the synthetic schema to test synthetic-to-real transfer.
   • Faithfulness Audit: A model-assisted audit (using GPT-5.2) to verify if declared aspect sentiments are visibly supported in the generated text.

Key Contributions

• Educational Benchmark Resource: A 10,000-review synthetic corpus with a 20-aspect pedagogical schema, significantly richer than typical public educational sentiment resources.
• Controlled Generation Design: A methodology that separates supervision targets from contextual nuance, allowing the same aspect labels to appear across varied student personas and course contexts without collapsing into repetitive prompt patterns.
• Reproducible Benchmark Setting: A documented train-validation-test protocol that enables direct comparison of local supervised models and inference-time generative baselines.
• Initial External Relevance: Evidence of partial synthetic-to-real transfer on a mapped real-world dataset, alongside diagnostics clarifying the limits of label faithfulness.

Results

• Internal Benchmark Performance: The task is nontrivial. The strongest untuned local model, bert-base-uncased, achieved a held-out micro-F1 of 0.2760 for aspect detection. A modest adjustment to the training schedule (lower learning rate, longer epochs) improved this to 0.2930. Joint encoders and TF-IDF baselines performed lower, while ALBERT and RoBERTa drifted toward high recall but poor precision.
• GPT-Based Inference: Batch inference with GPT-5.2 in zero-shot mode achieved a micro-F1 of 0.2519, outperforming TF-IDF and joint encoders but falling short of the strongest fine-tuned two-step transformers. Retrieval-based few-shot prompting yielded similar results (0.2501).
• External Transfer: On the mapped Herath dataset (9-aspect overlap), bert-base-uncased achieved a micro-F1 of 0.4593. This performance was higher than internal scores, likely due to the reduced aspect complexity and the dominance of the lecturer_quality signal in the real data.
• Generator Diagnostics:
  • Realism: The judge-editor procedure successfully reduced synthetic cues, with judge accuracy dropping to chance levels (0.4333) in the final cycle, though statistical equivalence to real reviews was not definitively established.
  • Faithfulness: While 77% of declared aspects were textually supported, sentiment polarity match rates were lower (42.3% at the aspect level, 21.2% at the full-review level), indicating that the corpus is a useful but noisy resource rather than a gold standard.

Significance and Claims
The paper claims that controlled synthetic supervision is a productive approach for studying educational ABSA in data-scarce domains. It does not claim to replace human annotation or to provide a perfect gold-standard dataset. Instead, it positions the synthetic corpus as a scaffold for:

1. Model Bootstrapping: Allowing institutions to pretrain models on synthetic data before fine-tuning on small, adjudicated real datasets.
2. Longitudinal Monitoring: Providing a stable yardstick for tracking model performance across terms and cohorts.
3. Stress-Testing: Exposing minority aspects (e.g., accessibility, grading transparency) that are often underrepresented in single-institution archives.

The study concludes that while the synthetic corpus introduces label noise, particularly regarding sentiment polarity, it successfully supports internal learnability, reproducible model comparison, and partial transfer to real educational feedback spaces. The work emphasizes the value of the combination of task framing, controllable generation, and benchmark integration over synthetic text generation in isolation.