A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol

Technical Summary: A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol

Problem Statement
Educational institutions accumulate vast quantities of open-ended teaching-evaluation feedback that remain unread due to volume constraints. While Natural Language Processing (NLP) offers a pathway to recover this qualitative signal, institutions require analytical procedures that are documented, reliability-checked, and validated rather than merely achieving high accuracy on leaderboards. A prior study established a validated protocol for classifying Spanish teaching feedback by thematic category and sentiment using a "frozen-encoder" design (BETO, 2019-era embeddings) and classical classifiers. This paper addresses two critical limitations of that prior work: (1) whether a protocol fixed to 2019-era representation methods remains competitive as embedding and Large Language Model (LLM) technologies advance, and (2) whether the protocol transfers effectively to a second language (English).

Methodology
The study employs a "durability benchmark" approach, holding the annotation protocol, validation procedure, and evaluation splits constant while varying the underlying model architecture and language.

• Protocol Consistency: The study reuses the exact annotation guide from the prior work, defining four thematic categories (methodology, evaluation, interaction, attendance/engagement) and three sentiment classes (positive, negative, neutral). Reliability is measured via test-retest intra-annotator agreement ($\kappa=0.82$ thematic, $0.88$ sentiment).
• Experimental Arms (Spanish): The protocol is re-run on the original Spanish institutional corpus (1,165 labeled comments, 233 held-out) across three representation generations:
  1. Sparse Lexical: TF-IDF features with linear classifiers.
  2. Frozen Transformer Embeddings: Mean-pooled embeddings from a 2019-era Spanish BERT (BETO) and a modern multilingual encoder (e5-base), feeding classical classifiers.
  3. Prompted LLMs: Zero-shot and few-shot prompting of instruction-tuned models (Claude Haiku 4.5, Claude Opus 4.8, and local Qwen2.5-1.5B).
• Cross-Language Transfer (English): The sentiment task is transferred to English using a newly constructed, balanced corpus of 45,000 comments sampled from RateMyProfessor reviews. Sentiment labels are derived from star ratings (4+ = positive, $\le$2 = negative) and validated against human-labeled data from EduRABSA (Cohen's $\kappa=0.66$). Thematic transfer is limited to a documented category mapping to EduRABSA entities, not a full classification task.
• Evaluation Metrics: Weighted and macro F1 scores are reported as primary metrics. Comparisons are treated as descriptive evidence of consistency rather than statistical significance tests (McNemar's test is used descriptively). Cost, latency, and auditability (via LIME) are also reported.

Key Contributions

1. Durability Benchmark: A re-evaluation of a validated teaching-feedback protocol across three generations of NLP representation (sparse, frozen embeddings, LLMs), demonstrating that the protocol itself is the stable variable while model performance varies.
2. Cross-Language Transfer Evaluation: A transfer of the sentiment classification task to English with an explicit claim boundary, utilizing a balanced, reproducible corpus and a documented star-to-sentiment labeling rule validated against human annotations.
3. Reproducible English Corpus: The release of a build recipe (script, seed, split indices) for a balanced 45,000-comment English sentiment dataset, allowing exact reproduction without redistributing the underlying copyrighted text.

Results

• Spanish Durability:
  • Thematic Classification: LLMs (Opus 4.8 and Haiku 4.5) achieved the highest weighted F1 scores (approx. 0.87–0.88), significantly outperforming frozen embeddings (approx. 0.74) and TF-IDF (0.60) on the held-out set. The improvement was most pronounced for rare thematic classes (e.g., attendance/engagement).
  • Sentiment Classification: The 2026 frontier model (Opus 4.8) showed no accuracy advantage over the cheaper model (Haiku 4.5) on the sentiment task (Haiku zero-shot: 0.923 vs. Opus zero-shot: 0.884). Both outperformed frozen embeddings, but the "parity-at-a-cost" pattern suggests the task ceiling is reached by cheaper models.
  • Cost vs. Performance: The frozen-encoder design (BETO and e5) remained competitive in accuracy while offering significantly lower inference costs and higher auditability compared to LLMs.
• English Transfer:
  • Sentiment: The modern frozen multilingual encoder (e5) was the strongest arm on the full test split (wF1 0.729). LLMs matched this performance only with few-shot prompting; zero-shot performance was lower (approx. 0.67).
  • Label Noise: The ceiling for English performance was bounded by the noise in star-derived labels (agreement with human labels $\kappa=0.66$), preventing models from exceeding this agreement level.
  • Thematic Transfer: The study did not claim a solved thematic classifier for English; instead, it provided a documented alignment between Spanish categories and English entity types (Course, Staff).

Significance and Claims
The paper argues that the protocol is durable, not the specific model. The contribution of the prior work—the documented annotation guide, reliability measurement, and validation design—remains valid and effective even as underlying representation methods evolve from 2019-era embeddings to 2026 frontier LLMs.

• Model Choice is a Deployment Decision: The choice between a cheap, auditable frozen encoder and a high-cost LLM is not a matter of methodological superiority but of institutional constraints (cost, latency, auditability).
• No Universal Frontier Gain: For the sentiment task, the most advanced models did not provide a descriptive accuracy advantage over cheaper alternatives, suggesting that for specific, well-defined tasks, "frontier" capability may not be necessary.
• Auditability: The study emphasizes that while LLMs may win on raw accuracy for thematic classification, their "prompt-bound" nature reduces auditability compared to inspectable sparse weights or LIME-explained classical models.

The authors explicitly state that the procedure (the protocol) is what institutions should carry forward, while the model running underneath it is a variable deployment choice. The paper does not claim to solve equity audits or demographic subgroup analysis, nor does it claim the English results are directly comparable to the Spanish gold-standard results due to differences in label quality (star-derived vs. human-labeled).