Evaluating NLP Embedding Models for Handling Science-Specific Symbolic Expressions in Student Texts

Technical Summary: Evaluating NLP Embedding Models for Science-Specific Symbolic Expressions

Problem Statement
Natural Language Processing (NLP) embedding models have become standard tools in educational data mining for transforming student text into numeric vectors to capture semantic content. However, a significant gap exists in handling science-related language, which frequently integrates symbolic expressions (formulas, equations, standard notations) with natural language. Current embedding models, trained primarily on general text corpora, struggle to process these mixed inputs. Existing research often overlooks this challenge or removes symbolic expressions entirely, risking biased findings and diminished performance in automated assessment. Specifically, there is a lack of empirical understanding regarding how contemporary embedding models differ in their ability to interpret physics-specific symbolic expressions as they appear in authentic, uncurated student responses, including variations in notation.

Methodology
The study employs two complementary approaches to evaluate six NLP embedding models (including German-specific, multilingual, open-source, and proprietary models) using a corpus of $N=100$ symbolic expressions extracted from authentic German student responses in physics tasks (concept mapping and problem-solving).

1. Similarity-Based Analysis:

   • Data Construction: For each symbolic expression ($SE_i$), five categories of text pairs were created: (1) Correct Literal Translations ($LT$), (2) Incorrect Literal Translations ($ILT$), (3) Correct Related Concepts ($RC$), (4) Incorrect Related Concepts ($IRC$), and (5) Off-topic statements ($OT$).
   • Metric: Cosine similarity was computed between the embedding of the symbolic expression and the embeddings of the paired text categories.
   • Statistical Evaluation:
     • ROC Curves: Receiver Operating Characteristic curves and Area Under the Curve (AUC) scores were calculated to quantify the models' ability to distinguish between correct and incorrect pairs (e.g., $LT$ vs. $ILT$).
     • Pairwise Comparison: A paired one-sided Wilcoxon signed-rank test was used to determine if similarity scores for correct pairs were significantly higher than for incorrect pairs. Effect sizes were measured using the matched-pairs rank biserial correlation coefficient.

2. Machine Learning Pipeline Integration:

   • Task: A supervised classification task using a Support Vector Machine (SVM) to categorize student propositions (concept A – linking statement – concept B) into four quality levels (wrong, superficial, simple, detailed).
   • Variable: The NLP embedding model was varied while the classifier (SVM) and data (3,322 propositions, 17% containing symbolic expressions) remained constant.
   • Metric: Classification performance was measured using Cohen's $\kappa$, comparing results for purely textual responses versus responses containing symbolic expressions.

Key Results

• Model Performance Hierarchy: OpenAI's GPT-text-embedding-3-large consistently outperformed all other models in both evaluation approaches.
  • In similarity-based analysis, it achieved the highest AUC scores for distinguishing correct from incorrect literal translations ($AUC = 0.73$) and related concepts ($AUC = 0.74$).
  • In the instance-based comparison, it correctly identified the higher similarity of correct pairs in 91% of cases for $LT$ vs. $ILT$ and 83% for $RC$ vs. $IRC$, with high statistical significance ($p < 0.001$).
• Performance of Other Models:
  • GPT-text-embedding-ada-002 and German Semantic STS V2 showed moderate performance, significantly better than random guessing in some comparisons but inferior to the top model.
  • G-SciEdBERT (fine-tuned for science education) and Gbert-large performed poorly, often failing to distinguish between correct and incorrect pairs significantly better than chance. Notably, the science-specific fine-tuning did not translate to better handling of symbolic expressions.
  • Several open-source models performed close to random guessing ($AUC \approx 0.50$) in distinguishing correct from incorrect related concepts.
• Impact on Classification: In the machine learning pipeline, GPT-text-embedding-3-large yielded the highest Cohen's $\kappa$. The performance gap was most pronounced for responses containing symbolic expressions, where the top model outperformed the weakest model by $\Delta\kappa = 0.16$. For purely textual responses, the gap was $\Delta\kappa = 0.22$.

Significance and Claims
The paper claims that while the performance advantage of the best model (GPT-text-embedding-3-large) is "moderate rather than decisive," it is practically significant for large-scale educational applications. A difference of 0.16 in Cohen's $\kappa$ could result in hundreds of additional student responses being accurately classified in large datasets, directly impacting the quality of automated feedback.

The study underscores that:

1. Selection Matters: Researchers and practitioners must carefully select embedding models when working with science-related language, as standard or even science-fine-tuned open-source models may fail to capture the semantics of symbolic expressions.
2. Trade-offs Exist: While proprietary models currently offer superior performance, their use involves trade-offs regarding cost, data privacy, and dependency on external providers.
3. Need for Specialized Models: There is a pressing need for open-source embedding models explicitly trained on scientific texts to robustly handle symbolic expressions, rather than relying on general language understanding.
4. Lower-Bound Estimate: The reported performance metrics represent a lower-bound estimate, as the study deliberately omitted contextual information to test inherent capabilities; performance in real-world applications with full sentence context may be higher.

The authors conclude that the proposed evaluation framework provides a suitable foundation for benchmarking future embedding models, including upcoming generations from major providers, and for guiding their application in data mining contexts involving scientific language.