Technical Summary: Testing the Validity of Embedding-Based Similarity and Clustering for Handwritten Physics Solutions

Problem Statement

In physics education, meaningful assessment requires analyzing the solution path—not just the final answer—to evaluate conceptual insight, mathematical modeling, and the coordination of diagrams and equations. While automated systems exist, they often restrict responses to multiple-choice or closed-form algebraic answers, failing to capture the breadth of student reasoning. Handwritten solutions remain the gold standard for high-stakes exams but are labor-intensive to grade at scale.

Recent advances in Large Language Models (LLMs) and text embeddings offer a computationally efficient alternative for organizing and comparing large collections of text. Embedding-based methods map text to high-dimensional vector spaces, allowing pairwise similarities to be computed cheaply via vector operations (e.g., cosine similarity) rather than expensive, pairwise LLM calls. However, a critical validity gap remains: while embeddings can produce clusters, it is unclear whether the geometry of the embedding space preserves grading-relevant distinctions. Specifically, does proximity in embedding space correspond to similarity in human-assigned scores, or do embeddings merely group solutions based on superficial surface features?

Methodology

The study evaluated the validity of embedding-based similarity and clustering using a dataset of 992 handwritten student-problem solutions from a high-stakes engineering thermodynamics exam at ETH Zurich. The methodology involved a multi-stage experimental design:

1. Data Preparation and Transcription

Handwritten solutions were transcribed into text using a multimodal LLM (GPT-5.4) under five distinct transcription conditions to vary the degree of normalization and interpretation:

• ASCII Verbatim: Minimal normalization; preserves idiosyncratic notation and layout in plain ASCII.
• LaTeX-Oriented: Mathematical expressions converted to LaTeX; prose remains in plain text.
• Formula Narrative: Formulas and symbolic steps narrated in natural language while retaining original variables.
• Solution Narrative: A concise narrative account of the student's visible solution process, avoiding evaluative terms.
• Hybrid Structured: A structured representation with fixed headings (e.g., problem labels, formulas, diagrams, apparent method) combining literal content with structural organization.

Crucially, the transcription prompts explicitly forbade the model from solving problems, correcting errors, or assigning scores.

2. Embedding Generation

Each of the five textual representations was processed using nine embedding mechanisms:

• General-Purpose: OpenAI text-embedding-3-large.
• Jina Embeddings v4: Various adapters (text-retrieval, text-matching) and quantization levels (F16, Q8, Q4).
• Math-Specialized: Qwen 2.5-Math models (1.5B and 7B parameters), where final-layer hidden states were mean-pooled to create vectors.

This resulted in a total of 44,640 usable embedding vectors (9 mechanisms × 5 representations × 992 responses).

3. Evaluation Metrics

Human-assigned scores served as the external reference standard but were not used during embedding generation or clustering. Two primary analyses were conducted:

• Pairwise Similarity: Correlation (Pearson and Spearman) between embedding similarity and the negative absolute difference in human scores.
• Hierarchical Clustering: Agglomerative clustering (average and complete linkage) was performed on distance matrices derived from embeddings. The quality of clusters was measured by the weighted within-cluster score standard deviation and the coverage of students in "useful" clusters (size ≥ 5).

4. Synthetic Validation

A synthetic dataset of statements regarding Carnot engine work was created to test whether embeddings cluster by conceptual correctness (flaw families) or surface formulation (statement pairs). This "toy system" allowed for a controlled examination of whether embeddings behave like experts (grouping by principle) or novices (grouping by surface features).

Key Results

1. Weak Correlation Between Embedding and Score Similarity

Across all models and representations, embedding similarity showed a consistent but modest relationship to score similarity.

• The best Spearman correlation was 0.198 (Hybrid structured + Qwen 7B + cosine similarity).
• Even the highest similarity percentiles retained substantial score variation; pairs with high embedding similarity did not necessarily have similar scores.
• The relationship was not linear enough to serve as a reliable proxy for grading equivalence.

2. Clustering Yields Score-Enriched but Not Score-Equivalent Groups

Hierarchical clustering produced groups that were slightly more homogeneous in score than random groupings, but the variation remained high.

• The best configuration (Formula narrative + Jina Retrieval Q8 + centered cosine + complete linkage) achieved a weighted useful-cluster score standard deviation of 2.547 points.
• The score range within these "best" clusters was approximately 7 to 9 points, indicating that clusters contained a wide mix of performance levels.
• Transcription Impact: Narrative and structured representations generally outperformed literal (ASCII/LaTeX) transcriptions, suggesting that converting symbolic work into semantically explicit text helps embeddings capture some grading-relevant structure.
• Model Impact: Differences between embedding mechanisms were marginal; no single model decisively outperformed others across all conditions.

3. The "Novice" Behavior of Embeddings

The synthetic data analysis revealed a critical limitation: embeddings tended to cluster statements based on surface formulation rather than conceptual correctness.

• In the force-directed layouts, correct statements and their flawed variants (sharing the same wording pattern) clustered together.
• Conversely, statements sharing the same conceptual flaw but different wording were often separated.
• This behavior mirrors the "novice" categorization style in physics education (grouping by surface features) rather than the "expert" style (grouping by underlying principles).

Significance and Claims

The paper concludes that state-of-the-art text embeddings do not provide a validated, unsupervised basis for grading handwritten physics solutions.

• Validity Limitation: While embedding geometry is "score-enriched" (containing some signal related to human scores), it is not "score-equivalent." High embedding similarity does not imply grading equivalence.
• Role in Assessment: Embeddings are suitable for exploratory organization, such as browsing large response sets, selecting exemplars, identifying unusual solutions, or supporting human-in-the-loop review workflows. They are not sufficient for autonomous grading without external validation against the specific assessment construct.
• Surface vs. Concept: The study suggests that current embedding mechanisms behave like novices, heavily influenced by surface features (notation, layout, phrasing) rather than the deep conceptual structure required for physics assessment.
• Future Direction: The authors argue that rubric-aware LLM grading or confidence-filtered human-in-the-loop systems are more promising avenues than relying on unsupervised embedding geometry alone. The study emphasizes that clustering results should be treated as hypotheses for human inspection rather than definitive grading categories.