A visual observation on the geometry of UMAP projections of the difference vectors of antonym and synonym word pair embeddings

1. Problem Statement

The paper addresses the challenge of detecting the geometric representation of antonymy (opposites) and synonymy (similarities) within high-dimensional embedding spaces of modern language models.

• Theoretical Context: Linguistically, antonyms are often defined as word pairs sharing all contextually relevant properties except one. In NLP, concepts are mapped to directions in vector spaces. The authors investigate whether this "single differing dimension" manifests as a detectable Euclidean direction or a specific geometric structure in embedding vectors.
• The Gap: Previous literature suggests that distinguishing antonyms from synonyms is a non-linear problem; simple Euclidean directions do not suffice. While classifiers can differentiate them, the underlying geometric structure of the difference vectors (the vector between two word embeddings) remains poorly understood.
• Objective: To explore the geometry of embedding difference vectors for antonym/synonym pairs and determine if a consistent, non-trivial structure exists across different model architectures.

2. Methodology

Data Sources

• Primary Dataset: The Stuttgart Dataset [NiWV17a], containing labeled pairs of antonyms and synonyms (categorized by adjectives, nouns, and verbs).
• Control Datasets: To rule out biases in the base vocabulary, the authors created two synthetic "shuffled" datasets:
  • Shuffled Antonyms: Randomly pairing words from the original antonym set.
  • Shuffled Synonyms: Randomly pairing words from the original synonym set.
  • Purpose: These controls test whether classifiers are learning the specific semantic relationship or merely the distribution of the words themselves.

Embedding Models

The study utilized five distinct embedding models to ensure robustness across architectures:

1. Static Embeddings: Word2Vec and GloVe (300 dimensions).
2. Transformer Context-Free: BERT Base (768 dimensions).
3. Proprietary API Models: OpenAI's text-embedding-3-small (1,536 dims) and text-embedding-3-large (3,072 dims).

Vector Construction & Projection

• Vector Construction: For each word pair $(w_1, w_2)$ with embeddings $(E_1, E_2)$, the authors computed the difference vector $D = E_2 - E_1$. (Concatenation was also tested as a control).
• Dimensionality Reduction: The high-dimensional difference vectors were projected into 2D space using UMAP (Uniform Manifold Approximation and Projection).
• Critical Hyperparameters:
  • Metric: The "swirl" phenomenon was only observed using the Euclidean metric. Using Cosine similarity caused the structure to disappear.
  • Label Agnosticism: UMAP was run on the entire dataset (including controls) without using labels during the projection phase; labels were applied only for visualization.

Classification Approach

The authors developed a transductive classifier based on the visual clustering:

1. Project all data (train + test + shuffled controls) to 2D via UMAP.
2. Remove the shuffled control points from the visualization.
3. Cluster the remaining points (using K-Means, Spectral, etc.).
4. Assign labels to test points based on the majority vote of the training points within the same cluster.

• Note: This is transductive (inferring labels for the whole test set simultaneously) rather than inductive (building a model to predict individual points).

3. Key Contributions & Observations

The "Swirl" Phenomenon

The central contribution is the discovery of a consistent, spiral-like or "swirl" structure in the 2D UMAP projections of difference vectors.

• Stability: The swirl appears consistently across different UMAP hyperparameters (neighbors, min_dist) and across vastly different model types (Word2Vec, BERT, OpenAI models).
• Differentiation: Within the swirl, the four datasets (Antonyms, Synonyms, Shuffled Antonyms, Shuffled Synonyms) form distinct, one-dimensional structures.
  • Antonyms separate clearly from Shuffled Antonyms.
  • Synonyms separate clearly from Shuffled Synonyms.
• Specificity: The structure vanishes if:
  • The projection method is changed to PCA or t-SNE (with most models).
  • The metric is changed to Cosine similarity.
  • Vector construction uses concatenation instead of difference.
  • The "shuffled" control datasets are removed (though a diffuse spiral remains, the structure becomes noisier).

Exception: text-embedding-3-large

The OpenAI text-embedding-3-large model exhibited a unique property: it retained non-trivial geometric structures (separation of antonyms/synonyms) even in PCA and t-SNE projections, suggesting a particularly strong linear or geometric encoding of these relationships in this specific model.

4. Results

Classification Performance

The transductive clustering method based on UMAP projections achieved state-of-the-art (SOTA) performance on the Antonym Synonym Differentiation (ASD) task, particularly on the stricter "Lexical" split (where no word appears in both train and test sets).

• Performance Gains: Applying UMAP projection significantly improved accuracy across all models compared to raw vector classification.
• Top Performer: text-embedding-3-large achieved the highest scores.
  • Lexical Split (Overall F1): 0.9660 (ShallowNN on UMAP) vs. 0.886 for the previous SOTA (ICE-NET).
  • Stuttgart Split (Overall F1): 0.9721 (ShallowNN on UMAP).
• Comparison: The UMAP-based transductive approach outperformed previous inductive methods (like Distiller, MoE-ASD, and ICE-NET) which typically rely on complex neural architectures.

Ablation Studies

• Metric Sensitivity: The Euclidean metric is crucial; Cosine similarity fails to reveal the structure.
• Control Necessity: Including shuffled pairs is essential for the "crisp" separation of the swirl, acting as a negative control that sharpens the geometric boundaries.

5. Significance and Conclusion

• Geometric Insight: The paper provides empirical evidence that antonymy and synonymy possess a distinct, stable geometric signature in embedding spaces, specifically visible in the difference vectors under Euclidean UMAP projection. This challenges the notion that these relationships are purely non-linear and unstructured.
• Methodological Implication: The results highlight the sensitivity of dimensionality reduction techniques. The choice of metric (Euclidean vs. Cosine) and projection algorithm (UMAP vs. t-SNE/PCA) drastically alters the perceived geometry of semantic relationships.
• Practical Application: The proposed transductive clustering method offers a simple, highly effective alternative to complex deep learning classifiers for semantic differentiation tasks.
• Future Directions: The authors suggest this "swirl" may relate to emerging universal structures in Large Language Models (LLMs) and warrants further theoretical investigation to understand why this specific geometry emerges.

Note on Reproducibility: The author provides source code and scripts to replicate the "Curious Swirl" at https://github.com/ramiluisto/CuriousSwirl.git. The text explicitly states that while coding assistants were used, the scientific content and writing are human-authored.