Mapping Semantic & Syntactic Relationships with Geometric Rotation

1. Problem Statement

The paper addresses a critical gap in the interpretability of modern Large Language Models (LLMs) and embedding models.

• The Limitation of Linear Arithmetic: Early word embeddings (e.g., word2vec) allowed for intuitive semantic manipulation via vector arithmetic (e.g., king - man + woman = queen). This relied on the Linear Representation Hypothesis (LRH), which posits that semantic concepts exist in linear subspaces.
• The Modern Challenge: Contemporary high-dimensional text representations (from transformers) are often opaque. While they perform well, their internal geometric structure does not straightforwardly correspond to semantic operations using simple Euclidean vector addition.
• The Geometric Mismatch: The authors argue that modern sentence embeddings reside on curved manifolds (specifically, unit hyperspheres) rather than flat Euclidean spaces. Existing steering and probing methods often assume Euclidean geometry, leading to inconsistent cross-lingual and cross-model performance.
• The Goal: To determine if discourse-level semantic-syntactic transformations (e.g., negation, conditionality) correspond to consistent geometric operations (rotations) on the embedding manifold that generalize across languages and model architectures.

2. Methodology: Rotor-Invariant Shift Estimation (RISE)

The authors propose RISE, a geometric framework that treats semantic transformations as rotational operations on a unit hypersphere, leveraging Riemannian geometry.

Core Concept

Instead of adding vectors in Euclidean space, RISE operates on the tangent space of the hypersphere where embeddings reside. It assumes that changing a sentence's meaning (e.g., adding negation) corresponds to moving along a geodesic (the shortest path on the sphere) rather than a straight line.

The Algorithm

RISE proceeds in three steps:

1. Canonicalization: For a neutral sentence embedding $n$ and its transformed variant $v$, the method computes an orthogonal transformation (a rotor $R(n)$) that maps $n$ to a canonical reference direction $e_1$. This aligns the starting point of all pairs to a common frame of reference, isolating the semantic change.
2. Prototype Learning:
   • The method computes the Riemannian logarithm $\log_n(v)$, which projects the displacement from $n$ to $v$ onto the tangent space at $n$.
   • The rotor $R(n)$ is applied to this tangent vector to align it with the canonical frame.
   • These aligned vectors are averaged across many sentence pairs to create a Prototype Vector ($\vec{p}$), representing the "average" semantic transformation (e.g., the vector for "negation").
   • Formula: $\vec{p} = \frac{1}{M} \sum_{i=1}^{M} R(n_i) \log_{n_i}(v_i)$.
3. Prediction: To apply a transformation to a new, unseen neutral sentence $n^*$:
   • The prototype $\vec{p}$ is rotated back into the tangent space of $n^*$ using the transpose of the rotor: $R(n^*)^\top \vec{p}$.
   • The Riemannian exponential map $\exp_{n^*}$ is applied to move along the geodesic from $n^*$ by the length of the prototype vector, yielding the transformed embedding $v^*$.
   • Formula: $v^* = \exp_{n^*}(R(n^*)^\top \vec{p})$.

Key Theoretical Properties

• Commutativity: The paper proves (Theorem A.1) that applying multiple RISE transformations (e.g., negation then conditionality) yields consistent results regardless of order (up to second-order error), supporting the hypothesis that these operations behave like vector additions in the tangent space.
• Manifold Alignment: Unlike Euclidean methods, RISE respects the intrinsic curvature of the embedding space.

3. Experimental Design

The authors evaluated RISE across a rigorous setup:

• Languages: 7 morphologically diverse languages (English, Spanish, Japanese, Tamil, Thai, Arabic, Zulu) covering multiple language families.
• Transformations: Three discourse-level semantic-syntactic changes:
  1. Negation: Logical reversal (high semantic precision).
  2. Conditionality: Converting declaratives to conditionals (modal semantics).
  3. Politeness: Increasing social formality (highly context-dependent).
• Models: Three embedding architectures with varying dimensions:
  • text-embedding-3-large (3072-dim, OpenAI)
  • bge-m3 (1024-dim)
  • mBERT (768-dim)
• Datasets: Synthetic Multilingual dataset (1,000 pairs per language generated via GPT-4.5), BLiMP (syntactic), and SICK (semantic relatedness).
• Baselines: Compared against Mean Difference Vectors (MDV) (a spherical-aware baseline) and Procrustes alignment (a standard linear rotation baseline).

4. Key Results

• Geometric Consistency: RISE successfully identified consistent geometric structures for discourse-level transformations.
  • Negation: Achieved the highest mean rotor alignment score (0.788), indicating strong, generalizable geometric structure across languages and models.
  • Conditionality: Showed the highest stability and lowest performance variability (0.038), suggesting conditional semantics are captured by robust geometric patterns.
  • Politeness: Showed the most variability (0.060), reflecting its dependence on cultural and linguistic context, yet still achieved moderate-to-strong scores (0.762 mean).
• Cross-Lingual Transfer: RISE demonstrated strong transfer capabilities. Models trained on one language could predict transformations in others with high accuracy (e.g., English prototypes working on Zulu).
  • Dimensionality vs. Architecture: Interestingly, bge-m3 (1024-dim) showed the most consistent cross-lingual performance, while text-embedding-3-large (3072-dim) had the highest raw scores. This suggests architecture and training methodology matter more than raw dimensionality.
• Cross-Model Transfer: RISE prototypes learned in text-embedding-3-large could be mapped to bge-m3 with significant success, proving that geometric semantic structures are shared across different model architectures.
• Baseline Comparison:
  • RISE vs. Procrustes: Procrustes (linear) failed significantly in cross-lingual and cross-model settings (scores dropped to <0.65), confirming that Euclidean assumptions are insufficient.
  • RISE vs. MDV: RISE generally outperformed or matched MDV, particularly in complex cross-lingual scenarios, validating the rotor-based alignment approach.
• Downstream Performance: In a negation classification task, RISE-transformed embeddings achieved 93.0% accuracy, outperforming MDV (87.2%).

5. Key Contributions

1. Extension of the Linear Representation Hypothesis (LRH): The paper extends the LRH from static word embeddings to dynamic, discourse-level sentence embeddings in multilingual contexts, proving that semantic transformations correspond to geodesic rotations on a manifold.
2. RISE Framework: Introduces a novel, mathematically grounded method (Rotor-Invariant Shift Estimation) for identifying and applying semantic transformations using Riemannian geometry, offering a principled alternative to Euclidean steering vectors.
3. Empirical Evidence of Universal Structure: Provides the first large-scale evidence that specific semantic-syntactic transformations (negation, conditionality) exhibit universal geometric structure that transcends language families and model architectures.
4. Identification of Bias: Reveals a persistent English-centric bias in current multilingual models, where geometric structures for English are more robust and transferable than those for low-resource languages (e.g., Zulu).

6. Significance

• Interpretability: RISE offers a new lens for understanding how LLMs encode meaning, moving beyond "black box" probing to geometric manipulation.
• Controllability: The method enables steering of model behavior (e.g., making a sentence more polite or negating it) without fine-tuning, using simple geometric operations.
• Theoretical Foundation: By validating that semantic operations behave like rotations on a hypersphere, the work provides a theoretical basis for future research into manifold-based NLP, suggesting that future models should be designed or analyzed with non-Euclidean geometry in mind.
• Practical Application: The ability to map semantic transformations across languages and models opens doors for cross-lingual control, bias mitigation, and robust semantic editing in multilingual AI systems.

In conclusion, the paper demonstrates that while modern embeddings are complex, they retain a hidden, interpretable geometric order. By respecting the curved nature of this space, RISE unlocks consistent, cross-lingual semantic control that linear methods cannot achieve.