Technical Summary: Native Hierarchical and Compositional Representations with Subspace Embeddings

Problem Statement

Traditional dense vector embeddings, which represent concepts as single points in Euclidean space, face fundamental geometric limitations when modeling natural language. While effective for capturing similarity via inner products, the flat, symmetric structure of Euclidean space is ill-suited for hierarchical and asymmetric relationships such as entailment and hyponymy. Standard metrics like cosine similarity cannot distinguish directional relationships (e.g., "dog" entails "animal," but "animal" does not entail "dog"). Furthermore, vector spaces lack native operators for logical conjunction, disjunction, and negation. Consequently, models often default to additive composition (treating phrases as bags of words), leading to "negation blindness" where embeddings fail to exclude concepts, degrading performance on queries involving logical negation. Existing alternatives, such as hyperbolic embeddings or region-based probabilistic models (e.g., boxes, cones), either impose rigid structural priors, lack closure under all logical operations, or sacrifice compatibility with efficient Euclidean retrieval pipelines.

Methodology

The authors propose Subspace Embeddings, a framework where concepts are represented not as points, but as linear subspaces (regions) within a high-dimensional space $\mathbb{R}^d$.

Core Geometric Principles

1. Representation: A concept is defined as the span of a set of latent basis vectors $\mathbf{X} \in \mathbb{R}^{d \times n}$. The subspace $\mathcal{S}$ is represented by its orthogonal projection operator $\mathbf{P} = \mathbf{U}\mathbf{U}^\top$, where $\mathbf{U}$ is an orthonormal basis.
2. Hierarchy and Generality:
   • Dimensionality: The effective dimension (rank) of the subspace encodes semantic generality. Broader concepts (e.g., "animal") occupy higher-dimensional subspaces, while specific concepts (e.g., "dog") occupy lower-dimensional subspaces nested within them.
   • Inclusion: Hierarchy is modeled via subspace inclusion ($\mathcal{S}_A \subseteq \mathcal{S}_B$), quantified by a Normalized Inclusion Score (NIS) derived from the trace of the product of projection matrices.
3. Logical Operations: Logical relations map directly to linear algebraic operations on subspaces:
   • Conjunction ($\land$): Subspace intersection ($\mathcal{S}_A \cap \mathcal{S}_B$).
   • Disjunction ($\lor$): Linear sum (span) of subspaces ($\mathcal{S}_A + \mathcal{S}_B$).
   • Negation ($\neg$): Orthogonal complement ($\mathcal{S}^\perp$).

Technical Implementation: Differentiable Subspaces

A primary challenge is that subspace rank is discrete and non-differentiable. To enable end-to-end training, the authors introduce Soft Projection Operators:

• Soft Projector: Instead of a hard projection, they define a soft projector $\tilde{\mathbf{P}}$ using Tikhonov regularization: $\tilde{\mathbf{P}} = \mathbf{X}(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^\top$.
• Differentiability: The hyperparameter $\lambda$ acts as a temperature, smoothing the eigenvalues of the projector into the range $[0, 1)$. This allows the model to learn the effective dimensionality of each subspace via gradient descent, enabling the subspace to "add or drop" basis vectors dynamically.
• Subspace Projection Head (SPH): A neural module that converts transformer hidden states into a set of latent vectors spanning a subspace. It utilizes Multi-Head Attention (MHA) for pooling, an MLP for rank expansion (breaking linear dependencies to achieve full rank), and the soft projector computation.

Training Strategy

The framework is trained end-to-end using standard objectives:

• Hierarchy Reconstruction: InfoNCE loss on subspace similarity.
• Link Prediction: Margin loss on the Normalized Inclusion Score (NIS).
• Natural Language Inference (NLI): A hybrid approach modeling entailment/contradiction via a Beta distribution over NIS scores and neutrality via an MLP classifier.

Key Contributions

1. Differentiable Subspace Learning: A novel method for learning language representations as linear subspaces with data-dependent, learnable dimensionality, overcoming the non-differentiability of discrete rank.
2. Emergent Compositional Structure: Demonstration that logical operations (conjunction, disjunction, negation) arise naturally from the geometry of the representations without explicit supervision for logical rules.
3. Scalability and Compatibility: The approach preserves compatibility with standard Euclidean vector search pipelines (Maximum Inner Product Search), enabling efficient large-scale retrieval, unlike non-Euclidean alternatives.

Experimental Results

The authors validate the framework on three axes:

1. Hierarchical Modeling (WordNet):

   • Reconstruction: Subspace embeddings (SE) achieve state-of-the-art performance on WordNet reconstruction, outperforming Euclidean (R128) and Hyperbolic (Poincaré P10, Lorentz H10) baselines in Mean Average Precision (mAP) and Mean Rank (MR).
   • Specificity Correlation: The learned effective rank of a subspace shows a strong Spearman correlation ($\rho \approx 0.68$) with the taxonomic rank of the concept, confirming that dimensionality acts as an emergent measure of semantic specificity.
   • Link Prediction: SE models significantly outperform baselines in sparse supervision settings (0%–25% edge coverage), achieving F1 scores up to 95.9%.

2. Natural Language Inference (SNLI):

   • Subspace embeddings match or slightly exceed the performance of strong bi-encoder baselines (e.g., all-MiniLM-L6-v2, mpnet-base-v2) on standard 2-way and 3-way entailment classification tasks.

3. Compositional Entailment (Zero-Shot):

   • Negation Handling: This is the most critical finding. Standard vector baselines degrade to near-random performance (49–69% AUC) on hypotheses involving negation. In contrast, subspace embeddings maintain high discriminative power (>90% AUC) on negated queries without explicit training on negation logic.
   • Comparison: Subspaces outperform Box Embeddings on negation tasks, as the complement of a box is not a box, whereas the orthogonal complement of a subspace is a valid subspace within the same space.

4. Efficiency:

   • Retrieval Speed: By formulating the inclusion score as a dot product between vectorized projection matrices, the method enables the use of fast approximate search libraries (e.g., FAISS). SE128 is nearly 8x faster than a 10-dimensional Poincaré baseline using brute-force search.
   • Compression: The learned subspaces exhibit rapid spectral decay, allowing for 4x compression via truncated SVD with negligible performance loss.

Significance and Claims

The paper claims that subspace embeddings offer a geometrically-grounded model of entailment that reconciles the efficiency of vector retrieval with the hierarchical and compositional structure of natural language.

• Native Logic: Unlike previous methods that require ad-hoc solutions or fail on negation, subspace embeddings natively capture logical composition through linear algebraic operations.
• Interpretability: The effective dimension of a subspace serves as an interpretable measure of information content and semantic specificity, emerging directly from the training geometry.
• Practicality: The framework does not sacrifice scalability; it remains compatible with standard Euclidean search infrastructure, making it viable for large-scale applications.

The authors conclude that this approach provides a promising path toward more expressive and interpretable representation learning systems, particularly for tasks requiring reasoning over hierarchical and negated concepts.