💬 NLP

Geometry of Semantic Space: Comparative Study of Discrete and Continuous Models

This paper compares the semantic geometry of supervised transformer embeddings (like CamemBERT) and lexical co-occurrence graphs using a French public debate corpus, revealing that while both share similar local topologies, graph-based models offer a clearer, more interpretable global structure that could guide the development of more stable and understandable neural architectures.

Original authors: Gabriel Bounias, Sabine Ploux

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gabriel Bounias, Sabine Ploux

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to map the entire landscape of human thought and language. You want to create a map where words that mean similar things are close together, and words that are very different are far apart.

This paper compares two different cartographers (map-makers) trying to draw this same map using the same source material: a massive collection of 10 million sentences from a French public debate.

The Two Map-Makers

1. The "Deep Learning" Cartographer (CamemBERT)
Think of this as a super-smart, high-tech GPS that has read almost everything written in French. It doesn't just look at words; it understands context. It places every word into a giant, invisible, 768-dimensional room (imagine a room with 768 different directions you can move, not just up, down, left, or right).

How it works: It calculates where a word belongs based on complex patterns it learned from billions of examples.
The Result: It's incredibly fast and good at local tasks (like knowing "bank" means a river edge in one sentence and a money place in another). However, the paper argues that when you look at the whole map, the points get squished together. It's like a crowded party where everyone is standing in a tight circle; it's hard to tell who is actually related to whom because everyone is too close to everyone else.

2. The "Graph" Cartographer (Co-occurrence Cliques)
This cartographer takes a more old-school, logical approach. Instead of a giant invisible room, they build a physical web.

How it works: They look for words that frequently appear together in the same sentences (like "dance," "opera," and "theater"). They group these into "cliques" (tight-knit friend groups). Then, they connect these groups if they share members or similar vibes.
The Result: This creates a clear, structured web. It's like a subway map where you can clearly see the lines connecting different neighborhoods. The distances between groups feel more natural and gradual.

The Big Discovery: Local vs. Global

The researchers compared the two maps to see which one better reflects how humans actually organize meaning.

The Local View (The Neighborhood):
If you zoom in very close, both maps look similar. If you pick a word like "ice floe," both models correctly identify that "glacier" and "snow" are nearby.

Analogy: If you are standing in a small park, both GPS and a paper map will tell you that the bench is right next to the tree.

The Global View (The Whole City):
This is where they diverge sharply.

The Deep Learning Map: As you move further away from your starting point, the map gets blurry. The "distance" between concepts loses its meaning. After about 10 steps, the model starts averaging everything out. It's like a fog that rolls in; you can't tell if you are walking toward the ocean or the mountains anymore because everything looks the same.
The Graph Map: This map maintains a clear structure all the way out. It shows a gradual transition. You can see how "dance" connects to "music," which connects to "art," which connects to "culture." The paper found that this model creates a "layered" geography where meaning shifts slowly and logically, rather than collapsing into a dense blob.

The "Crowded Room" Problem

The paper points out a specific flaw in the Deep Learning map called the "Curse of High Dimensionality."

The Metaphor: Imagine a room so huge (768 dimensions) that it's impossible to fill it up properly. Because the room is so vast, the data points (the words) end up floating in a tiny, dense cluster in one corner. They are all bunched up, making it hard to tell them apart.
The Graph Solution: The graph model uses the space more evenly. It spreads the "meaning" out, creating a more organized and readable structure where different topics (like ecology vs. agriculture) have their own distinct zones connected by clear bridges.

The Conclusion

The paper concludes that while the Deep Learning model (CamemBERT) is excellent at handling immediate, local context, it struggles to maintain a clear, organized structure for the entire landscape of meaning.

The Graph model, by contrast, offers a more human-readable, stable, and gradual organization of ideas. The authors suggest that future AI models might benefit from learning from this graph structure—essentially, giving the "super-smart GPS" a better blueprint of the city so it doesn't just get lost in a dense fog of data, but can navigate the whole world with clarity.

Technical Summary: Geometry of Semantic Space – Comparative Study of Discrete and Continuous Models

Problem Statement
The paper addresses the challenge of evaluating the quality of Natural Language Processing (NLP) models by analyzing the geometric consistency of the semantic structures they induce. While supervised vector embeddings (e.g., CamemBERT) achieve high performance, their induced geometries often exhibit unsatisfactory distributions that may distort the organization of meaning. Conversely, graph-based models relying on discrete lexical co-occurrences offer more explicit neighborhood structures. The authors hypothesize that comparing the geometries derived from these two paradigms on the same corpus can reveal their respective limitations and complementarities, specifically regarding how well they reflect the human-conceived organization of meaning.

Methodology
The study employs a comparative framework applied to the "Grand Débat National" corpus (approx. 10 million sentences from the 2019 French public consultation). To ensure a fair comparison, both models utilize the same linguistic units: regular co-occurrence cliques. These cliques are sets of words appearing together with high probability (measured via Pointwise Mutual Information, PMI), serving as contextualized semantic units rather than isolated words.

Two distinct models are constructed:

Graph Model ( $G_c$ ): A discrete graph where nodes are co-occurrence cliques. Edges are defined by a similarity function $s$ that combines lexical overlap and contextual similarity (weighted by $\lambda$ ) using Positive PMI (PPMI).
Supervised Vector Model ( $G_b$ ): A continuous model using CamemBERT (a RoBERTa-based Transformer pre-trained on French text). Co-occurrence cliques are embedded by averaging the contextual embeddings of their constituent words across relevant sentences, resulting in 768-dimensional vectors. A graph $G_b$ is subsequently derived from these vectors based on cosine or Euclidean similarity.

Comparison Techniques
The authors utilize two primary approaches to compare the semantic geometries:

Graph Embedding: The discrete graph $G_c$ is embedded into vector spaces using four methods (Force-Directed, Spectral, Isomap, Node2Vec) to allow direct geometric comparison with the CamemBERT space.
Structural Analysis: The vector space is converted into a graph $G_b$ $G_{b}$ , and both $G_c$ $G_{c}$ and $G_b$ $G_{b}$ are analyzed using graph theory tools, including:
- BFS Subtrees: To examine local semantic neighborhoods.
- Metric Correlation: Comparing graph-based distances against vector-based similarities.
- Centrality Measures: Analyzing degree and betweenness centrality distributions.
- Clustering: Using the Infomap algorithm to detect communities and evaluate cluster homogeneity and semantic coherence.
- Trustworthiness ( $T_w$ ): Measuring how well the local structure of the graph is preserved in the embedding space as dimensionality varies.

Key Results

Local vs. Global Consistency: The models show similar local topologies; CamemBERT embeddings align well with the discrete graph structure in the immediate neighborhood of a root clique. However, this correlation decays rapidly (within 10–15 nodes), indicating that the continuous model fails to preserve the gradual, progressive organization of meaning found in the discrete graph.
Semantic Distortion in Continuous Space: In the CamemBERT space, semantic distinctions are "smoothed out" and averaged quickly. The distribution of cosine similarities is centered around 0.9 (rather than 0), suggesting a "cone" of concentrated vectors and a lack of effective use of the high-dimensional space (a manifestation of the Curse of High Dimensionality).
Structural Divergence:
- $G_b$ (Vector-derived): Exhibits a heterogeneous structure with a few hyper-connected nodes (degree up to 100) and many isolated nodes. It lacks nodes with high betweenness centrality, meaning no single clique acts as a significant bridge between lexical fields.
- $G_c$ (Graph-derived): Displays a more homogeneous structure with a lower maximum degree (25) and a wider range of betweenness centrality. Specific cliques (e.g., linking ecology and agriculture) serve as structural bridges, creating a layered geometry of lexical proximity.
Cluster Quality: Clustering via Infomap reveals that $G_c$ produces more homogeneous clusters with higher semantic coherence. In contrast, $G_b$ clusters often contain thematically unrelated cliques (e.g., "wind power" and "noisy conversation").
Dimensionality: The graph embedding of $G_c$ achieves high trustworthiness (>0.7) with dimensions under 100, suggesting that the discrete semantic structure can be effectively represented in lower dimensions compared to the 768 dimensions used by CamemBERT.

Significance and Claims
The paper claims that while deep supervised models are effective for local context tasks, they suffer from structural limitations in representing the global organization of semantic space. The discrete graph model based on co-occurrence cliques demonstrates:

A better gradation of semantic distances.
A more even distribution of the embedding space.
A more homogeneous graph structure regarding connectivity and cluster coherence.

The authors suggest that these findings point toward a complementary perspective: integrating graph-based structural insights could guide neural architectures toward more stable, interpretable convergence and potentially reduce the effective dimensionality required for semantic representation. The study is modest in scope, noting it is limited to the French language, a single corpus, and one specific model, and calls for further research to validate these findings across other languages and architectures.

The Two Map-Makers

The Big Discovery: Local vs. Global

The "Crowded Room" Problem

The Conclusion

More like this