💬 NLP

Examining the Limits of Word2Vec with Toki Pona

This study demonstrates that Word2Vec can effectively capture semantic relationships even with an extremely small vocabulary of approximately 130 words, provided there is sufficient distributional data, and reveals that incidental non-core tokens in the corpus actually enhance word proximity rather than hinder performance.

Original authors: Daniel Zhenhan Huang, Hongchen Wu

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Zhenhan Huang, Hongchen Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart library assistant named Word2Vec. Its job is to understand how words relate to each other. Usually, this assistant learns by reading massive libraries containing hundreds of thousands of different words. It figures out that "king" is related to "queen" because they often appear in similar stories.

But what happens if you give this assistant a tiny, minimalist dictionary with only 130 words? That's the challenge this paper tackles using Toki Pona, a made-up language designed to be as simple as possible.

Here is the story of what the researchers found, explained through simple analogies:

1. The Tiny Dictionary vs. The Big Noise

The researchers gathered over 1.4 million sentences of Toki Pona from the internet. However, there was a catch. About 23% of these sentences contained "foreign" words—like people's names (Sonja), brand names (Discord), or slang that doesn't belong in the official 130-word dictionary.

Think of it like trying to teach a child to recognize colors using only a box of 130 crayons. But, every now and then, someone hands the child a random sticker of a car or a celebrity's face.

The Question: Does this "noise" (the stickers) confuse the child, or does it actually help them understand the crayons better?

To find out, the researchers built two versions of the assistant:

The "Full" Model: Learned from everything, including the random names and slang.
The "Pure" Model: Learned only from the strict 130-word dictionary, with all the "noise" removed.

2. The Test: Grouping Words Like a Puzzle

To see if the assistants were doing a good job, the researchers didn't ask them to solve complex riddles (which is hard with only 130 words). Instead, they asked them to sort words into groups, like putting all "animals" in one pile and all "colors" in another.

They used two methods to check the work:

The Silhouette Score: Imagine a group of friends standing in a room. A high score means the friends in the "Animal" group are huddled tightly together, while the "Animal" group is far away from the "Color" group.
The Centroid Check: They calculated the "center point" of each group (like the average location of all the animals) and checked how close each word was to its own group's center.

3. The Surprising Results

The results were counter-intuitive and fascinating:

The "Noise" Actually Helped: The model that included the random names and slang (the "Full" model) actually did a slightly better job at keeping related words close together than the "Pure" model.
- The Analogy: Think of the random words as extra glue. Even though "YouTube" isn't a Toki Pona word, it often appears near words for "website" or "video." This extra connection helped the assistant realize, "Ah, these words belong in the same neighborhood!" The noise didn't confuse the map; it made the neighborhoods denser and easier to find.
The Structure Held Up: Both models successfully figured out that words like "dog," "cat," and "bird" belong together, and "red," "blue," and "green" belong together. They did this even though the dictionary was so small that almost every word had to do double or triple duty (polysemy).
- The Analogy: It's like trying to organize a closet with only three hangers. You'd expect it to be a mess. But the assistant managed to hang the shirts on one, the pants on another, and the shoes on the third, keeping them surprisingly organized.
The "Pure" Model Was More Tense: The model with only the 130 words (no noise) created a space where words were very close to each other, but it was harder to tell the groups apart. It was like a crowded room where everyone is standing shoulder-to-shoulder; you can't easily see where one group ends and another begins. The "Full" model had a bit more breathing room, making the groups clearer.

4. The Big Takeaway

The main lesson from this paper is about how these AI assistants learn.

Usually, we think you need a huge vocabulary (thousands of words) to teach an AI about meaning. This study shows that you don't. Even with a tiny vocabulary of just 130 words, if the words appear in enough different contexts (patterns), the AI can still build a smart map of how things relate to each other.

It's like saying you don't need a million different tools to understand how a house is built; if you watch a master carpenter use just a hammer, saw, and screwdriver enough times in different situations, you can eventually understand the whole house.

In short: Word2Vec is surprisingly robust. It can learn deep meaning even from a tiny, minimalist language, and a little bit of "noise" (like names and slang) actually helps it draw the connections between words more clearly.

Technical Summary: Examining the Limits of Word2Vec with Toki Pona

Problem Statement
Word embeddings, particularly Word2Vec, have been extensively validated on natural languages with large vocabularies (tens to hundreds of thousands of words). However, their efficacy at the extreme lower bound of vocabulary size remains untested. This study investigates whether Word2Vec can successfully capture semantic relationships in Toki Pona, a constructed language with a core vocabulary of only ~130 words. In such a minimalist system, words are highly polysemous (e.g., tomo refers to both rooms and vehicles), and co-occurrence profiles are inherently less discriminative due to high-frequency interactions between a small set of terms.

A secondary challenge addressed is the presence of "linguistic noise." Approximately 23% of sentences in the Toki Pona community corpus contain non-core tokens, including named entities, loanwords, and neologisms. It is unclear whether these tokens provide necessary contextual enrichment or act as noise that obscures the semantic structure of the core vocabulary.

Methodology
The researchers constructed a corpus of 1.42 million sentences (7.95 million tokens) from the Toki Pona community (2016–2025). To isolate the impact of non-core tokens, two distinct datasets were prepared:

Full Corpus: Retains all tokens, including non-Toki Pona words.
Pure Toki Pona Corpus: Filters out all sentences containing non-core tokens, resulting in 1.10 million sentences (5.90 million tokens).

Two Word2Vec models were trained from scratch using the skip-gram algorithm with a window size of 5 over 10 epochs. Based on lexicon size and corpus characteristics, the optimal vector dimensionality was determined to be 24 dimensions.

Evaluation was conducted using three primary methods, adapted for the constraints of a minimal vocabulary:

Category Centroid Proximity: A ground-truth set of 27 semantic categories (derived from WordNet supersenses but adapted for Toki Pona's syntactic flexibility) was used. Cosine similarity between word vectors and category centroids was measured against a dynamic threshold (mean similarity + 1 standard deviation).
Unsupervised Clustering: Agglomerative hierarchical clustering with complete linkage was applied to determine if semantic clusters emerge naturally. Silhouette scores were calculated across varying distance thresholds to identify optimal cluster separation, compared against a random baseline.
Cross-Linguistic Comparison: Representational Similarity Analysis (RSA) was used to compare the semantic structure of the Toki Pona models against a pre-trained English model (Google News corpus) to assess convergent validity.

Key Results

Structural Stability: Both models achieved silhouette scores (0.261 for Full Corpus, 0.267 for Pure Toki Pona) significantly higher than the random baseline (0.166). The structural arrangement of clusters (e.g., animals, colors, kinship) remained largely invariant between the two models, indicating that sparse non-core tokens do not disrupt the fundamental semantic topology.
Impact of Non-Core Tokens: Contrary to the hypothesis that non-core tokens act as noise, the Full Corpus model demonstrated higher semantic density. The mean cosine similarity to category centroids was higher in the Full Corpus model ( $\mu=0.879$ ) compared to the Pure Toki Pona model ( $\mu=0.817$ ). This suggests that incidental tokens (e.g., the named entity Jutu for YouTube) provide contextual bridges that pull related core concepts (e.g., lipu for website) closer together in the vector space.
Category Retrieval: Both models successfully identified 124 out of 125 word-category assignments. While the overall structure was stable, the removal of non-core tokens affected specific abstract categories (e.g., evaluative.positive) more than concrete ones (e.g., animals).
Cross-Linguistic Consistency: Qualitative analysis of representational similarity matrices revealed that the relative proximity patterns between semantic categories in Toki Pona mirrored those found in the English model, suggesting the models learned genuine semantic structures rather than corpus artifacts.

Significance and Claims
The paper claims that Word2Vec's effectiveness is driven more by distributional patterns than by the sheer size of the lexicon, even at the extreme lower bound of ~130 words. The study demonstrates that meaningful semantic structure can emerge in a highly polysemous, minimalist language. Furthermore, it challenges the assumption that "noise" in the form of loanwords and named entities necessarily degrades embedding quality; in this context, such tokens appear to enhance the cohesion of semantic clusters by providing additional distributional context. The findings suggest that word embedding models are robust enough to function in environments with radically reduced vocabularies, provided sufficient distributional data is available.

1. The Tiny Dictionary vs. The Big Noise

2. The Test: Grouping Words Like a Puzzle

3. The Surprising Results

4. The Big Takeaway

Technical Summary: Examining the Limits of Word2Vec with Toki Pona

More like this