💬 NLP

A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments

This paper proposes a subword embedding-based method that trains on raw text to automatically detect and cluster lexical and orthographic variations in Luxembourgish user comments, demonstrating that distributional modeling can effectively reveal meaningful linguistic patterns in low-resource, noisy settings without relying on prior normalization or manual annotation.

Original authors: Anne-Marie Lutgen, Alistair Plum, Christoph Purschke

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anne-Marie Lutgen, Alistair Plum, Christoph Purschke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a bustling, chaotic marketplace where everyone is shouting different versions of the same words. Some people spell "tomorrow" as muer, others as muar, and some even as moar. Some say "long" as laang, others as lang. In a traditional computer language program, this chaos is usually treated as noise—like static on a radio. The computer's usual job is to "clean" the signal, forcing everyone to say the "correct" word so the machine can understand it.

This paper is about turning off the noise-canceling headphones and actually listening to the static.

The researchers, working with the Luxembourgish language, realized that these "mistakes" and variations aren't just errors. They are actually a rich, hidden map of how people really speak, where they are from, and how they feel.

Here is the simple breakdown of what they did and why it matters:

1. The Problem: The "Clean-Up" Crew

Usually, when computers analyze language, they act like a strict editor. If you write "colour" but the computer expects "color," it changes it. If you write a slang word, it deletes it.

The Analogy: Imagine trying to study a forest, but you first cut down every tree that isn't perfectly straight and painted it green. You end up with a neat lawn, but you've lost all the information about the ecosystem.
The Issue: For a small language like Luxembourgish, which doesn't have a huge dictionary or strict rules for online writing, this "cleaning" throws away valuable clues about culture and region.

2. The Solution: The "Word Detective" Tool

Instead of cleaning the text, the authors built a new tool that acts like a detective looking for patterns in a messy room.

How it works: They fed a massive pile of raw user comments (1.4 million of them!) into a computer model. Instead of asking "Is this word spelled right?", the model asked, "Who hangs out with whom?"
The Magic: The model learned that words that appear in similar sentences or are spelled similarly (even if misspelled) belong to the same "family."
The Result: It grouped words together automatically. It didn't need a pre-written list of "correct" spellings. It just looked at the data and said, "Hey, laang, lang, and laang all seem to be talking about the same thing."

3. What They Found: The Hidden Map

Once the computer grouped these words, the researchers looked at the "families" and found fascinating patterns:

The "Spelling Rebels" (Orthographic Variation): They found that people often spell words based on how they sound. For example, the word for "long" (laang) was often written as lang because the writer heard the vowel as short. The computer caught this and grouped them, showing that the "mistake" is actually a consistent rule used by many people.
The "Regional GPS" (Regional Variation): This is the coolest part. They found that certain spellings act like a GPS.
- Example: The word for "tomorrow" (muer). People in the south of Luxembourg often write it as muar, while people in the east might write moar. The computer didn't need a map; it figured out that these words cluster together based on who wrote them, effectively drawing a map of the country just by looking at spelling habits.
The "Time Travelers" (Lexical Variation): They found groups of words that mean the same thing but come from different eras or influences, like the German word schnell vs. the French-influenced séier for "fast." The computer showed that older generations tend to use one, while younger ones use the other.

4. Why This Matters

Think of this approach as archaeology for language.

Old Way: Digging up a site and only keeping the perfect, polished statues.
New Way: Digging up the site and keeping the broken pottery shards, the muddy footprints, and the graffiti. Why? Because the broken pieces tell you more about the daily life of the people than the perfect statues do.

The Big Takeaway:
This paper proves that we don't need to force small or informal languages into a "perfect" box to study them. By letting the computer embrace the messiness, we can discover:

How language actually evolves in real-time.
Where people are from just by how they type.
New words and meanings that dictionaries haven't caught yet.

It's a tool that turns "typos" into treasure, showing us that in the messy world of online comments, the "errors" are actually the most interesting part of the story.

1. Problem Statement

In Natural Language Processing (NLP), linguistic variation (spelling differences, orthographic inconsistencies, and regional forms) is often treated as "noise" and removed via normalization or predefined variant lists. This approach erases valuable sociolinguistic signals regarding geography, social structure, and dialect.

The Gap: Existing tools (e.g., VARD) rely on dictionaries and edit-distance methods to normalize text to a canonical form, rather than discovering new variants. Current embedding studies often focus on representation quality using curated variant pairs rather than mining unconstrained orthographic families from raw text.
The Challenge: For under-resourced languages like Luxembourgish, which has high orthographic diversity, partial standardization, and heavy code-switching, identifying variation without prior dictionaries or manual annotation is difficult.

2. Methodology

The authors propose a reproducible, unsupervised methodology that treats spelling diversity as linguistic structure rather than noise. The approach consists of three stages:

Stage 1: Distributional Subword Embeddings

Model: The authors use FastText to train subword embeddings on raw, unnormalized text.
Mechanism: Unlike standard word embeddings, FastText represents words as bags of character $n$ -grams. This allows the model to capture orthographic overlap and morphological similarity even for low-frequency or misspelled tokens.
Input: A large corpus of raw text (JSON format). Minimal cleaning is applied (e.g., removing @ mentions, optional lowercasing).
Configuration: Key hyperparameters include vector size (100), context window (5), and character $n$ -gram ranges (3–7).

Stage 2: Inducing Variant Families

Similarity Metrics: For a seed token $w$ $w$ , the system retrieves neighbors based on:
1. Cosine Similarity: Measures semantic and syntactic proximity in the embedding space.
2. Jaccard Similarity: Measures character $n$ -gram overlap to capture orthographic similarity.
Clustering Modes:
- Open Mode: Forms local "star" graphs around seeds.
- Strict Mode (Used in this study): Builds an undirected graph and extracts connected components.
Pruning & Filtering: Groups are filtered based on:
- Minimum family size ( $SNN\_MIN$ ).
- Maximum frequency ratio ( $MAX\_FREQ\__RATIO$ ) to prevent dominant standard forms from swallowing rare variants.
- Dimension-based aggregation (e.g., tracking which users or time periods use specific variants).
Output: A JSONL file of variant families and a CSV summary with cohesion scores (harmonic mean of cosine and Jaccard scores).

Stage 3: Qualitative Analysis

The automatically induced clusters are manually reviewed using a bottom-up approach.
Families are classified into categories (Orthographic, Morphological, Lexical, etc.) to validate linguistic coherence and trace patterns not captured by numerical criteria alone.

3. Key Contributions

Novel Methodology: A framework for inducing lexical and orthographic variation directly from raw text using subword embeddings and similarity-based grouping, without relying on predefined variant lists, normalization rules, or seed lexicons.
Empirical Study on Luxembourgish: A large-scale analysis of ~1.42 million user comments from the RTL3 news platform (2008–2024), demonstrating the method's ability to capture systematic variation in a low-resource, multilingual context.
Transparent Framework: The code is open-source, providing a reproducible pipeline that supports both quantitative clustering and qualitative linguistic interpretation.

4. Results

The study identified approximately 800 variant families, classified into seven categories:

Orthographic (394 families): The largest category. Captures non-official spellings (e.g., laang vs. lang) and phonological variants (e.g., êng vs. öng for "one"). The model successfully aligned standard and non-standard variants in the same embedding space.
Morphological (222 families): Captured conjugated verbs and inflections (e.g., fillen, fillt, fille).
Lexical (115 families): Identified semantically related lexemes and synonyms (e.g., séier vs. schnell for "fast").
Regional (8 families): Detected regional dialects (e.g., muer vs. muar vs. moar for "tomorrow"). The analysis revealed that the variant moar (eastern Luxembourg) was more frequent in comments than previously recorded in the official Variation Atlas.
Other Categories: Included Collocations, Tokenization issues, and "Other" (pragmatic expressions like hoho).

Key Findings:

Pattern Consistency: The model identified that specific orthographic error patterns (e.g., replacing diphthong <éi> with <èi>) cluster consistently across different words, suggesting a shared social or stylistic meaning.
Discovery of Unrecorded Variants: The method found variants (like moar) that were underrepresented in existing linguistic atlases but prevalent in user-generated content.
Dimensional Aggregation: The method successfully tracked how variants are distributed across users and time, revealing that certain variants are consistent to specific author groups rather than random noise.

5. Significance and Limitations

Significance:

Paradigm Shift: Moves from "normalization" (fixing text) to "mining" (discovering structure). It proves that distributional modeling can reveal meaningful sociolinguistic patterns in "noisy" data.
Applicability: Particularly effective for low-resource, non-standardized, or emerging standard languages where spelling diversity encodes social information.
Reproducibility: Provides a transferable framework for studying variation in multilingual contexts without needing extensive manual annotation.

Limitations:

Corpus Bias: The data comes from a single online platform (RTL3), which may not represent the entire Luxembourgish-speaking population or capture all regional/social variations.
Interpretation: The method identifies candidate families based on statistical similarity. It cannot inherently distinguish between linguistic variation, author-specific idiosyncrasies, or temporal shifts without manual qualitative validation.
Standardized Languages: The approach is less effective for highly standardized languages where orthographic variation is minimal and existing normalization pipelines already handle most variations.

Conclusion:
The paper demonstrates that subword embeddings combined with similarity-based clustering offer a powerful, unsupervised tool for uncovering the hidden structure of language variation. It provides a structured starting point for linguistic analysis in contexts where traditional normalization would obscure valuable sociolinguistic signals.