Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages

1. Problem Statement

Current state-of-the-art Natural Language Processing (NLP) models (e.g., BERT, GPT, Word2Vec) rely heavily on tokenization or sub-tokenization (e.g., WordPiece). While effective for high-resource languages like English, these approaches face significant limitations in low-resource and morphologically complex languages:

• Loss of Orthographic Similarity: Sub-tokenizers often split words arbitrarily. For example, "quality" might be a single token, while "qualification" is split into [qual], [ifi], [cation]. This breaks the semantic link between the shared root "qual" and the full word, preventing the model from inferring meaning based on spelling similarities.
• Handling Variations: Dictionary-based models struggle with spelling variations, typos, or different inflections (cases, numbers) unless the specific variation appears frequently in the training corpus.
• Data Scarcity: Low-resource languages (e.g., Faroese, Icelandic) lack the massive corpora required to learn meaningful representations for every inflected form or spelling variation.
• Context Window Constraints: Pure character-based models (processing every letter as a token) drastically increase sequence length, leading to quadratic memory growth in Transformers and reduced context windows, which is computationally prohibitive.

2. Methodology: Rich Character Embeddings (RCE)

The authors propose Rich Character Embeddings (RCE), a transformer-based approach that computes a single vector representation for a word directly from its character string, bypassing the need for a fixed dictionary of tokens or sub-tokens.

A. Input Representation

Instead of treating a word as a single token, RCE processes the word as a sequence of characters with specific encoding rules:

• Granularity: Text is split into words, and each word is processed character-by-character.
• Modifiers: Special characters (diacritics, uppercase letters, ligatures) are handled using a modifier system. For example, an uppercase 'T' is encoded as an [UP] modifier followed by the base character t.
• Special Tokens: Each word sequence is wrapped with [BEG] (beginning) and [END] tokens. Padding ([PAD]) is used for batch processing.
• One-Hot Encoding: Characters are converted into one-hot vectors based on a predefined alphabet dictionary (including base characters, digits, special symbols, and modifiers).

B. Model Architecture

• Encoder: The sequence of character one-hot vectors is passed through a Linear Layer to an embedding size, followed by Positional Encoding.
• Transformer Encoder: The data is fed into a standard Transformer Encoder.
• Output: Unlike sentence classification models that use the [CLS] token, RCE uses the output vector corresponding to the [BEG] token as the final word embedding. This single vector represents the entire word.

C. Training Objectives

To ensure the embeddings capture both semantic and syntactic information, the model is trained using a multi-task approach:

1. Context Prediction (Semantic): Similar to Word2Vec, the model predicts the character strings of surrounding words. This forces the embedding to capture semantic context.
2. Identity Prediction (Syntactic): The model attempts to reconstruct the input word's character string from its embedding (acting like an autoencoder). This preserves orthographic and morphological information (spelling, inflection).
3. Context Dictionary Prediction: A hybrid objective where the model predicts standard dictionary tokens for surrounding words if they exist, providing a simpler training signal to stabilize learning.

3. Key Contributions

• Drop-in Replacement: RCE can replace traditional dictionary or sub-token embeddings in existing architectures (from small models like Word2Vec to large models like BERT) without altering the rest of the model structure.
• Morphological Robustness: By operating at the character level, RCE naturally handles inflected forms, unknown words, and spelling variations without requiring them to be present in the training vocabulary.
• Hybrid Capability: The authors propose a hybrid model combining RCE (Transformer-based) with c2v (Convolutional character embeddings), leveraging the strengths of both architectures.
• Low-Resource Optimization: The method is specifically designed to perform well with limited data by leveraging character-level similarities across related languages (e.g., using Icelandic data to improve Faroese performance).

4. Experimental Results

The authors evaluated RCE on Low-Resource languages (Faroese, Latin, Uzbek) and Morphologically Complex languages (Latin, German, Icelandic) across several tasks:

• Intrinsic Evaluation (TopK & Odd-One-Out):

  • RCE and the hybrid RCE+c2v model consistently outperformed FastText and c2v alone.
  • The hybrid model achieved the highest scores, demonstrating that combining transformer and convolutional mechanisms yields superior semantic clustering.
  • Notably, models trained on related languages (e.g., Icelandic) using RCE outperformed models trained on the target language (Faroese) using FastText.

• Inflection Prediction (Latin):

  • In tasks predicting noun declension and case/number, RCE and c2v significantly outperformed FastText.
  • FastText showed a sharp performance drop when pre-trained on a different language (Italian) compared to the target (Latin), whereas RCE maintained robust performance, indicating better transferability of morphological rules.

• Stylistic Device Detection (German):

  • Chiasmus Detection: When relying only on embedding features (removing other hand-crafted features), the RCE+c2v hybrid significantly outperformed all baselines.
  • Metaphor Detection: RCE outperformed FastText and c2v individually, with the hybrid model achieving the highest accuracy (0.74).

• Downstream Task (SWAG-like Next Sentence Prediction):

  • When integrated into a BERT-like architecture, RCE outperformed standard WordPiece tokenization across English, German, Norwegian, and Icelandic.
  • In low-resource scenarios (Faroese), pre-training on related languages (Norwegian/Icelandic) with RCE yielded better results than pre-training on the target language with WordPiece.

5. Significance and Limitations

Significance:

• Bridging the Gap: RCE offers a solution for the "long tail" of languages where data is scarce and morphology is complex, areas where current sub-token models struggle.
• Efficiency: It avoids the quadratic memory explosion of pure character-level Transformers by generating a fixed-size vector per word, making it compatible with standard NLP pipelines.
• Generalization: The ability to infer meaning from spelling similarities allows the model to generalize to unseen words and variations, a critical feature for historical or endangered languages.

Limitations:

• Script Dependency: The current implementation is designed for linear, alphabetic systems (Latin script). It does not natively support 2D character structures (e.g., Chinese Hanzi) or non-Latin scripts (Cyrillic, Devanagari, Hangul) without redesigning the character-token mapping.
• Hardware Constraints: The evaluation was limited by available computational resources, restricting the scale of languages and datasets tested.

Conclusion

The paper demonstrates that Rich Character Embeddings (RCE) provide a superior alternative to sub-tokenization for low-resource and morphologically rich languages. By directly encoding character strings into semantic vectors, RCE captures both syntactic and semantic information more effectively than traditional methods, enabling better performance in downstream tasks and cross-lingual transfer scenarios.