KOMBO: Korean Character Representations Based on the Combination Rules of Subcharacters

1. Problem Statement

Current Pre-trained Language Models (PLMs) for Korean largely rely on subword tokenization methods (e.g., BPE, WordPiece, SentencePiece) borrowed from English NLP. While effective for English, these methods fail to account for the unique structural properties of the Korean writing system, Hangeul.

• The Linguistic Gap: Hangeul is a featural writing system where characters (syllables) are composed of subcharacters (Jamo: chosung [initial consonant], joongsung [vowel], and jongsung [final consonant]).
• The Limitation: Subword-based approaches treat Hangeul characters as opaque units or break them down arbitrarily based on frequency. They ignore the combination rules defined in the historical document Hunminjeongeum (1446), which dictates how subcharacters are spatially arranged (e.g., initial consonants placed above or to the left of vowels, final consonants below).
• Consequences: This oversight leads to:
  • Loss of structural linguistic information (e.g., difficulty handling compound words where subcharacters are inserted between morphemes).
  • Poor robustness against typos and morphological variations (conjugations).
  • Inefficient vocabulary sizes compared to character-level models.

2. Methodology: KOMBO Framework

The authors propose KOMBO (KOrean character representations based on the coMBinatiOn rules of subcharacters), a novel framework that integrates the design principles of Hangeul directly into the neural architecture.

Core Architecture

The model operates on a "subcharacter $\to$ character" hierarchy rather than the standard "subword $\to$ character" approach.

1. Initial Representations (Subcharacter Tokenization):

   • Input text is decomposed into atomic Jamo units (chosung, joongsung, jongsung).
   • If a character lacks a final consonant, a special empty token is used to ensure every character is represented by exactly three Jamo units.
   • These Jamo units are embedded into a vector space.

2. Contextualization:

   • Before merging, the model applies Local Transformer blocks followed by Gated Recurrent Units (GRU) to the sequence of Jamo embeddings. This captures the local context and sequential dependencies of the subcharacters.

3. Subcharacter Combination (The Core Innovation):

   • Instead of simple concatenation, the model merges Jamo units based on Hunminjeongeum rules:
     • Step 1 (Fusion): Chosung and Joongsung are merged via element-wise addition (reflecting their horizontal or top-left relationship).
     • Step 2 (Vertical Concatenation): The resulting representation is vertically concatenated with the Jongsung (reflecting the bottom position).
     • Step 3 (Aggregation): A Convolutional layer (with a $2 \times 1$ kernel) followed by Average Pooling processes the concatenated vector to generate a dense, unified character representation.

4. Transformer Stack:

   • The resulting character representations are fed into a standard Transformer encoder (similar to BERT) for higher-level contextualization.

5. Subcharacter Restoration:

   • To perform token-level tasks (like Masked Language Modeling), the model reconstructs the character representations back into their constituent Jamo units.
   • This involves duplicating the character vector, adding it to the original Jamo embeddings (inspired by U-Net skip connections), and passing it through GRU layers to ensure continuity.

6. Span-Subcharacter Masking:

   • During pre-training, instead of masking random tokens, the model masks consecutive triplets of Jamo (one full character: Chosung + Joongsung + Jongsung). This forces the model to learn the compositional relationships between subcharacters.

3. Key Contributions

• First Principles Approach: The paper is the first to explicitly ground Korean PLM architecture in the historical invention principles of Hangeul (Hunminjeongeum), moving beyond statistical tokenization.
• Novel Architecture: Introduces a hierarchical merging mechanism that mathematically mimics the spatial composition of Korean characters.
• Span-Subcharacter Masking: Proposes a masking strategy that aligns with the linguistic unit of the Korean character (the syllable block) rather than arbitrary subwords.
• Comprehensive Evaluation: Demonstrates that subcharacter-level modeling outperforms subword-level modeling across diverse tasks, including standard NLU, noisy text, and offensive language detection.

4. Experimental Results

The authors evaluated KOMBO (specifically the KOMBO-Jamo variant) against various tokenization baselines (Stroke, Cji, BTS, Jamo, Character, Morpheme, Subword, MorSubword) on standard and noisy datasets.

• Standard NLU Tasks:

  • KOMBO-Jamo achieved a 2.11% average improvement over the state-of-the-art Korean PLM (MorSubword) across five tasks: KorQuAD, KorNLI, KorSTS, NSMC, and PAWS-X.
  • It achieved the best results in 4 out of 5 tasks, with particularly strong performance in Machine Reading Comprehension (KorQuAD) and Semantic Similarity (KorSTS).
  • Efficiency: Despite using a vocabulary size of only 170 (compared to 32k for subword models), it outperformed models with massive vocabularies.

• Robustness to Typos (Noisy Data):

  • In experiments with injected typos (insertion, transposition, substitution, deletion), KOMBO-Jamo showed significantly higher robustness than subword-based models.
  • As the typo rate increased, the performance gap between KOMBO-Jamo and subword models widened, proving that understanding subcharacter structure helps the model recover from errors.

• Offensive Language Detection:

  • KOMBO-Jamo achieved the highest Macro F1 scores on three offensive language datasets (BEEP!, K-MHaS, KOLD).
  • The model demonstrated better handling of compound words often used in insults (e.g., "puppy" used as an insult), which subword models often tokenize incorrectly, breaking the semantic link.

• Ablation Studies:

  • Removing the contextualization layer caused a massive performance drop (~16%), proving the necessity of understanding subcharacter context before merging.
  • Removing the span-subcharacter masking reduced performance, confirming the importance of learning character composition.
  • Using standard downsampling methods (Attention/Linear pooling) instead of the proposed combination rules resulted in a >10% performance drop, validating the specific design of the merging layer.

5. Significance and Conclusion

This paper fundamentally challenges the "one-size-fits-all" subword tokenization paradigm for agglutinative and featural languages.

• Linguistic Alignment: It proves that aligning neural architectures with the intrinsic structural rules of a language (in this case, Hangeul) yields superior performance and robustness compared to purely statistical approaches.
• Efficiency: It demonstrates that a small, principled vocabulary (Jamo) can outperform large, data-driven vocabularies (Subwords) when the architecture respects the language's morphology.
• Future Impact: The framework suggests that for languages with complex morphological or featural structures, explicitly modeling the "atomic" building blocks and their combination rules is a critical step for next-generation PLMs. The authors also note potential for extending this hierarchical approach to subword-level representations in the future.