💬 NLP

SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models

The paper introduces SCRIPT, a model-agnostic module that injects subcharacter (Jamo) compositional knowledge into Korean pre-trained language models to enhance their performance on NLU and NLG tasks while better capturing grammatical and semantic regularities without requiring architectural changes or additional pre-training.

Original authors: SungHo Kim, Juhyeong Park, Eda Atalay, SangKeun Lee

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: SungHo Kim, Juhyeong Park, Eda Atalay, SangKeun Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching AI to See the "Legos" Inside Korean Words

Imagine you are trying to teach a robot to understand the Korean language. Currently, most AI models treat Korean words like solid bricks. If the word is "cold" (춥다), the AI sees it as one single, unbreakable block.

But Korean isn't built like a pile of bricks. It's built like LEGOs.

In Korean, every character is actually a small structure made of three tiny pieces called Jamo (like initial consonants, vowels, and final consonants). Just like you can snap LEGO bricks together to build different shapes, these Jamo snap together to create words. And when you change the tense or meaning of a word, you often only swap out one tiny LEGO piece, not the whole brick.

The Problem:
Current AI models are great at reading the "bricks" (subwords), but they are terrible at seeing the "LEGOs" inside. They miss the subtle changes. For example, changing a verb from "is cold" to "was cold" might just involve adding a tiny final consonant piece. The AI often misses this nuance because it's looking at the whole word as a single unit.

The Solution: SCRIPT
The authors created a module called SCRIPT. Think of SCRIPT as a special pair of glasses that you put on top of an existing AI. You don't have to rebuild the AI's brain; you just slide these glasses on.

These glasses allow the AI to:

Look under the hood: Instead of just seeing the whole word, the glasses break it down into its tiny Jamo components.
Understand the rules: The glasses know the "rules of the road" for how Korean letters are built (like knowing that a vowel always goes in the middle, and a final consonant goes on the bottom).
Reassemble the view: It then combines this "LEGO-level" understanding with the AI's original "brick-level" understanding to create a super-powered view of the word.

How It Works (The Creative Metaphors)

1. The "Dual-Channel" Strategy

Imagine a chef (the AI) who is trying to cook a complex Korean stew.

Channel A (The Old Way): The chef looks at the recipe and sees "Add Beef." They add a big chunk of beef.
Channel B (The SCRIPT Way): The chef also has a side screen showing the ingredients of that beef: "Salt, Pepper, Marbling."
The Fusion: SCRIPT takes the "Beef" from Channel A and the "Salt/Pepper" details from Channel B and mixes them together. Now, the chef knows exactly how the beef is seasoned, not just that it's beef.

2. The "Plug-and-Play" Module

Usually, to make an AI smarter about Korean, you have to fire the whole team and hire a new team from scratch (re-training the model). This is expensive and slow.
SCRIPT is like a software update. You don't need to replace the engine of your car; you just install a new GPS navigation system that knows the local streets better. It works with whatever car (AI model) you already have, whether it's a small sedan (BERT) or a massive truck (GPT-3).

3. The "LEGO" vs. "Brick" Analogy

The Brick (Subword): The AI sees the word "춥다" (cold).
The LEGO (Jamo): SCRIPT sees:
- Choseong (Initial): ㅊ (Ch)
- Jungseong (Vowel): ㅜ (u)
- Jongseong (Final): ㅂ (p)
The Magic: When the word changes to "추웠다" (was cold), the AI sees a totally different brick. But SCRIPT sees that the "Ch" and "u" stayed the same, and only the bottom piece changed. This helps the AI understand that these two words are cousins, not strangers.

Why Does This Matter? (The Results)

The researchers tested this "glasses" module on many different tasks, from reading comprehension to writing stories.

Better Grammar: The AI got much better at fixing mistakes. If a human wrote "I go school" instead of "I go to school," the AI could spot the missing tiny piece.
Smarter Writing: When asked to write a story, the AI used the correct endings and particles (the tiny grammatical glue in Korean) much more often.
Speed: Because SCRIPT is a lightweight add-on, it didn't slow the AI down significantly. It was much faster than previous methods that tried to rebuild the whole AI from the ground up.

The Bottom Line

Korean is a language of structure and composition. It's not just a list of words; it's a system of building blocks.

SCRIPT is a clever tool that teaches AI to stop looking at Korean words as static blocks and start seeing them as dynamic, buildable structures. By injecting this "LEGO knowledge" into existing AI models, the authors made them smarter, more accurate, and better at understanding the nuances of the Korean language—all without needing to rebuild the AI from scratch.

In short: They gave the AI a better pair of eyes to see the tiny details that make Korean work.

1. Problem Statement

Korean is a morphologically rich, agglutinative language with a unique featural writing system called Hangul. In Hangul, each character is a structured composition of smaller subcharacter units known as Jamo (Initial consonant/Choseong, Vowel/Jungseong, Final consonant/Jongseong).

The Limitation: Most contemporary Korean Pre-trained Language Models (PLMs) rely on subword tokenization (e.g., BPE, WordPiece). While effective for lexical semantics, subword tokenization fails to explicitly capture the internal compositional structure of Hangul characters.
The Consequence: This limitation reduces model sensitivity to fine-grained morphophonological variations (e.g., tense inflection, honorifics, phonological assimilation) that occur at the subcharacter level. Existing subcharacter-based models (e.g., Jamo-level models) often require full pre-training from scratch, suffer from weaker semantic representations, and incur high computational costs due to longer sequence lengths.

2. Methodology: SCRIPT

The authors propose SCRIPT (Subcharacter Compositional Representation Injection Module), a model-agnostic, plug-and-play module designed to inject subcharacter structural knowledge into existing subword-based PLMs without architectural changes or additional pre-training.

Core Architecture

SCRIPT operates at the embedding layer of a PLM using a dual-channel strategy:

Parallel Tokenization:
- Channel A (Original): The standard subword tokenizer produces the PLM's original subword sequence.
- Channel B (SCRIPT): A subcharacter tokenizer decomposes the text into Jamo units (I, V, F).
Hierarchical Compression (The "Structure-Aware" Path):
SCRIPT compresses the Jamo sequence back into subword-level representations through two stages, strictly adhering to Hangul's compositional principles:
- Stage 1: Subcharacter-to-Character:
  - Fusion of Choseong (I) and Jungseong (V): Merged via element-wise summation.
  - Addition of Jongseong (F): The F component is vertically concatenated with the (I+V) representation to mimic the visual spatial arrangement of Hangul.
  - Convolution & Pooling: A 2D convolution and average pooling are applied to merge these aligned components into a dense character representation.
- Stage 2: Character-to-Subword:
  - A GRU layer captures the sequential order of characters within a subword.
  - A pooling operation (selecting the final character representation) aggregates these into a unified subword representation ( $h_S$ ).
Fusion Mechanism:
- The compressed, structure-aware subword representation ( $h_S$ ) is fused with the original pre-trained subword embedding ( $e_S$ ) using a Cross-Attention layer.
- This creates a final fused representation ( $e_F$ ) that retains the rich semantic knowledge of the pre-trained model while incorporating explicit Hangul structural information.

Key Design Principles

Model-Agnostic: Can be applied to both Encoder-only (BERT) and Decoder-only (GPT) architectures.
No Pre-training: Requires only fine-tuning; the base PLM weights remain intact.
Granularity Alignment: The module compresses fine-grained Jamo data back to the subword granularity of the target PLM, ensuring compatibility.

3. Key Contributions

Empirical Motivation: The authors conducted a large-scale analysis of a Korean POS-tagged corpus, revealing that 92.75% of morphological modifications occur at the subcharacter level (e.g., changing a final consonant), rather than the character level. This validates the necessity of subcharacter-aware modeling.
Novel Module (SCRIPT): Introduction of a lightweight, plug-and-play module that injects compositional knowledge via embedding-level integration, avoiding the computational overhead of full subcharacter-based pre-training.
Comprehensive Evaluation: Demonstrated performance gains across a wide range of Korean NLU (Natural Language Understanding) and NLG (Natural Language Generation) tasks.
Linguistic Analysis: Provided evidence that SCRIPT reshapes the embedding space to better capture grammatical regularities and semantic cohesion among inflected predicates.

4. Experimental Results

The authors evaluated SCRIPT on four Korean PLMs (BERTbase, KoGPT2base, KoGPT3-1.2B, EXAONE-2.4B) across nine NLU and three NLG tasks.

NLU Performance: SCRIPT improved performance across all baselines on standard benchmarks (KorNLI, KorSTS, NSMC, PAWS-X) and the complex KoBEST suite.
- Example: BERTbase + SCRIPT outperformed the state-of-the-art Jamo-based model (KOMBObase) despite using the same underlying architecture, achieving an average gain of 1.6% on KoBEST tasks.
NLG Performance: Significant improvements were observed in generative tasks, particularly KoCommonGen (commonsense reasoning) and Korean GEC (Grammatical Error Correction).
- GEC Highlight: On the Kor-Learner dataset (rich in particle and conjugation errors), SCRIPT exceeded the best baseline by >3.2%, demonstrating its ability to handle subtle morphological errors.
Efficiency:
- SCRIPT adds minimal computational overhead compared to the base model (approx. 1.7 GB GPU memory increase vs. >10 GB for full subcharacter models like KOMBO).
- It converges faster during fine-tuning than both the base model and full subcharacter models.

5. Significance and Impact

Bridging the Gap: SCRIPT successfully bridges the gap between the semantic richness of large-scale subword pre-training and the morphological precision of subcharacter modeling.
Practical Deployment: Unlike previous approaches that require training new models from scratch, SCRIPT allows researchers and practitioners to upgrade existing, powerful LLMs with linguistic structural knowledge instantly.
Linguistic Insight: The study confirms that explicitly modeling the compositional rules of Hangul (Choseong $\to$ Jungseong $\to$ Jongseong) is critical for Korean NLP, particularly for tasks involving grammar, inflection, and generation.
Generalizability: While focused on Korean, the modular design suggests potential applicability to other languages with rich internal character structures or complex morphology.

In conclusion, SCRIPT represents a significant step forward in Korean NLP by proving that structural injection at the embedding level is a more efficient and effective strategy than structural retraining for morphologically rich languages.