Optimizing Korean-Centric LLMs via Token Pruning

1. Problem Statement

The proliferation of massive multilingual Large Language Models (LLMs) has introduced a "curse of multilinguality." While these models offer universal applicability, a significant portion of their parameters—specifically within vocabulary and embedding layers—is allocated to languages irrelevant to specific downstream applications (e.g., Korean-centric services).

• Redundancy: For monolingual or bilingual deployments, this creates unnecessary memory overhead and computational inefficiency.
• Cultural Dilution: In dominant English or Chinese-centric models, Korean data often comprises a small minority, risking the dilution of cultural nuance and linguistic proficiency.
• Deployment Constraints: Local deployment requires lightweight models, but existing optimization strategies often compromise performance.

The paper addresses the need to systematically quantify how token pruning (removing tokens for non-target languages) affects performance, stability, and efficiency in Korean NLP tasks.

2. Methodology

Target Models and Configurations

The authors evaluated a diverse set of open-weight multilingual LLMs, including:

• Qwen3 series (0.6B to 14B parameters)
• Gemma-3 (270M to 12B)
• Llama-3 family (3.1-8B, 3.2-3B)
• Others: Tri, Ministral-8B, and Aya-23.

Each model was processed into three distinct vocabulary configurations:

1. Original: Retains the full multilingual vocabulary.
2. EnKo: Pruned to retain only English and Korean tokens.
3. EnKoZh: Pruned to retain English, Korean, and Chinese tokens.

Pruning Procedure

The authors implemented a "language-aware filtering" strategy involving three stages:

1. Categorization: Tokens were classified by language using Unicode block ranges and script properties (Hangul, Latin, Hanzi).
2. Filtering & Remapping: Irrelevant tokens were discarded, and retained tokens were mapped to a new, continuous index space.
3. Physical Rearrangement: The embedding matrix and output projection layer were physically rearranged to align with the new indices.

• Note: The internal transformer blocks and positional encodings remained intact to preserve sequence modeling capabilities.

Evaluation Benchmarks

Performance was assessed across four categories using specific Korean-centric benchmarks:

• General Aptitude: KMMLU (General knowledge and logical reasoning).
• Cultural & Linguistic Understanding: HAERAE (Cultural literacy) and CLIcK (Linguistic nuance).
• Instruction Following: LogicKor (Reasoning) and KoMT-Bench (Constraint satisfaction).
• Machine Translation: WMT 24++ (Korean–English), scored via XCOMET-XXL.
• Stability: Word-level Pass Rate (WPR) to measure language confusion.
• Efficiency: Inference latency measured on the Google/WMT-pp dataset.

3. Key Results

General Aptitude and Cultural Alignment

• Stability: Pruned models showed minimal performance degradation.
• Variance: For the Qwen3 series, variance between configurations was negligible (< 0.01).
• Improvement: Some models, like Gemma-3-12b-it and Llama-3.2-3B-Inst, showed slight improvements in linguistic capability or accuracy after pruning, suggesting that narrowing the vocabulary search space may reduce generation noise in smaller models.

Instruction Following

• Architecture Dependency: Results varied significantly based on model architecture and pruning depth.
• Qwen3 Series: Retaining Chinese tokens (EnKoZh) consistently outperformed strict EnKo pruning. For example, Qwen3-4B scored 7.85 on LogicKor (EnKoZh) vs. 7.77 (Original), suggesting these models rely on latent cross-lingual alignments (specifically Chinese) for reasoning.
• Llama-3 Series: Larger models like Llama-3.1-8B-Inst showed improved performance in the stricter EnKo setting (5.36 → 5.57), indicating vocabulary reduction can sharpen instruction focus for sufficiently large models.

Machine Translation

• Significant Gains: Pruned configurations consistently matched or outperformed baselines.
• Notable Improvements:
  • Llama-3.1-8B-Inst: Improved from 0.5879 to 0.6342 (COMET score).
  • Aya-expanse-8b: Improved from 0.6957 to 0.7496.
• Mechanism: Eliminating extraneous tokens regularizes the output distribution, minimizing off-target hallucinations and enhancing the English-Korean translation pathway.

Language Confusion (Stability)

• Dramatic Recovery: Token pruning significantly mitigated language confusion.
• WPR Improvement: Qwen3-4B, which had the lowest baseline stability (0.8882), achieved a massive recovery ($\Delta$WPR = +0.1041) to 0.9923.
• Consistency: Stable baselines like Qwen3-0.6B achieved near-perfect consistency (> 0.999) post-pruning.

Computational Latency

• Memory vs. Speed: While vocabulary size was reduced substantially (e.g., 36% reduction in EnKo), the inference latency improvement was modest (0.89%).
• Bottleneck: This confirms that while pruning alleviates memory overhead related to embeddings, it does not significantly impact the computational bottlenecks found in attention mechanisms.

4. Key Contributions

1. Systematic Benchmark: The first comprehensive evaluation of token pruning on State-of-the-Art (SOTA) multilingual LLMs specifically for Korean NLP tasks.
2. Validation of Stability: Demonstrated that aggressive vocabulary compression eliminates language confusion, leading to near-perfect generation consistency (WPR > 0.99).
3. Performance Enhancement: Proved that pruning can actually improve machine translation quality and, in some cases, instruction following, by reducing noise and off-target hallucinations.
4. Architectural Insights: Highlighted that the optimal pruning strategy depends on the model's pre-training data; models heavily pre-trained on Chinese corpora (like Qwen) benefit from retaining Chinese tokens for reasoning, whereas others benefit from strict bilingual pruning.

5. Significance

This paper validates token pruning not merely as a compression technique, but as a critical optimization strategy for sovereign AI and resource-constrained local deployments.

• Efficiency: It offers significant memory savings by reducing parameter counts without altering the core transformer architecture.
• Quality: It enhances generation stability and translation quality, addressing the "curse of multilinguality."
• Strategic Deployment: The findings provide a roadmap for deploying high-performance, stable, and memory-efficient LLMs tailored specifically for Korean-centric applications, balancing the trade-off between model size and linguistic competence.