Open Korean Historical Corpus: A Millennia-Scale Diachronic Collection of Public Domain Texts

This paper introduces the Open Korean Historical Corpus, a large-scale, openly licensed dataset spanning 1,300 years and multiple writing systems, which enables quantitative analysis of major linguistic shifts in Korean and serves as a foundational resource for training large language models.

The Gist

Imagine the Korean language as a massive, ancient library that has been slowly changing its architecture for over a thousand years. For centuries, the books were written in a foreign language (Chinese characters), then they started mixing that foreign script with a new native alphabet (Hangul), and eventually, they switched almost entirely to the native alphabet.

The problem? Most modern computer programs (AI) trying to read this library only know how to read the newest books. They get completely confused by the old, mixed-up, or foreign-looking pages. Until now, there was no single, open map to help them navigate this history.

Enter the "Open Korean Historical Corpus" (OKHC).

Think of this paper as the announcement of a massive, free, digital time machine. The researchers have gathered 17.7 million documents (like 5.1 billion words) from the 7th century all the way to 2025. They didn't just grab modern news; they dug up ancient royal diaries, old legal codes, colonial-era newspapers, and even North Korean state media.

Here is a simple breakdown of what they did and what they found, using some everyday analogies:

1. The "Time-Traveling Bookshelf"

Before this project, if a researcher wanted to study how Korean changed over time, they had to visit 19 different libraries, ask for permission, and hope the books were even digitized. It was like trying to build a puzzle where the pieces were scattered across the world in locked boxes.

The team collected all these pieces and put them in one giant, open box.

• The Collection: It includes texts in Classical Chinese (the "Latin" of East Asia), Idu (a tricky system where Chinese characters were used to write Korean grammar), Mixed Script (Hanja + Hangul), and pure Hangul.
• The License: They made sure the "box" is open for everyone to use for research, provided they don't sell it for profit.

2. What the Data Revealed (The "Detective Work")

Once they had this massive dataset, they ran it through computers to see how the language evolved. Here are their three biggest discoveries:

• The "Idu" Sunset:

  • The Analogy: Imagine a specific type of slang that was super popular in the 1800s but died out quickly.
  • The Finding: They found that a writing system called Idu (using Chinese characters to write Korean sounds) peaked in the 1860s. Then, it crashed. Why? Because the government passed new laws in the 1890s forcing people to use the native alphabet instead. The data shows this wasn't a slow fade; it was a sudden switch.

• The "Script Switch" Speed Run:

  • The Analogy: Think of it like a country switching from driving on the left side of the road to the right. You might expect it to take decades of confusion, but in Korea, it happened fast.
  • The Finding: For hundreds of years, almost everything was written in Chinese characters. But starting around 1890, the switch to the Korean alphabet (Hangul) happened incredibly fast. By the 1980s, over 93% of the text was pure Hangul. The "mixed" era was a brief, intense transition period.

• The "North vs. South" Vocabulary Gap:

  • The Analogy: Imagine two cousins who grew up in different countries. They speak the same language, but one cousin calls a "truck" a "lorry" and uses different words for "government." If you try to translate a letter from one to the other, your dictionary might get confused.
  • The Finding: Because North and South Korea have been separated for decades, their vocabularies have drifted apart. The researchers found that modern AI tools, trained on South Korean text, get 51 times more confused when reading North Korean news. They don't recognize words because North Korea uses unique spellings for foreign words (like calling Germany "Doichwolland" instead of the standard "Do-il-beun").

3. Why This Matters

Think of current AI models (like the ones powering chatbots) as students who only went to school in the last 20 years. They are great at reading modern text but fail when they encounter an old diary or a document with a mix of scripts.

This new corpus is like giving those students a history textbook.

• For Historians: It allows them to study language changes with math and precision, rather than just guessing.
• For AI Developers: It provides the "training data" needed to teach computers how to read ancient Korean, mixed scripts, and even North Korean dialects. This could help preserve culture and make AI smarter about the full history of the language, not just the modern version.

The Catch (Limitations)

The authors are honest about the flaws.

• Survivorship Bias: The library is heavy on modern books because more of them survived and were digitized. The ancient books are scarcer, so the "time machine" is a bit blurry for the very distant past.
• No Spoken Words: This is a library of written text. It doesn't include recordings of how people actually spoke, so we can't study the "sound" of history, only the "look" of it.

In a nutshell: This paper is the creation of a massive, open-access digital archive that lets us finally see the entire story of the Korean language, from ancient times to today, helping both humans and computers understand how the language has changed, split, and survived.

Technical Summary

Here is a detailed technical summary of the paper "Open Korean Historical Corpus: A Millennia-Scale Diachronic Collection of Public Domain Texts."

1. Problem Statement

The paper addresses a critical gap in Natural Language Processing (NLP) regarding the Korean language: the lack of accessible, large-scale historical corpora.

• Linguistic Discrepancy: The history of Korean is marked by a complex evolution from spoken language to written forms, involving a shift from Classical Chinese (Hanja) to the native Hangul alphabet, and the use of mixed scripts (Hanja-Hangul) and transcription systems like Idu (Korean-style Sinitic).
• Data Scarcity & Licensing: Existing Korean corpora (e.g., Sejong, Modu) are predominantly focused on modern Hangul texts. Historical datasets are often restricted by government licenses that prevent derivative works, are distributed in non-digital formats (e.g., DVDs), or exist only as non-downloadable web archives.
• Research Barrier: This lack of open data hinders quantitative diachronic analysis, the study of linguistic shifts, and the training of Large Language Models (LLMs) capable of understanding archaic scripts, Sino-Korean vocabulary, and North Korean lexical divergence.

2. Methodology

The authors constructed the Open Korean Historical Corpus (OKHC) through a rigorous pipeline involving data collection, preprocessing, and schema design.

A. Data Collection

• Scope: The corpus spans 1,300 years (7th century to 2025) and covers 6 languages/varieties: Middle Korean, Early Modern Korean, Modern Korean, North Korean, Classical Chinese, and Modern Japanese.
• Sources: Data was aggregated from 19 distinct institutional sources, including the National Institute of Korean History, National Library of Korea, Academy of Korean Studies, and North Korean state media (KCNA, Kim Il Sung University).
• Acquisition: Data was gathered via web scraping (using Python libraries like BeautifulSoup, HTTPX, Selenium) and direct downloads. The collection period was May 2025 to October 2025.
• Volume: The final dataset contains 17.7 million documents and 5.1 billion tokens (measured by o200k_base tokenizer).

B. Text Preprocessing

• Normalization: All text was standardized to Unicode NFKC form to resolve compatibility variants.
• Cleaning: Source-specific heuristics were applied to remove:
  • Boilerplate text (missing content indicators).
  • Modern inline translations (isolating original texts).
  • Erroneous metadata merged into text fields.
  • High noise-to-text ratio documents caused by encoding errors.

C. Language and Script Identification

• Hybrid Approach: Due to the unreliability of standard language ID models on historical texts, the authors used:
  1. GlotLID: Constrained to predict specific relevant languages (Korean, Japanese, Chinese).
  2. Character-level Analysis: Calculating proportions of Hangul, Hanja, and Kana.
  3. Rule-Based Classification: Integrating model outputs with source metadata (e.g., classifying Annals of the Joseon Dynasty as Classical Chinese based on origin).
• Periodization: Korean texts were assigned historical periods (Middle, Early Modern, Modern) based on publication year.

D. Data Schema

• The corpus is distributed in JSON Lines format.
• Each document includes fields for unique ID, raw/normalized text, publication year, language, script type, source, copyright status, and computed analytics (e.g., text length).

3. Key Contributions

1. First Openly Licensed Historical Corpus: OKHC is the first dataset to offer broad temporal coverage (7th–21st century) and diverse writing systems (including Idu and mixed scripts) under an open license (CC BY-NC 4.0 for text, MIT for code).
2. Inclusion of Under-Represented Systems: It uniquely captures Korean-style Sinitic (Idu) and Hanja-Hangul mixed scripts, which are largely absent in modern NLP resources.
3. North Korean Text Inclusion: It includes a significant volume of North Korean texts (KCNA, etc.) for linguistic analysis, strictly adhering to the National Security Act by providing metadata/URLs for copyrighted material rather than full text where restricted.
4. Legal Compliance: The dataset navigates complex copyright laws (South Korean copyright, database rights) by providing full text for public domain materials (pre-1963) and metadata-only for copyrighted works.

4. Results & Quantitative Analysis

The authors leveraged the corpus to perform three major quantitative analyses:

A. Temporal Dynamics of Idu (Korean-style Sinitic)

• Finding: Idu usage peaked in the 1860s and declined sharply thereafter.
• Correlation: The sharp decline correlates with the Kabo Reform (1894), which mandated Hangul in official documents, and a 1908 directive favoring Hanja-Hangul mixed script over Idu. This suggests language policy, rather than just preservation bias, drove the obsolescence of Idu.

B. Diachronic Shift from Hanja to Hangul

• Finding: The transition was rapid, not gradual.
• Timeline: Until 1890, written records were dominated exclusively by Classical Chinese. Following the Kabo Reform, Hanja-exclusive texts plummeted, and Hangul usage surged.
• Current State: By the 1980s, Hangul comprised over 93% of characters in documents, reflecting the successful transition to a Hangul-dominant society.

C. North Korean Lexical Divergence

• Finding: Modern tokenizers struggle significantly with North Korean text due to orthographic and lexical differences.
• OOV Rates: North Korean texts produced 3 to 51 times higher Out-of-Vocabulary (OOV) rates compared to South Korean texts across various models (e.g., mBERT, KLUE-BERT).
• Cause: The divergence stems from North Korea's unique orthography for loanwords (e.g., transliterating "Germany" as 도이췰란드 vs. South's 독일) and distinct native vocabulary (e.g., 돐 for "anniversary").

5. Significance and Future Impact

• Foundational Resource: OKHC provides the necessary data foundation for quantitative diachronic linguistics, allowing researchers to study language evolution across genres and eras with statistical rigor.
• LLM Pre-training: The corpus enables the pre-training of Large Language Models to better understand:
  • Archaic writing systems (Idu, Old Hangul).
  • Sino-Korean vocabulary in historical contexts.
  • North Korean dialects and orthography.
• Digital Humanities: It facilitates the digitization and analysis of centuries of historical records that were previously illegible to most modern Koreans due to script barriers.
• Ethical Considerations: The authors explicitly warn of biases in historical texts (gender, class, regional discrimination) and the political nature of North Korean state media, urging researchers to mitigate these biases during model training.

In summary, the Open Korean Historical Corpus bridges a critical data gap in Korean NLP, transforming the study of Korean linguistic history from a qualitative, manual endeavor into a scalable, quantitative science.