Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text

Language is often thought of as a stream of sound, but for many written systems, it begins as a stream of symbols. In most modern writing, spaces or punctuation marks tell the reader where one word ends and the next begins. However, some ancient scripts do not use these visual cues. Instead, they present a continuous line of characters, leaving the reader to figure out where the meaningful units lie. This is the challenge faced by scholars studying Tangut, a language that was once the official script of the Xixia dynasty in China but has been extinct for centuries. The surviving texts are precious windows into a lost world of history and culture, yet they remain difficult to read because the words are not separated. Without a way to automatically identify these word boundaries, computers cannot effectively search, translate, or analyze these historical documents.

For decades, researchers have focused on simply recognizing the individual characters in Tangut manuscripts, turning images of ink on paper into digital text. But recognizing a character is only the first step; understanding the language requires knowing which characters belong together to form a word. This task is exceptionally difficult because there are no native speakers left to consult, and the few experts who can read Tangut must rely on a combination of context, reconstructed pronunciation, and ancient dictionaries to guess where words begin and end. The scarcity of expert knowledge and the lack of large digital libraries make this a problem of extreme resource scarcity, where the usual tools of modern artificial intelligence often fail because they require vast amounts of labeled data to learn.

A team of researchers has now taken the first systematic step toward solving this puzzle by teaching a computer how to segment Tangut words using very limited human guidance. They started with a small, carefully curated collection of 2,750 text segments, totaling nearly 32,000 words, which were manually divided by specialists. This dataset, drawn from a Buddhist scripture and a secular encyclopedia, served as the foundation for their training. However, relying solely on this tiny amount of labeled data would not be enough to build a robust system. To overcome this, the researchers combined three distinct sources of knowledge: the expert-labeled examples, a traditional dictionary of Tangut words, and a massive collection of unlabeled Tangut text that had never been divided into words.

The researchers designed a system that treats the dictionary not as a rigid rulebook, but as a source of hints. In many cases, a single character could be part of several different dictionary entries, creating a web of overlapping possibilities. Instead of forcing the computer to pick just one path immediately, the system preserves all these potential candidates. It then uses a method of "reliability calibration" to weigh these hints. By looking at how often a dictionary entry appears in the text and how often it matches the expert's boundaries, the system learns to trust some dictionary entries more than others. This prevents the computer from blindly following a dictionary entry that might be incorrect for the specific context of the manuscript.

To further improve its understanding, the system also learned from the unlabeled text, which contained hundreds of thousands of characters but no word divisions. By analyzing how often characters appeared next to each other and how varied their neighbors were, the system built a statistical map of the language. It learned that certain character combinations appear frequently together, suggesting they form a single word, while others appear more randomly, suggesting a boundary between them. This statistical intuition was combined with a modern neural network, a type of computer program designed to recognize patterns, which was pre-trained on the unlabeled text to understand the context of characters before it ever saw the labeled examples.

The results of this approach were significant. When tested on unseen portions of the text, the system achieved a high level of accuracy, correctly identifying word boundaries in more than 91 percent of cases. This performance was notably better than systems that relied only on the small amount of labeled data or only on the dictionary. The study found that the combination of expert knowledge, dictionary hints, and statistical patterns from unlabeled text was essential; removing any one of these components caused the system's performance to drop. The system also proved particularly good at recognizing words that did not appear in the training examples, a common challenge in low-resource languages, by using the statistical patterns and dictionary knowledge to make educated guesses.

One of the most important findings was that the system's ability to generalize depended heavily on the type of text it was reading. While it performed exceptionally well on the secular encyclopedia, which made up the majority of the data, it was slightly less accurate on the religious scripture. This suggests that the system is still learning to adapt to different styles and vocabularies within the language. The researchers also noted that while the system could identify word boundaries with high precision, the task of assigning grammatical labels to those words remains a preliminary step, as the inventory of word types is still being refined by experts.

This work represents a crucial foundation for the future of Tangut studies. By successfully integrating traditional linguistic resources with modern machine learning techniques, the researchers have created a tool that can help scholars process and analyze these ancient texts more efficiently. The system does not replace the need for human experts, but it provides a powerful aid that can handle the repetitive task of segmentation, allowing researchers to focus on deeper historical and linguistic questions. As the team continues to expand their annotated data and refine their models, they hope to eventually apply these methods to a wider range of documents, bringing the lost language of the Xixia dynasty closer to modern understanding.