A Lemmatized Corpus of the Pali Canon with Critical Apparatus
This paper presents an open-access, lemmatized corpus of the 1.6-million-token Pāli Canon featuring a critical apparatus across five textual traditions, specialized handling for sandhi and metrical variants, and high-coverage lemmatization that enables advanced computational linguistic tasks previously infeasible for this less-resourced language.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Language is a living thing, but the words of ancient texts often feel like they are trapped behind glass. For scholars studying the classical languages of the ancient world—such as Sanskrit, Latin, and Greek—computers have become powerful tools for unlocking these texts. These tools can count words, track how meanings shift over time, and even map the grammatical relationships between words in a sentence. However, there is a significant gap in this digital progress. While the great classical languages have been fully mapped by computers, the Pāli language, which holds the sacred scriptures of Theravāda Buddhism, has largely been left behind. The texts exist in vast quantities, but they remain difficult for machines to read because the words change their shape constantly depending on their role in a sentence, and they often run together in ways that hide their true meaning. Without a way to break these words down into their basic dictionary forms, computers cannot understand the stories, the philosophy, or the history contained within them.
Dan Zigmond, an independent scholar, has now built a bridge across this gap. He has created a massive, lemmatized corpus of the Pāli Canon, the complete collection of Buddhist scriptures. In simple terms, "lemmatization" is the process of taking every single word in a text and finding its original, unchanging root form. Imagine a library where every book is written in a code where the word "run" might appear as "running," "ran," "runs," or "runner," and the computer sees these as four completely different words. Lemmatization is the act of telling the computer that all those variations are actually just the word "run." Zigmond's work does this for the entire Pāli Canon, a collection containing approximately 1.6 million words. By doing this, he has turned a chaotic jumble of ancient text into a structured, searchable database that computers can finally use to learn and analyze.
The challenge of working with Pāli is that the language is incredibly complex. A single noun can appear in dozens of different forms depending on whether it is the subject of a sentence, the object, or showing possession. Furthermore, the language uses a process called sandhi, where words at the end of one phrase blend seamlessly with the words at the start of the next, often changing their spelling to make the sentence flow smoothly. To a human reader, this is natural; to a computer, it looks like a single, unrecognizable string of letters. Zigmond's project had to solve these puzzles for every single word in the canon. He did not just rely on one version of the text. Instead, he gathered five different historical editions of the scriptures, representing traditions from Sri Lanka, Burma, Thailand, and European scholarship. By comparing these five versions side by side, he could spot differences, correct errors, and create a "critical apparatus"—a detailed record that shows where the ancient manuscripts disagree and which reading is most likely correct.
The result of this massive effort is a dataset where 99.78 percent of every word occurrence has been successfully identified and linked to its dictionary root. This is a staggering level of coverage for a language that has never before had such a resource. The remaining tiny fraction of words that the computer could not solve are mostly rare, poetic terms found in the verse sections of the texts, or complex compounds that the existing dictionaries have not yet fully described. The researcher found that about half of these unsolved words are likely legitimate Pāli words that simply need to be added to the dictionary, while the rest are variations caused by poetic meter or spelling differences across regions. This distinction is crucial because it tells us exactly where the work needs to continue. The project also revealed that by reducing the thousands of different surface forms of a word down to a single root, the total vocabulary size of the entire canon shrinks by more than three times. This reduction makes it possible to use modern statistical methods to find patterns in the text that were previously invisible.
With this new resource, researchers can now perform tasks that were previously impossible. They can group texts together based on their vocabulary to see how different parts of the canon relate to one another. They can train computers to understand the grammar of Pāli, which could eventually lead to machines that can parse sentences and show how words connect to each other, much like a map of a sentence's structure. The work also allows for direct comparisons with Sanskrit, the language from which Pāli evolved, helping scholars trace how Buddhist ideas and words changed as they moved from one culture to another. The entire system, including the code used to build it and the data itself, has been released to the public, inviting other scholars to improve it. While the project does not yet include every possible grammatical tag or a human-verified check of every single word, it provides the essential foundation upon which all future digital work on these ancient texts will be built. It transforms the Pāli Canon from a static collection of old books into a dynamic, living dataset that the modern world can finally explore.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.