Multidimensional semantic retrieval of classical Chinese poetry for literary heritage discovery
This paper presents a multidimensional semantic retrieval framework for classical Chinese poetry, featuring a large-scale annotated dataset and a Gain-aware Listwise Ordinal Loss (GLO) optimization method that significantly enhances the discovery of poems based on nuanced aspects like emotion, intent, and theme.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
For centuries, the world's greatest poetry has lived in the quiet spaces between words, where a single image of falling flowers might carry a heavy weight of sorrow, or a description of a river might whisper of political change. In the digital age, scholars and lovers of literature have vast libraries of these texts at their fingertips, yet finding a specific poem based on a feeling or a complex idea remains a stubborn challenge. Standard computer searches work well when you know exactly what you are looking for, such as an author's name or a specific title. However, they often stumble when a researcher asks for something more abstract, like a poem that captures the specific melancholy of a spring rain, or one that expresses a particular kind of regret about the passage of time. This is because human language is flexible and layered; two poems can share the same theme but feel entirely different, or use completely different words to describe the same deep emotion. To bridge this gap between human curiosity and digital search, researchers have begun teaching computers to understand the subtle shades of meaning that make literature alive.
A team of researchers from Shanghai Normal University and Huzhou Normal University has taken a significant step forward in this field by creating a new way to search classical Chinese poetry. They recognized that to find the right poem, a computer needs to understand four distinct layers of meaning: the emotion the poet felt, the specific purpose behind writing the poem, the central subject matter, and the deeper philosophical thought it conveys. To teach a computer to navigate these layers, the team first had to build a massive training ground. They started with a collection of nearly one million poems and used advanced language models to analyze each one, breaking them down into those four specific categories. From this analysis, they generated nearly 30,000 unique search questions, or queries, each designed to probe a different aspect of a poem's meaning.
The real work began when they needed to teach the computer which poems actually answered these questions. They gathered a group of human experts, including students and scholars of literature, to act as judges. For each of the thousands of search questions, the computer was asked to pull up a list of candidate poems. The human judges then read the query and the poems, assigning a score from zero to three. A score of zero meant the poem was completely irrelevant, while a score of three meant it was a perfect match that captured the requested meaning with precision. Scores of one and two were used for poems that were partially relevant or broadly related. This process resulted in a gold-standard set of over 8,000 examples where human judgment defined the truth, allowing the researchers to see exactly where the computer was right and where it was wrong.
With this data in hand, the researchers developed a new method to train their computer models, which they call a gain-aware listwise ordinal loss. In simpler terms, this is a set of rules that guides the computer on how to learn from its mistakes. Instead of just telling the computer "this poem is right" or "this poem is wrong," the new method teaches it to understand the hierarchy of relevance. It learns that a poem with a score of three should be ranked significantly higher than one with a score of two, and that the gap between them should be clear and distinct. This approach forces the model to not only find relevant poems but to arrange them in an order that reflects the subtle differences in how well they match the user's request. The researchers tested this method against several other existing techniques and found that their approach consistently produced better results, especially when trying to distinguish between poems that were merely related and those that were deeply relevant.
The results showed that this new training method significantly improved the computer's ability to retrieve the right poems. When the researchers asked the system to find poems that were highly relevant to a specific emotional or philosophical query, the success rate jumped dramatically compared to the untrained models. For instance, when searching for poems that perfectly matched a complex query combining river imagery, historical reflection, and a farewell to a friend, the trained model successfully identified the most appropriate poems and ranked them at the very top of the list. In contrast, the untrained models often found poems that matched only one part of the request, missing the full picture. The study also revealed that the computer struggled most with poems involving deep emotional expression and abstract thought, suggesting that these are the areas where human-like understanding is most difficult to replicate.
This work does not solve the problem of understanding poetry completely, but it provides a powerful new tool for literary discovery. By creating a dataset that respects the nuances of human judgment and a training method that learns from those nuances, the researchers have shown that computers can be taught to retrieve literature based on meaning rather than just keywords. The dataset and the method are now available for other scholars to use, offering a way to explore the vast heritage of classical Chinese poetry through the lens of specific research questions. The findings suggest that while machines may never fully replace the human experience of reading, they can become much more effective partners in the search for meaning, helping us find the exact words we need to understand the past.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.