Machine translation of modern Chinese poetry into Japanese: A comparative evaluation of LLMs, neural machine translation systems, and prompt strategies

Language is a vast, shifting landscape where meaning is carried not just by words, but by rhythm, silence, and the specific weight of a phrase. For decades, computers have learned to translate this landscape, moving from rigid, rule-based systems to neural networks that mimic the human brain's ability to find patterns. These modern systems, known as neural machine translation, have become remarkably good at turning news articles or technical manuals from one language into another, capturing the facts with high precision. More recently, a new generation of artificial intelligence, called large language models, has entered the field. Unlike their predecessors, which were trained specifically to translate, these models are general-purpose thinkers that can follow complex instructions and understand context in a way that feels more like conversation. They promise to handle the messy, creative parts of language that machines usually struggle with. But poetry remains the ultimate test. A poem is not just a message to be decoded; it is an experience built on imagery, cultural nuance, and a delicate balance between what is said and what is left unsaid. Translating a poem requires a computer to do more than swap words; it must reconstruct a feeling. This raises a difficult question: can these new, conversational machines actually translate poetry better than the established tools we use every day, and does the way we ask them to work change the result?

To find out, researchers set out to compare the performance of these different systems on modern Chinese poetry, a genre known for its condensed imagery and emotional depth. They chose a specific collection of twelve poems by the contemporary poet Yu Xiuhua, ensuring the texts were fresh and had not been previously translated by machines or humans. The team pitted two of the most popular online translation services, which rely on standard neural networks, against two of the leading large language models. To see if the instructions given to the machines mattered, they tested the large language models under two different conditions. In the first scenario, they gave the models a simple, direct command to translate the text. In the second, they asked the models to first explain the meaning of the poem in detail before attempting the translation, a strategy designed to force the machine to understand the context before generating words. The resulting translations were then evaluated by a panel of eight experts in Chinese and Japanese literature. These experts rated the poems not just on whether the meaning was correct, but on how well the translation flowed, whether it sounded like a real poem, and how much it felt like it had been written by a machine.

The results revealed a clear advantage for the newer, instruction-following models. When given the simple command to translate, the large language models produced versions that were significantly more faithful to the original meaning, more fluent in Japanese, and much closer to the original poetic style than the standard online translators. The standard tools often produced translations that were grammatically correct but felt flat, losing the rhythm and the specific cultural weight of the words. The large language models, however, seemed to grasp the poem as a whole unit, preserving the tension and imagery that make poetry resonate. For instance, where a standard translator might turn a ritualistic phrase into a literal description of speed, the large language models managed to keep the sense of a solemn command, capturing the spiritual tone of the original. They also showed far fewer signs of being generated by a computer, avoiding the rigid sentence structures that often betray machine translation.

However, the story becomes more complex when the researchers changed the instructions. The idea that giving a machine more context—asking it to explain the poem first—would always lead to a better translation proved to be false. The effect of this "explanation-first" strategy depended entirely on which specific model was being used. For one of the large language models, the extra step of explaining the poem actually made the translation worse. The resulting poems became too explicit, losing their mystery and poetic compression, as if the machine had over-analyzed the text and stripped away its soul. For the other model, the same strategy was a success; the explanation step helped it organize its thoughts, leading to translations that were more coherent and stylistically stronger. This suggests that there is no single "best" way to prompt these machines. What works for one system can hinder another, depending on how that specific system processes information.

Perhaps the most surprising finding was what happened when the researchers compared the two large language models after they had both been given the detailed, explanation-first instructions. Under these specific conditions, the performance gap between the two models disappeared. While they might have started with different strengths and weaknesses, the shared instruction guided them toward a similar level of quality. They produced translations that were statistically indistinguishable from one another in terms of accuracy, flow, and style. This suggests that the way we ask a machine to work can be just as important as the machine itself. In the realm of poetry translation, the interaction between the model's internal design and the human's prompt creates the final result. The study concludes that we cannot simply pick the most powerful artificial intelligence and expect it to handle literature perfectly. Instead, translating poetry with machines is a delicate negotiation between the tool and the instructions, where the goal is to find the specific combination that allows the machine to step back and let the poem breathe.