💻 computer science

Evaluating GPT -5 Literary Metaphor Translation from English into Arabic through Recognition, Interpretation, and Rendering

This study evaluates GPT-5's translation of literary metaphors from English to Arabic using a stage-aware framework, revealing that while the model excels at recognizing and interpreting metaphors, it significantly struggles with rendering them compared to both DeepL and human translation, thereby exposing a critical gap between comprehension and creative production in AI.

Original authors: Hani Hamad M Albelihi, Arwa Alsaleh

Published 2026-11-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hani Hamad M Albelihi, Arwa Alsaleh

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Language is more than a code for exchanging facts; it is a landscape of images, where a writer might describe a heavy heart as a stone or a fading memory as a mist. When a translator moves a story from one language to another, they must carry not just the facts of the plot, but these delicate pictures. This is especially true for metaphors, the figurative language that gives literature its emotional depth. For decades, human translators have been the only ones trusted to navigate this terrain, but the rise of artificial intelligence has changed the conversation. Large language models, the powerful computer systems behind modern chatbots, can now produce fluent translations that read smoothly and make grammatical sense. Yet, a crucial question remains: when these machines encounter a metaphor, do they truly understand the picture behind the words, or do they simply rearrange the furniture while leaving the image behind?

Researchers at Qassim University set out to answer this by looking inside the black box of artificial intelligence. They focused on a specific type of machine, a model known as GPT-5, and tested its ability to handle the literary metaphors found in Kazuo Ishiguro's novel The Buried Giant. Instead of just asking the machine to translate a sentence and judging the final result, the researchers broke the process down into three distinct steps. First, they asked the machine to simply identify which words were metaphors. Second, they asked it to explain what those metaphors meant within the story. Finally, they asked it to translate the passage into Arabic. By separating these tasks, the team could see exactly where the machine succeeded and where it stumbled, rather than just guessing why a final translation might have failed.

The study examined 202 metaphorical expressions from the first two chapters of the novel. The results revealed a surprising gap between understanding and doing. The artificial intelligence was remarkably good at the first two steps. It correctly identified the metaphors in the text and provided accurate explanations of their meaning, often matching the performance of a human expert. It knew that a "welcome antidote to the noon sun" was a metaphor for relief from heat, and it could explain that connection clearly. However, when the machine was asked to actually translate the passage into Arabic, its performance dropped significantly. While it understood the concept, it struggled to recreate the image in the new language. In more than a third of the cases, the machine either removed the metaphor entirely, turning a vivid image into a plain statement, or it distorted the meaning so that the original picture was lost.

To see how this compared to other methods, the researchers pitted the machine against two other translators: a specialized translation software called DeepL and a published Arabic translation by a professional human translator named Kholoud Amro. The professional human translation was the clear winner, preserving the imagery and emotional weight of the original text far better than either machine. The specialized software, DeepL, performed better than the general chatbot, but it still fell short of the human standard. The study found that while the chatbot could produce fluent sentences, it often failed to maintain the consistency of images across the story. For instance, if a character was described as a "boatman" in one sentence, the machine might call him a "vessel" in the next, breaking the flow of the narrative. The human translator, by contrast, maintained a consistent and vivid image throughout.

The researchers identified four main reasons why the machines struggled. First, they often could not find the right equivalent word in Arabic that carried the same cultural weight. Second, they failed to keep the images consistent as the story progressed. Third, they sometimes misunderstood the context, leading to a translation that made sense on its own but was wrong for the story. Finally, and perhaps most importantly, the machines lacked the ability to creatively rebuild a metaphor. When a direct translation didn't work, a human translator might change the image slightly to fit the new language while keeping the feeling the same. The machines, however, tended to either stick rigidly to the original words or drop the image altogether.

This study suggests that the current way we evaluate artificial intelligence in translation might be too generous. If we only look at whether a translation is fluent or grammatically correct, we might miss the fact that the soul of the text—the metaphors and the imagery—has been lost. The machine can tell you what a metaphor means, but it cannot yet recreate the experience of that metaphor in a new language with the same skill as a human. While these tools are powerful and improving, they still lack the deep, intuitive grasp of culture and imagery that allows a human translator to make a story feel alive in a different tongue. The gap between recognizing a metaphor and successfully painting it in a new language remains a significant hurdle for artificial intelligence to clear.

Technical Summary: Evaluating GPT-5 Literary Metaphor Translation from English into Arabic

Problem Statement
Current evaluation methods for AI-assisted literary translation often rely on fluency or general semantic accuracy, which may fail to detect the loss of figurative meaning, imagery, and rhetorical force in metaphor translation. Existing frameworks typically assess only the completed target text, making it difficult to determine whether a translation failure stems from a failure to recognize the metaphor, an inadequate interpretation of its conceptual meaning, or an inability to render it effectively in the target language. Furthermore, there is a paucity of empirical research comparing Large Language Models (LLMs), dedicated Neural Machine Translation (NMT) systems, and professional human translations specifically within the English–Arabic language pair, which presents significant structural and rhetorical challenges.

Methodology
The study employed a mixed-methods design to evaluate 202 literary metaphors extracted from the first two chapters of Kazuo Ishiguro's The Buried Giant. The research utilized a "stage-aware" framework, separating the translation process into three analytically distinct dimensions:

Recognition: Identifying an expression as metaphorical.
Interpretation: Construing the intended conceptual and emotional meaning.
Rendering: Producing the final Arabic translation.

Three translation sources were compared:

GPT-5: Used via the OpenAI web interface to generate recognition, interpretation, and rendering outputs based on a standardized prompt.
DeepL: Used for English-to-Arabic rendering.
Professional Translation: Kholoud Amro's published Arabic translation of the novel served as the human benchmark.

Metaphors were identified using the Metaphor Identification Procedure (MIP) and MIPVU. Evaluation was conducted on a 0–3 ordinal scale (0: Failure, 1: Weak, 2: Adequate, 3: Excellent) by two independent raters, achieving high inter-rater reliability (weighted Cohen's κ = .89–.93). Quantitative analysis included descriptive statistics and non-parametric inferential tests (Friedman and Wilcoxon signed-rank tests), while qualitative analysis focused on recurring linguistic and translational patterns.

Key Results

The Recognition–Rendering Gap: GPT-5 demonstrated strong performance in metaphor recognition (Mean = 2.30) and interpretation (Mean = 2.29), with over 76% of cases rated adequate or excellent. However, its rendering performance was substantially weaker (Mean = 1.37), with only 50% rated adequate or excellent and 37.6% receiving a score of zero. Statistical analysis confirmed a significant difference between the high recognition/interpretation scores and the low rendering scores ( $p < .001$ ).
Comparative Performance: In terms of rendering quality, the published professional translation significantly outperformed both AI systems (Mean = 2.55). DeepL achieved moderately higher rendering scores (Mean = 1.68) than GPT-5, though both AI systems lagged significantly behind the human translation.
Failure Modes: GPT-5 exhibited high rates of omission and de-metaphorization (22.8%) and mistranslation (14.9%). DeepL showed lower rates of these failures but struggled with contextual disambiguation (e.g., confusing "boatman" with the vessel). The professional translation had the lowest failure rate (10.9%) but was not error-free, occasionally resorting to literalism that weakened symbolic meaning.
Qualitative Factors: Four factors were identified as critical to successful rendering:
1. Lexical Equivalence: The availability of stable, context-appropriate Arabic equivalents.
2. Discourse-Level Consistency: Maintaining lexical continuity for recurring metaphors across the narrative.
3. Contextual Disambiguation: Correctly identifying referents within the narrative context.
4. Rhetorical Recreation: The ability to reformulate metaphors (e.g., into similes) to preserve visual force and rhetorical impact in the target language.

Significance and Claims
The study claims that conventional fluency-based or comprehension-based evaluations may overestimate AI performance in literary metaphor translation because they fail to distinguish between understanding a metaphor and successfully rendering it figuratively. By introducing a stage-aware framework, the authors demonstrate that a model can successfully recognize and interpret a metaphor yet fail to preserve its imagery and rhetorical function in the target text.

The research highlights a substantial "recognition–rendering gap," suggesting that the ability to explain a metaphor does not guarantee the ability to translate it effectively. The findings indicate that successful literary metaphor translation requires more than propositional accuracy; it demands attention to imagery, lexical continuity, and rhetorical adaptation. While professional human translation currently outperforms AI in these specific tasks, the study provides a replicable methodology for diagnosing specific limitations in AI figurative language processing, particularly in linguistically distant pairs like English–Arabic. The authors explicitly note that their findings are limited to a specific corpus and translation direction and do not claim universal human superiority, but rather point to specific areas where current AI systems struggle compared to professional human practice.

More like this