Can ChatGPT Really Understand Modern Chinese Poetry?

1. Problem Statement

While Large Language Models (LLMs) like ChatGPT have demonstrated proficiency in poetry generation and translation, their ability to truly understand poetry remains unexplored. Previous research has focused solely on evaluating experimental outputs (e.g., generated poems) without addressing the fundamental cognitive issue of comprehension. Poetry, particularly modern poetry, is inherently subjective and difficult to interpret. The core research question is: Does ChatGPT possess a genuine understanding of modern Chinese poetry, or does it merely mimic surface-level patterns?

2. Methodology: The ECUMP Framework

The authors propose a comprehensive framework called ECUMP (Evaluation of ChatGPT's Understanding of Modern Poetry). The methodology consists of four key stages:

A. Dimension Identification

Collaborating with eight professional modern poets, the authors identified five critical dimensions for evaluating poetry comprehension:

1. Content: The subject matter and narrative of the poem.
2. Expression Methods: Technical execution, subdivided into:
   • Language: Innovation and lexical choices.
   • Imagery: Visual and sensory images.
   • Rhetorical Techniques: Metaphors, similes, symbolism.
   • Rhythm: Pauses, line breaks, and stanzas.
   • Defamiliarization: Challenging conventional perceptions.
3. Thought & Emotion: The underlying ideas and feelings conveyed.
4. Modernity: The presence of contemporary consciousness or sensibility.
5. Poeticity: A subjective measure of the poem's essence; specifically, identifying the single most poetic sentence in the work.

B. Dataset Construction

• Source: 48 modern Chinese poems written by six professional poets.
• Categories:
  • Com-Poetry (40 poems): Common types of modern poetry.
  • Spe-Poetry (8 poems): Special types incorporating ancient Chinese literature, classical imagery, and historical figures, yet maintaining modern forms.
• Total: 147 stanzas, 750 lines.

C. Prompt Engineering

The authors designed and optimized a specific prompt (Table 1) to guide GPT-4 (gpt-4-0125) to interpret poems across the five dimensions.

• Optimization: An initial issue where GPT-4 claimed the poem text was incomplete was resolved by explicitly stating, "This is a complete modern poem," in the prompt.
• Task: GPT-4 was asked to analyze the poem based on the defined dimensions and provide specific examples and explanations.

D. Evaluation Strategy

• Primary Evaluation (Human): The original poets evaluated GPT-4's interpretations of their own work.
  • Scoring (0–100): 0 = Complete inconsistency; 100 = Perfect alignment with author intent.
  • Poeticity Scoring: 0 (No poeticity), 50 (Poetic but not the best), 100 (Correctly identified as the most poetic sentence).
• Secondary Evaluation (LLM): Other LLMs (o1-preview, GPT-4o, Qwen2.5-Plus/Max) were used as evaluators to compare against human scores.

3. Key Results

A. Overall Performance

• Alignment: ChatGPT's interpretations aligned with the original poets' intents in over 73% of cases across most dimensions.
• Strongest Areas:
  • Imagery (Com-Poetry): 81.18/100. GPT-4 excels at identifying explicit visual and sensory elements.
  • Content: 80.33/100. Strong performance on poems with clear themes or narrative structures.
  • Rhetorical Techniques (Spe-Poetry): 88.75/100. High accuracy when explicit cues (e.g., "like," "as") are present.
• Weakest Areas:
  • Poeticity: The most significant failure point. GPT-4 failed to identify the "most poetic sentence" correctly in more than 50% of cases. In some instances, it selected sentences entirely devoid of poetic essence.
  • Language (Spe-Poetry): Lowest score (73.75/100). GPT-4 struggles with the nuances of classical imagery and cultural context embedded in special modern poetry.
  • Rhythm: Generally low scores (76.15 for Com-Poetry), indicating difficulty in analyzing subtle rhythmic patterns beyond simple line breaks.

B. Comparative Analysis (Com-Poetry vs. Spe-Poetry)

• Com-Poetry: GPT-4 performs best on imagery and content. It struggles with implicit rhetorical devices lacking explicit cues.
• Spe-Poetry: GPT-4 surprisingly performs well on Modernity (82.50) and Rhythm (82.50), successfully bridging ancient themes with modern sensibilities. However, its grasp of Language drops significantly due to the complexity of classical lexicon.

C. LLM vs. Human Evaluation

• LLMs used as evaluators deviated significantly from the professional poets' assessments.
• Even advanced models like o1-preview and GPT-4o failed to replicate human nuance in evaluating poetic understanding.
• Conclusion: Human evaluation remains indispensable for poetry tasks; current LLMs are not reliable automated evaluators for this domain.

4. Key Contributions

1. First Systematic Evaluation: The first study to systematically evaluate an LLM's understanding (not just generation) of modern poetry.
2. Comprehensive Framework (ECUMP): Introduced a multi-dimensional evaluation framework specifically tailored for modern Chinese poetry.
3. Expert-Driven Methodology: Established a rigorous evaluation protocol involving the original poets as the ground truth, ensuring high reliability and nuance.
4. Empirical Insights: Provided detailed analysis of where LLMs succeed (explicit imagery, clear narratives) and fail (poetic essence, cultural nuance, implicit rhetoric).

5. Significance and Implications

• Foundation for Future Research: This study establishes a baseline for understanding the limitations of LLMs in creative and cultural domains.
• Task Dependency: The results suggest that while LLMs can assist in summarizing content or identifying obvious metaphors, they lack the deep cultural and contextual understanding required to grasp the "soul" of a poem (poeticity).
• Human-in-the-Loop Necessity: The findings underscore that for high-stakes cultural tasks like poetry analysis, human expertise cannot be replaced by automated evaluation or generation.
• Prompt Engineering: Demonstrates that specific prompt design (e.g., clarifying text completeness) is crucial for eliciting better performance from LLMs in complex literary tasks.

In summary, while ChatGPT shows a 73% alignment with poets on general comprehension, it fundamentally struggles with the essence of poetry (poeticity) and cultural nuance, highlighting that current LLMs are tools for assistance rather than true interpreters of artistic intent.