💬 NLP

Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

This paper introduces Poller, a novel LLM-based evaluation method that adopts the poet's perspective to significantly reduce errors in assessing Chinese poetry understanding across specialized dimensions, effectively bridging the gap between automated efficiency and human expertise.

Original authors: Shanshan Wang, Derek F. Wong, Jingming Yao, Lidia S. Chao

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shanshan Wang, Derek F. Wong, Jingming Yao, Lidia S. Chao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a box of beautiful, complex, modern Chinese poems. You want to know if a computer (specifically, a Large Language Model or LLM) can understand them well enough to grade someone else's explanation of those poems.

The Problem: The "Polite Robot" vs. The "Real Poet"
The researchers found that standard AI models are terrible at this job. It's like asking a polite, well-read robot to judge a painting. The robot sees the colors and the shapes, but it misses the soul of the piece.

In the paper's experiments, when the AI tried to grade a human's understanding of a poem, it was overly generous. It gave scores of 90 or 95 out of 100, even when the human's explanation was shallow or missed the point. The AI was essentially saying, "Oh, you used big words? That's great! Here's an A+!" It lacked the deep, cultural, and emotional context that a real poet would have.

The Solution: "Poller" (The Role-Playing Trick)
To fix this, the researchers invented a method called Poller (Poetry LLM Evaluator).

Think of it like this: Instead of asking the AI to be a generic "teacher," they ask the AI to pretend to be the actual author of the poem.

Here is how they do it:

The Costume Change: They feed the AI a "character sheet" for the real poet. This includes the poet's biography, their personal struggles, their specific views on art, and even what famous critics have said about their work.
The Perspective Shift: The AI is told, "You are now this poet. You wrote this poem. Now, look at this student's explanation of your work. Does it capture what you were trying to say?"
The Verdict: By stepping into the poet's shoes, the AI stops being a polite robot and starts thinking like a creator. It begins to notice the subtle metaphors, the specific rhythm, and the hidden emotions that a generic AI would miss.

The Results: From "Too Nice" to "Spot On"
The results were dramatic.

Before Poller: The AI was like a friend who never wants to hurt your feelings, giving high scores to almost everything. The gap between the AI's grade and a real human expert's grade was huge.
After Poller: The AI became a strict, insightful critic. When it played the role of the poet, its grading became much closer to what a human expert would give.

For example, when judging how well someone understood the rhetorical techniques (the fancy tricks poets use) or defamiliarization (making the familiar look strange to make you think), the AI's errors dropped by nearly 95%. It went from being wildly inaccurate to being almost as good as a human expert.

The Takeaway
The paper concludes that AI can be a good judge of poetry, but only if you trick it into wearing the poet's hat. By forcing the AI to adopt the specific perspective, background, and soul of the author, we bridge the gap between "fast computer processing" and "deep human expertise."

What the Paper Does NOT Say

It does not claim this method works for all types of writing (like news articles or legal contracts); it is specifically tested on modern Chinese poetry.
It does not say this will replace human poets or critics entirely; it just offers a better way to automatically grade poetry understanding tasks.
It does not claim the AI actually feels emotions; it just simulates the poet's perspective so well that the grading becomes accurate.

In short: To get a computer to understand poetry, you have to make it pretend to be the poet. Once it does, it finally gets the joke.

Technical Summary: Poller – Evaluating Poetry Understanding with Large Language Models

Problem Statement

Traditional automatic evaluation methods (e.g., BLEU, ROUGE) are ill-suited for modern Chinese poetry due to the genre's reliance on figurative language, rhythm, imagery, and cultural nuance. While human evaluation remains the gold standard, it is costly, time-consuming, and impractical for large-scale datasets. Previous attempts to use Large Language Models (LLMs) as evaluators for poetry-related tasks have yielded unsatisfactory results; LLMs often exhibit significant bias, such as favoring their own generated text or focusing on superficial linguistic features while ignoring deep cultural context and authorial intent. Consequently, there is a critical gap between automated efficiency and human expertise in evaluating poetry understanding.

Methodology: The Poller Framework

The authors propose Poller (Poetry LLM Evaluator), a novel method designed to bridge this gap by leveraging LLMs to simulate the perspective of the poem's author. The core hypothesis is that to accurately evaluate a reader's understanding of a poem, the evaluator must adopt the specific persona, background, and intent of the poet.

Core Workflow

Data Collection: The study utilizes 40 high-quality, unpublished modern Chinese poems written by five professional poets.
Generation of Understanding: An LLM (GPT-4-Turbo) generates interpretations of these poems across eight specialized dimensions:
- Content, Language, Imagery, Rhetorical Techniques, Rhythm, Defamiliarization, Thought & Emotion, and Modernity.
Human Benchmark: The five original poets evaluate these interpretations, providing a ground-truth score (0–100) for each dimension.
LLM Evaluation with Role-Playing: Instead of acting as a generic judge, the evaluating LLM is prompted to play the role of the specific poem's author. This is achieved by injecting detailed author information into the prompt, including:
- Basic Information ( $I_i$ ): Occupation, background, and personal history.
- Poetic Views ( $V_i$ ): The author's own experiences and philosophy regarding poetry.
- Critics' Comments ( $C_i$ ): Professional reviews of the author's published works.
Prompt Strategies: The study tests four prompting strategies ( $P_1$ $P_{1}$ to $P_4$ $P_{4}$ ) to refine the role-playing mechanism:
- $P_1$ : The model plays the author and acts as a judge.
- $P_2$ : Emphasizes that the poem was written by the model (as the author) and the understanding comes from a reader.
- $P_3$ : Requires the model to explicitly declare its role as the author before scoring.
- $P_4$ : Combines $P_2$ and $P_3$ .

Key Contributions

First Investigation of LLMs as Poetry Evaluators: This work is the first to systematically investigate the capability of LLMs to evaluate the "understanding of poetry" task, moving beyond generation or translation evaluation.
The Poller Method: The proposal of a role-playing framework where LLMs adopt the author's persona using detailed biographical and critical context to emulate human judgment.
Multi-Dimensional Validation: The study validates the method across multiple mainstream LLMs (GPT-4-Turbo, DeepSeek-R1, Qwen2.5-Turbo, Claude3.5-sonnet) and eight distinct evaluation dimensions.

Experimental Results

The study compares the baseline LLM evaluation (without role-playing, $P_0$ ) against the Poller method ( $P_1$ – $P_4$ ) using the absolute deviation from human scores as the primary metric.

Error Reduction: The Poller method significantly reduces the evaluation error between LLMs and humans.
- GPT-4-Turbo: Achieved a 57.37% reduction in average evaluation error (from 14.38 to 6.13) using the optimal strategy ( $P_3$ ).
- DeepSeek-R1: Achieved a 46.95% reduction in average error (from 12.61 to 6.69).
- Claude3.5-sonnet: Achieved a 41.87% reduction.
- Qwen2.5-Turbo: Achieved a 23.08% reduction.
Dimension-Specific Performance: The method showed exceptional improvements in specific dimensions:
- Rhetorical Techniques: DeepSeek-R1 using $P_3$ reduced error by 94.55% (from 11.37 to 0.62).
- Defamiliarization: GPT-4-Turbo using $P_3$ reduced error by 89.53% (from 10.98 to 1.15).
Baseline Comparison: Without the Poller method, LLMs consistently over-scored human interpretations (e.g., GPT-4-Turbo averaged 93.46 vs. human 79.08), indicating a lack of critical judgment. The Poller method brought LLM scores much closer to human benchmarks.

Significance and Claims

The paper claims that Poller effectively bridges the gap between automated efficiency and human expertise in poetry evaluation. By forcing LLMs to adopt the author's perspective through detailed role-playing, the method mitigates the bias and superficiality inherent in standard LLM evaluations.

The authors assert that:

Current LLMs are not suitable for poetry evaluation in their default state.
However, with the Poller framework, LLMs become viable tools for assisting in the evaluation of poetry understanding tasks.
The approach establishes a foundation for automated evaluation in poetry-related tasks, particularly for modern Chinese poetry, by simulating the "assumed character" of the poet to grasp cultural and emotional subtext.

Limitations: The authors note that the study is limited to modern Chinese poetry (free form) and relies on the availability of specific poet background information. While the core idea of adopting the author's perspective is considered transferable, applying it to other poetic forms or traditional texts where author data is scarce remains an open challenge.