Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs

Technical Summary: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs

Problem Statement
The rapid advancement of Large Language Models (LLMs) has blurred the distinction between human-written and AI-generated text, posing a significant challenge in the domain of modern Chinese poetry. While previous detection studies have established that LLMs are often ineffective as detectors for general text, and traditional text-based detectors (e.g., RoBERTa, Fast-DetectGPT) struggle with modern Chinese poetry, no prior research has specifically addressed this modality. The unique nature of poetry, which relies heavily on "expressing emotions through scenery" (using visual imagery to convey subjective feelings), suggests that text-only analysis may miss critical semantic cues. Existing benchmarks indicate that traditional detectors achieve low accuracy (often below 70%) on LLM-generated modern Chinese poems, failing to capture the subtle nuances that distinguish human creativity from AI imitation.

Methodology: The IMAGINE Framework
To address these limitations, the authors propose IMAGINE (IMAge-semaNtic GuIded poEtry detection), a novel framework that leverages Multi-modal Large Language Models (MLLMs) to integrate image-semantic guidance into the detection process. The methodology is grounded in the poetic principle that poems are often inspired by visual scenes; thus, the presence of a corresponding image can provide deeper semantic context than text alone.

The framework operates through a two-part, example-driven approach:

1. Example-Driven Learning: The model is first provided with a pair of examples: one human-written poem and one AI-generated poem. Crucially, these poems share the same title, theme, meaning, imagery, and emotional tone, and the AI poem retains all nouns from the human poem but differs in writing style. A corresponding image reflecting the content of these poems is also provided. The model is informed of the relationships between the image and the poems, as well as the specific sources (human vs. AI).
2. Guided Classification: The model is then presented with a new, unknown poem and a corresponding image generated based on that poem's content. Using the context established in the examples, the model analyzes the interplay between the text and the image to determine the source of the poem.

The prompt design (specifically IP3) instructs the MLLM to act as a professional poetry expert, explicitly noting that the image was generated to reflect the poem's title, theme, imagery, and emotion, and that it embodies the noun elements of the human poem. The model is tasked with judging the "Poem Text" specifically, using the image as a semantic guide.

Dataset Construction
The study utilizes a custom dataset comprising:

• 800 Human-written poems: Sourced from the AIGenPoetry dataset, authored by professional poets.
• 3,200 LLM-generated poems: Created by four distinct models (GPT-4.1, GLM-4, DeepSeek-V3, and DeepSeek-R1) using a prompt that forces the AI to mimic the human poem's core elements (title, theme, imagery, emotion) while altering the style.
• 800 Corresponding Images: Generated using Qwen-Image (selected as the best performer among six models via poet evaluation) to visually represent the content of the human poems.

Key Contributions

1. Dataset Creation: The construction of a semantically grounded dataset for detecting AI-generated modern Chinese poetry, requiring LLMs to retain comprehensive features of human poetry (theme, meaning, imagery, emotion) to ensure a rigorous evaluation setting.
2. IMAGINE Framework: The proposal of a novel image-semantic guided detection method that outperforms text-only baselines. This approach effectively integrates information such as meaning, imagery, and feeling from the image to form a complementary judgment with the text.
3. Performance Benchmarking: The establishment of initial performance baselines for LLMs as poetry detectors and the demonstration that example-driven, multimodal approaches can significantly enhance these capabilities.

Experimental Results
The authors evaluated various detectors, including traditional classifiers (RoBERTa, Binoculars, etc.) and LLM-based detectors (Gemini, Qwen, GPT-5) under different prompting strategies (vanilla, text-only enhanced, and image-semantic guided).

• Baseline Limitations: Traditional text-only detectors, including the best-performing RoBERTa, struggled with the task. RoBERTa achieved a Macro-F1 of roughly 73–84% depending on the generator, while other detectors often fell below 50%.
• Text-Only LLMs: Plain-text LLM detectors showed mixed results, with Gemini performing best among text-only models but still lagging behind the proposed method.
• IMAGINE Performance: The image-semantic guided method (IP3) significantly improved detection accuracy.
  • The Gemini detector using the IMAGINE framework achieved a Macro-F1 score of 85.65% across all test cases (human and AI-generated poems).
  • This result surpasses the best-performing traditional detector (RoBERTa) and represents a substantial improvement over the text-only baseline (TP1, 70.28%) and the text-only enhanced baseline (TP2, 79.50%).
  • The method was particularly effective in narrowing the performance gap between different LLM detectors and improving detection for poems generated by DeepSeek-R1, where the Macro-F1 jumped from 54.90% (baseline) to 94.75%.

Significance and Claims
The paper claims that the integration of image semantics is a critical factor in detecting AI-generated poetry. By leveraging the visual representation of the poem's content, the model can better grasp implicit meanings and emotional depth that are often lost in text-only analysis. The authors assert that their method reaches the state-of-the-art level for this specific task, proving that multimodal information is essential for distinguishing between human and AI creativity in the poetic domain.

The study also highlights that not all LLMs are equally suitable for detection tasks; performance varies significantly by model architecture and training data. However, the consistent improvements observed across different LLMs when using the IMAGINE framework validate the robustness of the image-semantic approach.

Limitations
The authors acknowledge that their method is specifically inspired by the human practice of writing poetry based on visual scenes. Consequently, they note that the approach may not be fully applicable to other text genres, such as news articles or academic papers, which lack the same strong reliance on visual imagery and "expressing emotions through scenery."