Evaluating the Creativity of LLMs in Persian Literary Text Generation

1. Problem Statement

While Large Language Models (LLMs) have demonstrated creative capabilities in English, their ability to generate culturally nuanced and stylistically rich literary text in non-English languages, particularly Persian, remains underexplored. Existing research faces three primary gaps:

• Linguistic Bias: Most creativity benchmarks (e.g., Alternative Uses Task) focus on English, ignoring the specific linguistic and cultural depth required for Persian literature.
• Evaluation Limitations: Standard metrics often fail to capture the subjective nature of creativity, cultural alignment, and the specific rhetorical devices (e.g., metaphor, antithesis) central to Persian literary tradition.
• Resource Scarcity: There is a lack of standardized datasets and evaluation frameworks for assessing creativity in low-to-mid-resource languages like Persian.

2. Methodology

The authors propose a comprehensive framework involving dataset creation, metric adaptation, and automated evaluation.

A. Dataset Construction: CPers

• Name: CPers (Creativity in Persian).
• Scale: 4,371 single-sentence literary texts.
• Content: Spanning 20 diverse emotional and cultural topics (e.g., Love, Longing, Nowruz/Persian New Year, Father).
• Composition: Authored by native Persian speakers (non-professionals and inspired by classical figures), averaging ~26 words per sentence.
• Annotation: A subset of 200 texts (100 human-written, 100 model-generated) was manually annotated for creativity scores and rhetorical devices.

B. Evaluation Framework (Adapted TTCT)

The authors adapted the Torrance Tests of Creative Thinking (TTCT) to the Persian context, defining four core dimensions:

1. Originality: Uniqueness, distance from clichés, and use of literary devices.
2. Fluency: Grammatical correctness, naturalness to native speakers, and contextual appropriateness (redefined from "idea count" to "linguistic flow" due to the implicit nature of Persian creativity).
3. Flexibility: Ability to express topics from new perspectives or using varied styles (ironic, philosophical, etc.).
4. Elaboration: Detail, vocabulary variety, and ability to evoke mental imagery or specific emotions.

• Scoring: A 5-point scale based on 12 culturally tailored questions (3 per dimension).

C. Automated Evaluation (LLM-as-a-Judge)

• Judge Selection: The authors compared GPT-4o and Claude 3.7 Sonnet against human annotators using the Intraclass Correlation Coefficient (ICC).
• Result: Claude 3.7 Sonnet demonstrated significantly higher alignment with human judgments (high ICC across all dimensions) and less bias toward model-generated text compared to GPT-4o. It was selected as the primary automated judge.

D. Experimental Setup

• Models Evaluated: Six state-of-the-art LLMs: GPT-3.5, GPT-4.1, DeepSeek-V3, DeepSeek-R1, Qwen2.5, and Gemma.
• Prompting: Zero-shot prompting in Persian ("Write a literary text in one sentence about {Topic}").
• Analysis: Beyond creativity scores, the study analyzed word frequency (lexical patterns) and the usage of four literary devices: Simile, Metaphor, Antithesis, and Hyperbole.

3. Key Results

A. Model Performance

• Top Performer: DeepSeek-R1 achieved the highest overall creativity score (4.44/5), leading specifically in Flexibility and Elaboration.
  • Reasoning: The authors attribute this to DeepSeek-R1's "reasoning-oriented" training (Chain-of-Thought), which encourages structured, detailed, and diverse content generation.
• Fluency Leader: GPT-4.1 scored highest in Fluency (4.99), producing grammatically perfect and natural Persian, but scored lower in Originality due to reliance on conventional expressions.
• Originality vs. Fluency Trade-off: Qwen2.5 showed high Originality but poor Fluency, often generating novel but unclear or unnatural sentences.
• Consistency: DeepSeek models showed low standard deviations, indicating stable performance across runs.

B. Lexical and Stylistic Analysis

• Repetitive Patterns: Most models (especially GPT-3.5/4.1 and Gemma) relied heavily on binary oppositions (Light/Dark, Hope/Despair) and common metaphors (Heart, Sky).
• Human vs. Model: Human texts displayed more varied imagery (e.g., "Gaze," "Sky") and intuitive emotional nuance.
• Reasoning Impact: Reasoning-enhanced models (DeepSeek-R1, QwQ-32B) reduced the frequency of repetitive words (like "dark" or "hope") and increased the use of vivid, imaginative terms (e.g., "Sunrise," "Sky").

C. Literary Device Usage

• Human Balance: Human writers used a balanced mix of devices, often integrating multiple types (metaphor + simile) in a single sentence.
• Model Bias: Models tended to overuse Similes and Metaphors while underutilizing Antithesis and Hyperbole.
• Judging Limitations: LLMs (even Claude) struggled to accurately distinguish between similes and metaphors when acting as judges for rhetorical devices, highlighting a gap in nuanced literary understanding.

4. Key Contributions

1. First Systematic Evaluation: The first study to systematically evaluate LLM creativity in Persian literary text generation.
2. CPers Dataset: Introduction of the first large-scale, culturally grounded dataset (4,371 texts) for Persian creativity, including a human-annotated benchmark subset.
3. Adapted Framework: A culturally adapted TTCT framework that redefines "fluency" for implicit, metaphorical languages and validates an LLM-as-a-Judge approach (Claude 3.7) for low-resource languages.
4. Insights on Reasoning: Evidence that reasoning-oriented models (DeepSeek-R1) produce more elaborated and flexible literary content, suggesting a link between reasoning capabilities and creative generation.
5. Device Analysis: A detailed analysis showing that the balance and nuanced deployment of rhetorical devices are more critical for perceived creativity than the sheer quantity of devices used.

5. Significance and Implications

• Cultural NLP: The work underscores the necessity of culturally specific evaluation metrics. Direct translation of English benchmarks fails to capture the depth of Persian literary expression.
• Model Design: The results suggest that training strategies emphasizing reasoning (e.g., "think before answering") can positively impact creative text generation, not just logical problem-solving.
• Future Directions: The paper highlights the need for multi-dimensional evaluation (not just fluency) and suggests that future models must be trained to better understand and deploy a wider variety of rhetorical devices to match human literary nuance.
• Resource Gap: By releasing CPers, the authors provide a critical foundation for future research in low-resource language creativity, moving beyond English-centric benchmarks.

Conclusion

The paper concludes that while LLMs are effective tools for generating Persian literary text, they currently lack the diversity, cultural nuance, and balanced rhetorical deployment of human writers. DeepSeek-R1 emerges as the strongest contender due to its reasoning capabilities, but no single model excels across all creativity dimensions. The study advocates for continued refinement of evaluation frameworks that prioritize cultural alignment and stylistic richness over mere grammatical correctness.