GhazalBench: Usage-Grounded Evaluation of LLMs on Persian Ghazals

The paper introduces GhazalBench, a benchmark demonstrating that while large language models can generally understand the meaning of Persian ghazals, they struggle with exact verse recall in completion tasks due to limited training exposure, a gap that narrows in recognition settings and is less pronounced for English sonnets.

The Gist

Imagine you are at a party in Tehran, and someone quotes the first half of a famous poem by Hafez, a legendary Persian poet. The crowd immediately knows the second half and finishes the sentence together. It's a shared cultural memory, like everyone knowing the chorus to a massive hit song.

Now, imagine you ask a super-smart AI (a Large Language Model) to do the same thing. The paper "GhazalBench" asks a simple but profound question: Does this AI actually "know" the poem, or is it just guessing based on the vibe?

Here is the breakdown of what the researchers found, using some everyday analogies.

1. The Setup: The "Hafez Challenge"

The researchers created a test called GhazalBench. They took 50 famous poems (Ghazals) by Hafez and asked different AI models to do two things:

• The Translator: "Explain this poem in simple, plain Persian prose."
• The Completer: "I'll say the first line; you say the exact second line."

They also tested the AI with different "hints," like giving it a summary of the meaning, or just a few keywords from the second line.

2. The Big Discovery: The "Meaning vs. Memory" Gap

The results revealed a funny split personality in the AI.

The Good News (The Translator):
When asked to explain the meaning of the poem, the AI was excellent. It could tell you, "This poem is about a lover who is sad because their beloved left."

• Analogy: Imagine you are at a concert. The AI is like a music critic who can perfectly describe the feeling of the song, the lyrics' emotion, and the story behind it. It understands the "soul" of the music.

The Bad News (The Completer):
When asked to recite the exact words of the second line, the AI often failed. It would get the meaning right but mess up the specific words, rhyme, or rhythm.

• Analogy: Now, imagine asking that same music critic to sing the song note-for-note from memory. They might hum the melody and get the emotion right, but they might forget the specific lyrics or mix up the verses. They "get" the song, but they don't have the sheet music burned into their brain.

3. The "Recognition" Trick

The researchers found something interesting: If they gave the AI a multiple-choice quiz (e.g., "Which of these three lines is the correct second line?"), the AI got much better at it.

• Analogy: This is like the difference between Free Recall and Recognition.
  • Free Recall: "What is the capital of France?" (Harder).
  • Recognition: "Is the capital of France Paris, London, or Berlin?" (Easier).
    The AI is great at recognizing the right answer when it sees it, but terrible at pulling it out of thin air.

4. The Language Barrier (Persian vs. English)

The researchers also tested the AI on Shakespeare's sonnets in English.

• The Result: The AI was much better at reciting Shakespeare than Hafez.
• The Reason: It's not that the AI is "bad" at poetry. It's that the AI was trained on way more English books and poems than Persian ones.
• Analogy: Think of the AI's brain as a library. The English section is a massive, well-organized library with millions of books. The Persian section is a small, dusty room with only a few books. If you ask the librarian (the AI) to find a specific book, they can do it easily in the English section but might struggle in the Persian section, even if they understand the story perfectly.

5. Why Does This Matter?

This paper is important because it shows that understanding a culture is different from memorizing its texts.

• The Problem: If we only test AI on whether it can "understand" a poem (by asking it to explain it), we might think it's culturally fluent. But if you ask it to participate in a real cultural moment (like finishing a quote at a party), it might fail.
• The Lesson: To truly respect and interact with a culture, an AI needs to be able to recall the exact words people use, not just the general ideas.

Summary

GhazalBench is like a "cultural fluency test" for AI. It found that while AI models are great at understanding the story of Persian poetry, they are often terrible at remembering the exact lyrics. They are like a fan who loves a band and knows all the songs' meanings but can't sing the lyrics without looking at the screen.

The researchers hope this test helps build better AI that doesn't just "talk about" culture, but can actually "speak" it.

Technical Summary

Here is a detailed technical summary of the paper "GhazalBench: Usage-Grounded Evaluation of LLMs on Persian Ghazals."

1. Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in general linguistic fluency and semantic understanding, their ability to interact with culturally canonical texts remains under-evaluated. Specifically, in Persian culture, poetry (particularly the ghazals of Hafez) is not just a literary artifact but a living part of daily communication, used for quoting, paraphrasing, and completing verses based on partial cues.

Existing evaluations of LLMs on poetry focus primarily on creative generation (stylistic imitation) or aesthetic quality, failing to capture how poetry functions as a shared cultural resource. Furthermore, prior work on memorization often frames verbatim recall as a privacy risk rather than a functional capability. There is a lack of benchmarks that assess whether models can access the exact surface form of culturally significant texts under realistic usage conditions (e.g., completing a verse from a first line or recognizing it from a paraphrase).

2. Methodology: GhazalBench

The authors introduce GhazalBench, a benchmark designed to evaluate LLM interactions with Persian ghazals through "usage-grounded" diagnostic scenarios. The benchmark construction and evaluation pipeline involves:

• Dataset Construction:

  • Extracted the first 100 ghazals from the Divan of Hafez.
  • Used Gemini 2.5 Flash to retrieve couplets and their corresponding prose explanations.
  • Manually verified all data against the original printed edition and obtained poetic meter data from the Ganjoor repository.
  • The final experimental set consists of 50 ghazals (438 couplets).

• Evaluation Dimensions:
  The benchmark assesses two complementary abilities:

  1. Poem-to-Prose Understanding: Models must paraphrase a specific couplet into neutral, standard Persian prose.
     • Evaluation: Performed via LLM-as-Judge (Gemini 3 Pro) scoring six dimensions (Semantic Fidelity, Naturalness, Clarity, Interpretation Avoidance, Neutral Tone, Length Control) and automatic metrics (BERTScore, chrF++, BLEU-4).
  2. Canonical Verse Retrieval: Models must retrieve the second verse of a couplet given the first verse. This is tested under varying cue conditions to distinguish between recall (free generation) and recognition (selection from options):
     • Canonical Continuation: First verse only (Completion vs. Multiple Choice).
     • Prose-mediated: First verse + prose explanation.
     • Prose-Paraphrase Robustness: First verse + paraphrased prose (to test semantic vs. lexical overlap).
     • Fragment-Conditioned: First verse + two salient words from the second verse.
     • Shuffled-Form-Conditioned: First verse + word-shuffled second verse (to test sensitivity to word order).

• Models Evaluated:

  • Proprietary: Claude Sonnet 4.5.
  • Open-Weight: DeepSeek V3.2, Gemma 3 27B Instruct, Gemma 3 12B Instruct.
  • Cross-Lingual Control: Parallel experiments conducted on Shakespeare's Sonnets (English) to determine if limitations are architectural or data-dependent.

3. Key Contributions

1. GhazalBench Benchmark: The first usage-grounded benchmark for evaluating LLMs on Persian poetry, moving beyond generation to interaction patterns (paraphrasing, recalling, recognizing).
2. Recall vs. Recognition Framework: A diagnostic approach grounded in cognitive psychology (Tulving et al.) that disentangles semantic understanding from access to canonical surface form.
3. Systematic Dissociation: Empirical evidence showing a clear gap where models excel at understanding meaning but fail at exact form-level recall.
4. Training Signal Density Hypothesis: Cross-lingual evidence suggesting that poor performance on Persian is due to training data scarcity rather than inherent architectural limitations, as models perform significantly better on English sonnets.

4. Key Results

A. Poem-to-Prose Understanding

• All models demonstrated strong semantic understanding, achieving high scores in paraphrasing tasks.
• Claude Sonnet 4.5 consistently outperformed open-weight models across all dimensions, particularly in "Semantic Fidelity" and "Interpretation Avoidance."
• Automatic metrics (BERTScore) showed high semantic alignment, while lexical metrics (BLEU, chrF++) highlighted differences in stylistic precision.

B. Verse Recall (Completion Tasks)

• Poor Performance: Models struggled significantly with free recall (generating the exact second verse from the first).
  • Gemma 3 models exhibited near-zero recall rates across all conditions.
  • Claude Sonnet 4.5 and DeepSeek V3.2 achieved low but non-zero recall (e.g., ~3.6% to 7.1% with only the poet's name).
• Cue Sensitivity:
  • Providing prose explanations offered modest improvements.
  • Providing word-shuffled versions of the target verse resulted in the highest recall rates (e.g., 11.9% for Claude), suggesting models rely heavily on lexical identity rather than robust sequence memorization.
  • Salient words (sparse lexical cues) did not improve performance and sometimes degraded it.

C. Verse Recognition (Multiple Choice)

• Significant Gap: Performance in multiple-choice settings was markedly higher than in completion settings.
  • For example, Gemma 3 models, which had near-zero recall, performed competitively (and sometimes better than DeepSeek) in multiple-choice identification.
  • Accuracy jumped to ~90-98% for top models when provided with options.
• Error Analysis: When models failed, they tended to select distractors from the same ghazal rather than different ones, indicating sensitivity to thematic/stylistic properties within a poem.
• Word Order Sensitivity: In binary-choice tasks with shuffled distractors, models performed well above chance, proving they can distinguish canonical order from random permutations.

D. Cross-Lingual Comparison (English Sonnets)

• Models achieved substantially higher recall rates on Shakespeare's sonnets compared to Persian ghazals (e.g., Claude reached 96.3% recall on English with only the poet's name vs. 3.6% on Persian).
• This confirms that the inability to recall Persian verses is a distributional issue (lack of training data) rather than an architectural failure.

5. Significance and Implications

• Rethinking Evaluation: The study argues that standard benchmarks relying on recognition or paraphrase tasks overestimate an LLM's ability to engage with culturally grounded language. A model that can explain a poem may still fail to participate in the cultural practice of quoting or completing it.
• Cultural Competence: The findings highlight a critical gap in multilingual LLMs: semantic competence transfers more easily than form-level retrieval. Models can understand the meaning of a cultural text without having access to its exact surface form.
• Future Directions: The authors suggest that future training strategies should aim to strengthen access to canonical forms without encouraging excessive verbatim memorization (privacy risks). Evaluation frameworks must evolve to jointly assess meaning, form, and cue-dependent access.

In conclusion, GhazalBench reveals that while LLMs are becoming culturally aware in terms of semantics, they remain "amnesiac" regarding the precise textual forms of canonical literature in low-resource languages, a limitation driven by training data density rather than model architecture.