Technical Summary: LLMs Got Rhythm? Hybrid Phonological Filtering for Greek Poetry

Problem Statement

Large Language Models (LLMs), despite their proficiency in general NLP tasks, exhibit significant deficiencies in phonologically-grounded phenomena, specifically rhyme detection and generation. This limitation is particularly acute in Modern Greek, a lower-resource language where rhyme is a structural pillar of poetry ranging from the Cretan Renaissance to contemporary hip-hop.

The core issue stems from the misalignment between LLM tokenization (based on statistical co-occurrence, e.g., BPE or WordPiece) and phonological units (syllables, stress patterns, and rhyme domains). In Greek, historical orthography creates many-to-one mappings (e.g., the sound /i/ can be spelled in six different ways), and rhyme is strictly defined by stress position (Masculine, Feminine-2, Feminine-3). LLMs often rely on visual similarity ("eye rhyme") rather than phonetic identity, leading to failures in distinguishing minimal pairs differing only in stress or in generating valid rhyming structures. Pure LLM generation for constrained poetry tasks was found to fail catastrophically, with validity rates under 4%.

Methodology: Hybrid Neuro-Symbolic Architecture

The authors propose a hybrid system that integrates the generative and reasoning capabilities of LLMs with deterministic, rule-based phonological algorithms. The architecture consists of two primary components:

1. The Phonological Engine (Symbolic Verifier)

Based on the theoretical framework of Topintzi et al. (2019), this module performs deterministic processing:

• Syllabification: Identifies syllable boundaries based on Greek-specific phonotactic constraints.
• Stress Detection: Utilizes Greek orthographic accent marks to identify stress, handling clitic extension (e.g., weak object pronouns).
• Rhyme Domain Extraction: Defines the rhyme domain from the stressed vowel to the end of the phonological phrase, capable of spanning multiple words (Mosaic rhymes).
• Phonetic Transcription: Converts orthography to phonetic representation to bypass visual mismatches.
• Classification: Assigns compound labels (e.g., F2-TR-S-IDV) based on stress position, onset matching (Rich rhymes), pre-stress vowels (IDV), and imperfections.

2. LLM Integration and Agentic Pipeline

The system evaluates multiple prompting strategies (Zero-Shot, Chain-of-Thought, RAG-augmented) across proprietary models (Claude 3.7/4.5, GPT-4o, Gemini 2.0) and open-weight models (Llama 3.1/3.3, Mistral Large).

For Rhyme Generation, the authors implement an Agentic Generate-Verify-Refine Loop:

1. Generate: The LLM produces a poem draft based on a theme and rhyme constraints.
2. Verify: The Phonological Engine checks every rhyme pair against the rules.
3. Refine: If errors are detected, the engine formats specific feedback (e.g., "Stress mismatch: Expected F3, found F2") and updates the prompt for the LLM to regenerate.
   This loop runs for up to 15 attempts before returning a result or a warning.

Key Contributions

1. Dataset: A curated corpus of over 40,000 rhymes aggregated from the Anemoskala archive and an Interwar Poetry dataset, standardized and annotated according to a comprehensive Greek rhyme taxonomy.
2. Hybrid Detection System: A system combining LLM prompting with rule-based verification, achieving 100% verification accuracy compared to a maximum of 54% for pure LLM approaches.
3. Agentic Generation Pipeline: A loop that elevates the validity of generated poems from near 0% to 73.1% by enforcing strict phonological constraints.
4. Multi-Model Evaluation: A comprehensive benchmark across eight models and four prompting configurations, highlighting the "Reasoning Gap" in phonological tasks.

Results

Rhyme Identification

• Reasoning Gap: There is a significant performance disparity between models based on prompting. Claude 4.5 achieved 53.8% accuracy with Chain-of-Thought (CoT) and RAG, compared to 26.9% without CoT. Pure LLM approaches generally struggled, with open-weight models performing poorly (e.g., Llama 3.1 8B at 3.8–15.4%).
• Stress Sensitivity: All models struggled most with Feminine-3 (proparoxytone) rhymes, with accuracy often below 22%.
• Feature Detection: Mosaic rhymes (spanning word boundaries) were undetectable by any model in quantitative feature matching (0% accuracy), though Claude 4.5 with CoT+RAG showed qualitative success in qualitative analysis.
• Model Size: Larger models (Llama 3.1 70B) significantly outperformed smaller variants (Llama 3.1 8B), suggesting that rhyme detection in Greek benefits from increased parameter counts.

Rhyme Generation

• Validity Improvement: Pure LLM generation resulted in 0–4% valid poems. The inclusion of the verification loop increased validity to 73.1% for Claude 3.7, 42.3% for GPT-4o, and 34.6% for Claude 4.5.
• Error Correction: The loop successfully corrected stress pattern mismatches (e.g., defaulting to F2 instead of F3), refined feature precision (distinguishing Rich vs. Pure rhymes), and eliminated non-rhyming pairs.
• Creativity vs. Hallucination: The verification loop encouraged capable models (particularly Claude 4.5) to generate plausible neologisms (e.g., pyrafí for "fire-colored") when standard vocabulary failed, rather than producing nonsense. This suggests a trade-off where strict constraints drive creative morphological invention.

Significance and Claims

The paper posits that LLMs fundamentally lack the sub-token phonological awareness required for rigorous poetic tasks, particularly in languages with complex orthography-stress relationships like Modern Greek. The authors claim that hybrid neuro-symbolic architectures are essential for mastering these structural nuances.

By decoupling reasoning (LLM) from phonological scoring (Symbolic Engine), the system leverages the semantic richness and creativity of LLMs while offloading the precise computational constraints to a deterministic module. The authors conclude that while LLMs possess latent creative capabilities, they cannot independently satisfy the strict phonological constraints of poetry without external verification. The released codebase and the 40,000+ rhyme corpus are presented as foundational resources for future research into phonologically-aware NLP and low-resource language poetry generation.

The authors remain modest regarding limitations, noting that their corpus is smaller than those available for high-resource languages and that the iterative verification process introduces computational overhead compared to single-pass inference. Future work is proposed to extend this to metrical verification and Reinforcement Learning from Phonological Feedback (RLPF).