💬 NLP

Instruction-Guided Poetry Generation in Arabic and Its Dialects

This paper introduces a large-scale, instruction-based dataset in Modern Standard Arabic and its dialects that enables Large Language Models to controllably generate, revise, and analyze poetry according to specific user requirements like style and rhyme.

Original authors: Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Fajri Koto

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Fajri Koto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Arabic poetry as a grand, ancient library filled with millions of beautiful books. For centuries, people have written poems in this library using strict rules about rhythm, rhyme, and style, much like building a house with specific blueprints. However, until now, computers (specifically Large Language Models or LLMs) have mostly been like librarians who can only read the books, count the pages, or guess who wrote them. They haven't been very good at writing new poems themselves, especially when asked to follow specific rules or write in different regional accents (dialects).

This paper introduces a new project called InstructPoet, which teaches computers how to become poets themselves. Here is how they did it, explained simply:

1. The Recipe Book (The Dataset)

To teach a computer to cook, you need a good cookbook. The researchers gathered a massive collection of Arabic poems from the internet, covering everything from ancient times to modern day.

The Cleanup: They organized this messy pile of poems into a neat, uniform format, like sorting a jumbled box of LEGOs into sorted bins.
The Labels: They added "metadata" (labels) to every poem, tagging them with details like "Romantic," "War," "Abbasid Era," or "Gulf Dialect."
The Translation: They didn't just use standard Arabic (MSA). They created instructions in five different flavors: Standard Arabic plus four major dialects (Gulf, Levantine, Nile Valley, and North African). This is like teaching the computer to speak not just formal English, but also Texan, Scottish, and Australian English, so it can talk to anyone naturally.

2. The Four Training Games (The Tasks)

Instead of just asking the computer to "write a poem," they set up four specific training games to make the computer smarter:

Generation (The Blank Page): "Write a romantic poem about the sea using a specific rhythm." The computer has to create something from scratch.
Continuation (The Cliffhanger): "Here are the first three lines of a poem; please finish the rest." The computer has to keep the story and rhythm going without breaking the flow.
Revision (The Editor): "Here is a poem with some typos and broken rhythm; fix it." The computer acts like a proofreader, restoring the poem to its perfect state.
Analysis (The Quiz): "Read this poem and tell me: What era is it from? What is the rhyme scheme?" The computer acts like a critic taking a test.

3. The Classroom (Training the Models)

The researchers took four different "student" computers (existing AI models) and taught them using this new dataset.

The Students: They used two models that are good at many languages (LLaMA and Qwen) and two that specialize in Arabic (ALLaM and Fanar).
The Teaching Style: They tried two methods. One was Joint Training (throwing all the games at the student at once), and the other was Curriculum Learning (teaching the easy stuff first, like analysis, and saving the hard stuff, like writing from scratch, for later).

4. The Report Card (Results)

After the training, they tested the students in two ways:

The Robot Grader: They used another AI to grade the poems on things like grammar, flow, and whether the computer actually followed the rules (like using the right rhyme).
The Human Judges: They hired native Arabic speakers who love poetry to read the poems and rate them on a scale of 1 to 5.

The Findings:

Huge Improvement: Before training, the computers were terrible at writing poetry that followed rules. After training, they got significantly better.
The Best Student: The ALLaM model, which was already designed for Arabic, became the top poet after training. It scored the highest in fluency and coherence.
The Hardest Task: Writing a poem from scratch (Generation) was the easiest for the computers. Fixing a broken poem (Revision) was the hardest, because it required the computer to understand the "soul" of the poem while fixing the technical errors.
Dialects Worked: The computers got much better at understanding and writing in different Arabic dialects, not just the formal version.

The Bottom Line

This paper doesn't claim that computers will replace human poets or that they can write perfect, soul-stirring masterpieces yet. Instead, it shows that we can now build a digital assistant that helps people write poetry. If you want a poem in a specific style, with a specific rhyme, in your local dialect, this new system can help you draft it, fix your mistakes, or even analyze what you've written. It turns the computer from a passive reader into an active, rule-following creative partner.

1. Problem Statement

Despite the central role of poetry in Arabic culture and language, current Large Language Models (LLMs) struggle with controllable Arabic poetry generation. Existing research has predominantly focused on analytical tasks (e.g., meter detection, poet attribution, theme classification) using relatively small datasets. There is a significant gap in:

Generative Capabilities: Few models can generate poetry that adheres to strict structural constraints (meter, rhyme) and stylistic requirements.
Dialectal Diversity: Most existing work is limited to Modern Standard Arabic (MSA), neglecting the rich diversity of regional dialects (Gulf, Levantine, North African, Nile Valley).
Instruction Following: Current models often fail to follow complex user instructions regarding constraints (e.g., "Write a romantic poem in the Abbasid style using the Basīṭ meter").

2. Methodology

The authors propose a comprehensive framework involving dataset construction, instruction fine-tuning (IFT), and rigorous evaluation.

A. Dataset Construction

The team aggregated a massive corpus from public sources (PoetsGate, Adab, Ashaar, etc.) and processed it through a multi-step pipeline:

Unification & Cleaning: Standardized verse structures (one verse per line), removed single-verse poems, and normalized metadata (era, genre, meter).
Metadata Enrichment: Used Gemini 2.5 Pro to automatically generate semantic metadata, including keywords (themes) and keyphrases (syntactic summaries), achieving 96% quality in manual evaluation.
Deduplication: Removed intra-source duplicates and ensured no data leakage between training and testing sets (specifically excluding the FannOrFlop benchmark from training).
Instruction Template Creation:
- Tasks: Defined four core tasks: Generation (composing new poems), Continuation (completing partial poems), Revision (fixing corrupted/damaged poems), and Analysis (Multiple Choice Questions on metadata).
- Subtasks: Created 54 subtasks targeting specific constraints (e.g., conditioned on rhyme, meter, era, or dialect).
- Linguistic Coverage: Designed instruction templates in Modern Standard Arabic (MSA) and four major dialects (Gulf, Levantine, Nile Valley, North African).
- Scale: The final dataset contains 1.35 million training pairs and 24.8k testing pairs, with 3,220 unique instruction templates.

B. Model Training

The authors fine-tuned four distinct LLMs using LoRA (Low-Rank Adaptation):

Models: Two Arabic-centric models (ALLaM-7B, Fanar-1-9B) and two multilingual models (LLaMA-3-8B, Qwen3-8B).
Training Regimes:
1. Joint Training: All tasks shuffled and optimized together.
2. Curriculum Learning: Tasks presented in increasing difficulty order (Analysis $\to$ Continuation $\to$ Generation $\to$ Revision).

C. Evaluation Framework

A multi-faceted evaluation strategy was employed:

LLM-as-a-Judge: Used Gemini 2.5 Flash to evaluate compliance, fluency, coherence, and poetic quality on a 1–5 scale.
Automated Metrics: Used lm-eval-harness for MCQ accuracy and standard metrics (BERTScore, ROUGE-L, Rhyme Adherence) for generation tasks.
Human Evaluation: Two native Arabic-speaking literary experts evaluated 400 generated samples blindfolded across the four criteria.

3. Key Contributions

InstructPoet-Ar Dataset: The first large-scale, instruction-based dataset for Arabic poetry covering MSA and four major dialects, supporting generation, continuation, revision, and analysis.
Comprehensive Task Framework: A unified system handling 54 subtasks that bridge creative generation and analytical understanding.
Dialectal Robustness: Demonstrated that instruction tuning significantly improves model performance across diverse Arabic dialects, not just MSA.
Curriculum vs. Joint Training: Provided empirical evidence on how training regimes affect poetic structure learning, showing that structured exposure (curriculum) often aids specific skills.

4. Results

Automatic & LLM-as-a-Judge Evaluation

Task Difficulty: Performance followed the expected difficulty gradient: Generation (easiest) > Continuation > Revision (hardest).
Dialects: Fine-tuning improved performance across all dialects without a single dialect being consistently "easiest" or "hardest."
Model Performance:
- ALLaM-7B (Arabic-centric) consistently outperformed others, achieving the highest scores in fluency and coherence.
- Qwen3-8B showed the largest relative improvement (+63%) after fine-tuning, suggesting strong adaptability despite a weaker Arabic baseline.
- Curriculum Learning: Generally yielded slight improvements over random shuffling, particularly in structured tasks.

Human Evaluation

Significant Gains: Fine-tuned models significantly outperformed their base counterparts.
- ALLaM-7B (FT): Scored 3.99/5.0 (up from 3.02).
- Qwen3-8B (FT): Scored 3.66/5.0 (up from 2.24).
Strengths: Fluency and Coherence were the strongest areas, often approaching human-level quality.
Weaknesses: Poetic Quality (artistic depth, metaphor) remained the lowest-scoring dimension (max 3.82/5.0), indicating that while models can mimic structure, capturing true artistic nuance is still challenging.
Statistical Significance: All improvements were statistically significant ( $p < 0.0001$ ).

5. Significance and Future Work

Bridging the Gap: This work shifts the focus from passive analysis to active, controllable creation in Arabic NLP, enabling practical tools for writers and educators.
Cultural Preservation: By supporting dialects and historical eras, the framework aids in preserving and generating diverse forms of Arabic literary heritage.
Limitations:
- Reliance on LoRA rather than full fine-tuning may limit the capture of complex stylistic nuances.
- The dataset is heavily skewed toward historical/classical poetry; contemporary and free-verse poetry are underrepresented.
Future Directions: The authors plan to expand into contemporary poetry, free verse, and full fine-tuning of larger models to further enhance artistic depth and dialectal coverage.

In conclusion, InstructPoet-Ar establishes a new benchmark for Arabic poetry generation, proving that instruction-tuned LLMs can effectively handle the complex structural, rhythmic, and dialectal constraints of Arabic poetry, though significant room remains for improvement in artistic creativity.