PersianPunc: A Large-Scale Dataset and BERT-Based Approach for Persian Punctuation Restoration

This paper introduces PersianPunc, a large-scale dataset of 17 million samples, and a fine-tuned ParsBERT model that achieves a 91.33% macro-averaged F1 score for Persian punctuation restoration, offering a more efficient and accurate alternative to large language models for real-time applications.

The Gist

Imagine you are listening to someone tell a story over a very bad phone connection. They speak clearly, but the line is so static-filled that all the pauses, commas, and periods disappear. You hear a continuous stream of words:

"No mercy needed execute him"

Without punctuation, this sounds terrifying. But what if the speaker actually meant:
"No mercy needed, execute him" (Wait, that's still bad).

Let's try another example from the paper:
"Forgiveness no need to execute him" vs. "Forgiveness, no need to execute him."

In the first version, it sounds like a command to kill. In the second, it sounds like a plea for life. Punctuation is the difference between a tragedy and a happy ending.

This paper is about teaching computers to fix this problem specifically for the Persian language. Here is the story of their solution, broken down simply.

1. The Problem: The "Run-On Sentence" Monster

In the world of Automatic Speech Recognition (ASR)—the tech that turns your voice into text (like Siri or Google Assistant)—the computer often forgets to put in commas, periods, or question marks. It just dumps a giant block of text.

For Persian, this is a huge headache. Persian is a "morphologically rich" language, meaning words change shape a lot, and the rules for where to put a comma are tricky. If a computer gets this wrong, it doesn't just look messy; it changes the meaning of the sentence entirely.

2. The Solution: Building a Massive Library (PersianPunc)

To teach a computer to fix this, you need to show it millions of examples of "bad" text and the "corrected" version.

The authors didn't just find a few examples; they built a giant library called "PersianPunc."

• The Scale: They gathered 17 million sentences. That's like reading every book in a massive national library.
• The Mix: They didn't just use formal news articles. They mixed in medical texts, Wikipedia, and even casual chat from Telegram and blogs. This ensures the computer learns how to punctuate a serious doctor's report and a friend's funny text message.
• The Cleaning: They acted like strict librarians, filtering out messy data, removing duplicates, and making sure the "correct" versions were actually correct.

3. The Brain: A Specialized Tutor (ParsBERT)

Once they had the library, they needed a teacher. They used a model called ParsBERT.

• The Analogy: Think of ParsBERT as a Persian-speaking tutor who has already read almost everything written in Persian. They took this tutor and gave it a specific homework assignment: "Look at this sentence without punctuation, and tell me exactly where the commas and periods go."
• The Result: This tutor became incredibly good at the job, getting it right 91.33% of the time.

4. The Showdown: The Specialist vs. The Generalist

The researchers also tested the "giants" of the AI world: Large Language Models (LLMs) like GPT-4. These are the super-smart, general-purpose AI brains that can write poetry, code, and answer trivia.

They asked the giants to fix the punctuation, but they gave them a strict rule: "Do not change any words. Only add punctuation."

Here is where the plot twist happens:

• The Giant (LLM): It was smart, but it was too smart. It had a bad habit of over-correcting. If it saw a word it thought was slang or slightly misspelled, it would "fix" the word itself, changing the meaning. It also required a massive amount of electricity (computing power) to do this.
• The Specialist (ParsBERT): It was lighter, faster, and followed the rules perfectly. It only added the punctuation and left the words exactly as they were.

The Verdict: The specialized tutor (ParsBERT) beat the giant AI in accuracy and was much cheaper to run. For real-time applications (like fixing a live phone call), you want the lightweight specialist, not the heavy, over-eager giant.

5. Why This Matters

This paper is a gift to the Persian-speaking world.

• For Developers: They now have a free, massive dataset and a working model to build better voice assistants.
• For Everyone: It means that when you talk to a Persian AI, it will understand the tone and meaning of your voice much better, turning a confusing run-on sentence into a clear, readable story.

In a nutshell: The authors built a massive training ground, taught a specialized Persian AI how to spot the missing pauses in speech, and proved that a focused, efficient expert is better than a distracted, over-achieving giant for this specific job.

Technical Summary

Here is a detailed technical summary of the paper "PersianPunc: A Large-Scale Dataset and BERT-Based Approach for Persian Punctuation Restoration."

1. Problem Statement

Punctuation restoration is a critical preprocessing step for Automatic Speech Recognition (ASR) outputs and informal digital text, significantly impacting the readability and performance of downstream NLP tasks (e.g., machine translation, sentiment analysis).

• The Gap: Despite the importance of Persian punctuation, the field suffers from a lack of large-scale, high-quality datasets and comprehensive models. Existing resources are small-scale, domain-specific, or lack public availability.
• The Challenge: Persian punctuation is semantically vital; minimal changes can drastically alter meaning or sentiment (e.g., changing a negative command to a positive request). Furthermore, while Large Language Models (LLMs) show promise, they often suffer from over-correction (altering words rather than just adding punctuation) and high computational costs, making them unsuitable for real-time ASR pipelines.

2. Methodology

A. Dataset Construction: PersianPunc

The authors curated PersianPunc, a large-scale dataset comprising 17,102,014 unique samples.

• Data Sources: Aggregated from six diverse corpora covering both formal and informal registers:
  • Formal: Bijankhan-Peykare Corpus, Persian Medical QA, Persian Wikipedia.
  • Informal: Persian Telegram Channels, Farsi Stories, Blog Dataset V2.
• Preprocessing Pipeline:
  • Normalization: Converted English punctuation to Persian equivalents, removed non-Persian characters, and normalized whitespace.
  • Segmentation & Filtering: Text was split into sentences. Strict filters were applied to ensure quality:
    • Minimum length of 10 characters.
    • Crucial Constraint: Sentences must contain at least two target punctuation marks (excluding simple sentence terminators) to focus on the complex task of internal punctuation (commas, colons, etc.).
    • Removal of URLs, emojis, excessive special symbols, and mixed-language content.
  • Deduplication: Used SHA-256 hashing on normalized text to remove duplicates across the 20M+ initial pool.
• Splits: A stratified 1M subset was created for training (989k), validation (10k), and testing (1k).

B. Model Architecture

The task is formulated as a token-level sequence labeling problem.

• Base Model: ParsBERT, a monolingual Persian BERT model pre-trained on a large Persian corpus.
• Head: A linear classification layer with dropout ($p=0.1$) projecting embeddings to 5 classes:
  1. EMPTY (no punctuation)
  2. COMMA (،)
  3. PERIOD (.)
  4. QUESTION (؟)
  5. COLON (:)
     (Note: Exclamation marks and semicolons were excluded as they are less frequent or interchangeable in Persian.)
• Token Handling: Labels are assigned only to the first subword of each word to align predictions with word-level punctuation.
• Training: Optimized with AdamW (LR $2\times10^{-5}$), batch size 85 (accumulated to 680), for 3 epochs.

3. Key Contributions

1. PersianPunc Dataset: The first large-scale (17M samples), publicly available dataset specifically curated for Persian punctuation restoration, spanning diverse domains and registers.
2. Systematic Curation Framework: A reproducible methodology for aggregating, filtering, and splitting Persian text, including detailed analysis of punctuation distribution patterns.
3. Efficient BERT-Based Solution: A fine-tuned ParsBERT model that achieves state-of-the-art performance while avoiding the computational overhead and over-correction issues associated with LLMs.
4. Critical Evaluation of LLMs: A rigorous comparison demonstrating that while LLMs (GPT-4o) are capable, they introduce significant "over-correction" errors (deleting/rewriting words) unsuitable for strict ASR post-processing.

4. Results and Analysis

Performance Metrics

• Macro-Averaged F1 Score: 91.33%
• Micro-Averaged F1 Score: 97.28%
• Full Sentence Match (FSM) Rate: 61.80% (Percentage of sentences where the predicted punctuation sequence exactly matches the gold standard without any text modification).

Class-Specific Performance (Test Set)

• Period (.): 98.71% F1 (High frequency, consistent usage).
• Colon (:): 90.45% F1.
• Question Mark (؟): 88.89% F1.
• Comma (،): 80.03% F1 (Lower due to nuanced, context-dependent usage in Persian).

Comparison with Baselines and LLMs

Model	Macro F1	FSM Rate	Notes
ParsBERT (Ours)	91.33%	61.80%	Lightweight, no text modification.
GPT-4o	85.96%	50.10%	High over-correction (~5% of samples).
GPT-4o-mini	79.54%	38.01%	Lower performance, high over-correction.
ViraPart (Prior)	92.13%	N/A	Evaluated on different (smaller) dataset.
CRF (Prior)	69.00%	N/A	Evaluated on different (smaller) dataset.

Key Findings on LLMs:

• GPT-4o exhibited over-correction in ~5% of cases, including deleting words, replacing informal terms with formal ones, and "correcting" perceived spelling errors.
• This behavior is detrimental to ASR pipelines where the transcribed text must remain verbatim.
• LLMs require significantly higher computational resources compared to the lightweight ParsBERT model.

5. Significance and Future Work

• Resource Availability: The release of the 17M sample dataset and the fine-tuned model fills a critical gap in Persian NLP resources, enabling research in low-resource, morphologically rich languages.
• Practical Application: The proposed approach is highly efficient, making it suitable for real-time ASR post-processing where speed and fidelity to the original transcript are paramount.
• Future Directions:
  • Developing domain-specific models (e.g., for literature or news).
  • Integrating prosodic information from speech to aid punctuation restoration.
  • Jointly handling punctuation and Zero-Width Non-Joiner (ZWNJ) insertion to address broader Persian text normalization challenges.

In conclusion, the paper establishes that a specialized, fine-tuned BERT model trained on a massive, curated dataset outperforms general-purpose LLMs in both accuracy and reliability for Persian punctuation restoration, offering a scalable solution for real-world applications.