Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining

This paper demonstrates that synthetic rewriting acts as a scale-dependent quality multiplier for Portuguese continued pretraining, significantly boosting performance when applied to high-quality source data but offering minimal gains for low-quality data.

The Gist

Imagine you are trying to teach a student (an AI model) a new language (Portuguese) using a limited amount of textbooks. You have two main questions:

1. Does it matter if the textbooks are good quality?
2. Can a smart tutor rewrite those textbooks to make them better, even if the originals were messy?

This paper, titled "Synthetic Rewriting as a Quality Multiplier," answers these questions by running a massive experiment with AI. Here is the story of what they found, explained simply.

The Setup: The Library and the Tutor

The researchers started with a giant library of Portuguese text called ClassiCC-PT. They sorted this library into two piles:

• The "Gold" Pile: High-quality documents (like textbooks, scientific papers, and educational materials).
• The "Rough" Pile: Lower-quality documents (like messy forum posts, spam, or poorly written blogs).

They took a small, equal amount of text from each pile (10 billion words) to use as their "base curriculum."

Then, they brought in a Super Tutor (a 7-billion-parameter AI model). The Tutor's job was to take every single document from both piles and rewrite it in four different styles:

• Easy: Simplified for a child.
• Medium: Clean and encyclopedic (like Wikipedia).
• Hard: Technical and formal.
• Q&A: Turned into a quiz format.

This process created a huge amount of new synthetic data (about 40 billion words total) for the students to learn from.

The Students: Two Different AI Models

They tested this curriculum on two different "students" (AI models):

1. The Small Student (1.1 Billion parameters): A smaller, less powerful brain.
2. The Big Student (7 Billion parameters): A larger, smarter brain.

They taught these students using the rewritten data and then tested them on a massive Portuguese exam called PoETa V2 (44 different tasks like math, reasoning, and understanding text).

The Big Discovery: Rewriting is a "Quality Multiplier"

Here is the most important lesson from the paper, using a simple analogy:

Imagine you have a recipe.

• High-Quality Data is like a recipe written by a Michelin-star chef.
• Low-Quality Data is like a recipe scribbled on a napkin with missing ingredients.
• Rewriting is like a food critic tasting the recipe and rewriting the instructions to be clearer and more precise.

What happened with the Big Student (7B)?

• Chef's Recipe + Critic's Rewrite: The student learned incredibly well. The rewrite acted like a super-charger, boosting performance significantly. The critic made the good recipe even better.
• Napkin Recipe + Critic's Rewrite: The student barely improved. Even though the critic tried to fix the napkin recipe, the missing ingredients (bad source data) meant the final dish was still mediocre. The rewrite couldn't fix the fundamental flaws.

The Conclusion: Rewriting doesn't turn trash into treasure. Instead, it acts as a multiplier. If you start with gold, rewriting turns it into platinum. If you start with lead, rewriting just gives you slightly shinier lead.

The Twist: Size Matters (The Small Student)

When they tested the Small Student (1.1B), the rules changed.

• The Small Student didn't care much about the rewriting.
• In fact, the Small Student did just as well (or sometimes better) with the original, messy napkin recipes as it did with the rewritten, polished ones.

Why?
Think of the Small Student as a child who learns best by seeing many different examples, even if they are messy. The rewritten data was too "clean" and "structured," which actually made it less diverse. The Small Student needed the raw chaos of the internet to learn patterns, while the Big Student needed the clean, structured data to learn deep concepts.

The Verdict

1. Don't try to fix bad data with AI rewriting. If your source data is low quality, rewriting it won't save you. You need to curate (select) good data first.
2. Rewriting is a force multiplier for good data. If you already have high-quality Portuguese text, rewriting it in different styles makes your AI model significantly smarter.
3. Bigger models need cleaner data. The larger, smarter AI models benefit the most from this "quality multiplier" effect. Smaller models might actually prefer raw, diverse data over polished rewrites.

In short: Synthetic rewriting is like a high-end photo filter. It makes a beautiful photo look stunning, but it can't turn a blurry, dark photo into a masterpiece. You still need to start with a good picture.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) have seen uneven progress across languages, with non-English languages like Portuguese suffering from a scarcity of high-quality, large-scale monolingual corpora. While continued pretraining (adapting English-centric models on target-language data) is a standard strategy, it is often constrained by limited data budgets.

Recent work suggests synthetic data generation (specifically document rewriting) can improve pretraining efficiency. However, a critical gap exists in understanding how the quality of the source data interacts with rewriting. It remains unclear whether rewriting can compensate for low-quality inputs or if it merely amplifies the quality of already curated data. This paper addresses this by asking: Can synthetic rewriting amplify the benefits of limited Portuguese pretraining, and does the original data quality matter?

2. Methodology

The authors conducted a controlled study using ClassiCC-PT, a 120B-token Portuguese corpus annotated with continuous STEM and Educational quality scores (0–5).

Data Construction

• Source Subsets: Two 10B-token subsets were created:
  • High-Quality (Edu): Documents with STEM/Educational scores > 2.5.
  • Low-Quality (Non-Edu): Documents with scores between 0.5 and 2.0.
• Synthetic Rewriting: Each subset was rewritten using an instruction-tuned Qwen-2.5-7B model into four distinct styles:
  1. Easy: Simplified vocabulary and sentence structure.
  2. Medium: Clean, encyclopedic (Wikipedia-like) style.
  3. Hard: Technical and formal with domain-specific terminology.
  4. QA: Converted into question-and-answer format.
• Training Conditions:
  • Rewritten Conditions: Combined 10B original tokens + 30B synthetic tokens (4 styles) = **40B unique tokens**.
  • Baseline Conditions: 10B original tokens repeated for multiple epochs to match the 40B token budget (ensuring equal compute cost, though lower unique diversity).

Models and Evaluation

• Base Models: Two English-centric models were continued pretrained:
  • 1.1B parameters (TinyLLaMA-based).
  • 7B parameters (LLaMA-2-based).
• Benchmark: PoETa V2, a comprehensive Portuguese benchmark with 44 tasks across linguistic and reasoning domains.
• Metric: Normalized Performance Metric (NPM), scaling scores from 0 (random) to 100 (perfect) to allow cross-task averaging.

3. Key Contributions

1. Systematic Quality-Rewriting Interaction: This is the first study to explicitly isolate the interaction between source data quality and synthetic rewriting for Portuguese, demonstrating that rewriting is not a universal fix but a quality multiplier.
2. Scale-Dependent Findings: The paper reveals that the efficacy of rewriting is heavily dependent on model scale, showing strong benefits at 7B but negligible or inconsistent benefits at 1.1B.
3. Controlled Experimental Design: By using a fixed token budget and a high-quality annotated corpus (ClassiCC-PT), the study disentangles the effects of data quantity, data quality, and synthetic transformation.

4. Key Results

A. The 7B Model (Large Scale)

• High-Quality Data: Rewriting high-quality data yielded a significant gain of +3.4 NPM over the unmodified high-quality baseline (41.0 vs. 37.7). The learning curve continued to rise, suggesting sustained learning.
• Low-Quality Data: Rewriting low-quality data provided only a marginal gain of +0.5 NPM (35.8 vs. 35.2).
• Conclusion: Rewriting acts as a multiplier for good data, not a substitute for poor data. The "quality gap" between high and low data widens when rewriting is applied.
• Category Nuances:
  • Exams & Brazil-specific tasks: Showed the largest quality effects (e.g., +13 NPM gap between high/low quality with rewriting).
  • Ethics: Rewriting helped regardless of source quality.
  • General Knowledge: Uniquely, rewriting hurt performance on high-quality data, suggesting potential distortion of factual content.

B. The 1.1B Model (Small Scale)

• Weak Interaction: The clear hierarchy observed in the 7B model disappeared. Unmodified low-quality data performed competitively with rewritten high-quality data.
• Performance: Absolute NPM scores were lower (11–15 range). The curves for different conditions crossed frequently.
• Hypothesis: Smaller models may lack the capacity to leverage the structural complexity introduced by rewriting, or they benefit more from the raw diversity of unfiltered web data than from cleaner, homogeneous rewrites.

5. Significance and Implications

• Data Curation Strategy: The study argues against using synthetic rewriting as a "magic bullet" to salvage low-quality data. Instead, it should be applied as a multiplier to already curated, high-quality datasets to maximize returns.
• Resource Allocation: For low-resource languages like Portuguese, efforts should focus on filtering for high-quality source data first, then applying rewriting. Applying rewriting to noisy data yields diminishing returns.
• Model Scale Considerations: The benefits of synthetic rewriting are scale-dependent. Practitioners training smaller models (<1B) may not see the same gains from rewriting as those training 7B+ models and might prioritize raw data diversity instead.
• Limitations: The study notes that rewritten conditions had more unique tokens than baselines (40B vs 10B unique), which could partially explain gains, though the asymmetric results across quality tiers suggest quality is the primary driver. Future work should include matched-diversity controls.

In summary, the paper establishes that synthetic rewriting is a powerful tool for amplifying high-quality data in continued pretraining, but its effectiveness is strictly conditional on both the quality of the source material and the capacity of the model being trained.