Tucano 2 Cool: Better Open Source LLMs for Portuguese

The paper introduces Tucano 2, a fully open-source suite of 0.5–3.7 billion parameter large language models for Portuguese, featuring enhanced datasets and training recipes that achieve state-of-the-art performance on language benchmarks while providing comprehensive, reproducible resources for the community.

The Gist

Imagine you're trying to teach a brilliant student how to speak, write, and think in Portuguese. But here's the catch: most of the world's best teachers and textbooks are written in English, and the few Portuguese resources available are either too old, too expensive, or kept behind locked doors.

This paper, "Tucano 2," is like a group of dedicated teachers (from the University of Bonn and the Polyglot project) who decided to build their own open-source school for Portuguese. They didn't just build a classroom; they built the entire curriculum, the textbooks, the grading system, and the teacher training manuals, and then gave it all away for free.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: The "Language Gap"

Think of the internet as a giant library. For English, the library is overflowing with millions of high-quality books. For Portuguese, the library is much smaller, and many of the books are messy, full of errors, or written in a way that's hard for a computer to understand.

• The Old Way: Big tech companies build massive "multilingual" models (like a Swiss Army knife) that try to speak 100 languages at once. They are good, but they aren't great at Portuguese because they have to spread their brainpower too thin.
• The Tucano Way: They wanted to build a specialist—a "Portuguese-only" expert that knows the language inside and out, without needing to be a jack-of-all-trades.

2. The Ingredients: "GigaVerbo-v2" (The Cookbook)

To teach a computer, you need data. The team created a massive dataset called GigaVerbo-v2.

• The Raw Material: They scraped the web for Portuguese text, but the internet is full of spam and junk.
• The Filter: They used a "smart filter" (powered by an AI judge) to sort the good books from the bad. They looked for educational quality (is this a good textbook?) and toxicity (is this mean or harmful?).
• The Secret Sauce (Synthetic Data): They realized some topics (like advanced math or coding) were missing. So, they used other powerful AIs to write new, high-quality Portuguese lessons specifically to fill those gaps. It's like hiring a ghostwriter to fill in the missing chapters of a textbook.

3. The Brain: "Tokenizer Transplantation"

Imagine you have a dictionary with 150,000 words, but you only need 50,000 to speak Portuguese fluently. Using the big dictionary makes the computer slow and clumsy.

• The Trick: They took a powerful, pre-trained AI (from the Qwen family) and performed a "brain transplant." They replaced its giant, messy dictionary with their own custom, efficient Portuguese dictionary.
• The Result: The model kept all the intelligence it already had but became much faster and more efficient at speaking Portuguese, like swapping a heavy, clunky backpack for a sleek, custom-fitted one.

4. The Training: "The Three-Stage Diet"

They didn't just feed the model random data. They designed a specific "diet" (training recipe) with three stages:

1. Warm-up: Feeding it high-quality educational content to build a strong foundation.
2. Stable: Adding in more variety, including synthetic data and reasoning tasks.
3. Refinement: Focusing heavily on pure Portuguese data to polish the accent and grammar.

• The "Think" Mode: They also created a special version of the model that learns to "think before it speaks." When asked a hard math problem, it doesn't just guess; it writes out its reasoning steps (like showing your work in math class) before giving the final answer.

5. The Results: Beating the Giants

They tested their new models (ranging from tiny 0.5 billion parameters to a robust 3.7 billion) against other models.

• The Surprise: Even though their models were smaller than the massive "multilingual" giants, they outperformed them on Portuguese tasks.
• Why? Because they were specialists. A specialist doctor knows more about the heart than a general practitioner who knows a little about everything.
• The "Instruct" vs. "Think" Models:
  • Instruct: Great at following commands, writing code, and summarizing text.
  • Think: Great at solving complex logic puzzles and math problems, all while reasoning entirely in Portuguese.

6. The "Open Source" Promise

The most important part of this paper isn't just the models; it's the transparency.

• Most AI companies say, "Trust us, our model is good," but they hide the data and the code.
• The Tucano team said, "Here is the recipe, here are the ingredients, here is the code, and here is the energy bill." They released everything so that anyone, anywhere, can learn from it, improve it, or build their own version.

The Big Picture Analogy

Imagine the world of AI as a race.

• The Big Tech Giants are driving massive, expensive Formula 1 cars that are built to race on any track (any language), but they are heavy and slow on the specific, winding roads of Portuguese.
• The Tucano Team built a lightweight, custom-built motorcycle. They didn't have the biggest engine, but they tuned the tires, the suspension, and the fuel specifically for the Portuguese road.
• The Result: On the Portuguese track, the motorcycle zooms past the F1 car. And because they shared the blueprints, everyone else can now build their own motorcycles.

In short: This paper proves that you don't need billions of dollars to build a great AI for a specific language. You just need high-quality data, smart engineering, and a commitment to sharing your work with the world.

Technical Summary

Here is a detailed technical summary of the paper "Tucano 2: Better Open Source LLMs for Portuguese" by the Polyglot project team.

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs), high-quality, fully open-source resources for Portuguese remain scarce. Existing solutions suffer from several critical gaps:

• Reproducibility: Many high-performing Portuguese models are closed-source or lack transparency regarding data composition, training recipes, and evaluation protocols.
• Performance Gaps: Multilingual models often underperform on Portuguese-specific tasks compared to English-centric counterparts, while native Portuguese models often lack the scale or reasoning capabilities of frontier multilingual models.
• Data Quality: There is a lack of large-scale, high-quality, annotated Portuguese corpora that filter for educational value and toxicity, and a scarcity of synthetic data to fill domain gaps.
• Evaluation Noise: Existing Portuguese benchmarks often rely on generative tasks that provide noisy signals during early training stages, making it difficult to track progress efficiently.

2. Methodology

The authors propose a comprehensive, fully open pipeline for developing Portuguese LLMs, covering data curation, tokenization, pretraining, continual pretraining, and post-training.

A. Data Curation: GigaVerbo-v2

The team constructed GigaVerbo-v2, a massive corpus of approximately 320 billion tokens (372 million documents).

• Construction: Built using a FineWeb 2-inspired pipeline involving extraction, language identification (FastText + GlotLID), heuristic filtering, and deduplication.
• Quality Filtering: They employed Qwen2.5-32B as an LLM judge to annotate 700k documents for educational quality and toxicity. These annotations were distilled into lightweight classifiers (BERTimbau, BERTabaporu) to filter the full corpus.
• Synthetic Augmentation: To address domain gaps, they created GigaVerbo-v2 Synth (9.3 billion tokens) using Qwen2.5 models to generate high-quality educational, legal, and reasoning data.
• SFT & Preference Data: They assembled GigaVerbo-v2 SFT (4M examples) covering 12 task types (coding, math, reasoning, RAG) and GigaVerbo-v2 Preferences (28k pairs) for alignment.

B. Tokenization

They trained a custom SentencePiece tokenizer on a 40-40-20 mixture of Portuguese, English, and code.

• Vocabulary: 49,152 tokens.
• Efficiency: Achieved a subword fertility of 1.51 (vs. ~1.9 for Qwen3/Llama) and 2.88 characters per token, resulting in an estimated 30% reduction in compute costs compared to Qwen3's tokenizer for Portuguese tasks.

C. Training Strategies

The paper introduces two distinct training regimes:

1. From-Scratch Pretraining (Tucano2-0.6B-Base):
   • Trained a 670M parameter model from scratch using a 3-stage curriculum (Warmup-Stable-Decay).
   • Optimization: Utilized a hybrid AdamW + Muon optimizer for improved gradient stability.
   • Data Mix: 63% Portuguese (prioritizing educational content) and 37% English/reasoning data.
2. Continual Pretraining (Tucano2-qwen Family):
   • Adapted Qwen3 base models (0.6B, 1.7B, 4B) to Portuguese.
   • Tokenizer Transplantation: Used Orthogonal Matching Pursuit (OMP) to transplant the authors' efficient 49k-token Portuguese tokenizer into Qwen3 models, removing ~68% of embedding parameters without retraining.
   • Adaptation: Continued pretraining on purely Portuguese data (Web + Synthetic) with minimal compute (<0.3% of original pretraining FLOPs).

D. Post-Training

• Instruct Models: Trained on the full SFT mixture + non-reasoning preference data.
• Think Models: Trained exclusively on reasoning-intensive tasks (Chain-of-Thought) to enable native Portuguese reasoning traces, addressing the difficulty of generating high-quality CoT in Portuguese.
• Alignment: Used Anchored Preference Optimization (APO) for safety and helpfulness alignment.

E. Evaluation Framework

The authors introduced a Unified Portuguese Evaluation Suite divided into two tiers to address signal noise:

• Easy Set: Log-likelihood based benchmarks (ARC, Calame, PIQA, HellaSwag, Lambada) providing reliable signals at <200B tokens.
• Hard Set: Advanced benchmarks (ENEM, OAB, MMLU, Belebele) requiring >660B tokens for meaningful signals.
• Post-Training Benchmarks: Adapted versions of IFEval, GSM8K, RULER (long-context), and HumanEval for Portuguese.

3. Key Contributions

1. GigaVerbo-v2 & Synth: The largest openly available, high-quality Portuguese corpus with educational/toxicity annotations and synthetic augmentation.
2. Tucano 2 Model Family: A suite of open models (0.5B to 3.7B) including Base, Instruct, and Think variants.
3. Tokenizer Transplantation: A successful application of OMP to adapt Qwen3 models with a custom, efficient Portuguese tokenizer.
4. Dual-Reasoning Strategy: The separation of "Instruct" and "Think" models to specifically tackle the challenge of Portuguese Chain-of-Thought generation.
5. Full Reproducibility: Release of all datasets, training recipes, logs, code, and evaluation harnesses under permissive licenses.

4. Results

• From-Scratch Performance: Tucano2-0.6B-Base achieved an aggregate NPM score of 40.28 on the Easy Set, outperforming Qwen3-0.6B (26.41) and Curió-1.1B (39.16) despite using significantly less compute.
• Continual Pretraining: Tucano2-qwen-3.7B-Base achieved a total average NPM of 59.21, surpassing Qwen3-4B-Base (57.86) and Qwen2.5-7B (57.97) while using negligible additional compute.
• Post-Training:
  • Tucano2-qwen-3.7B-Instruct achieved the highest Knowledge & Reasoning score (56.22) in the 3-4B range, beating Qwen3-4B and Gemma-3-Gaia-PT.
  • Tucano2-qwen-3.7B-Think achieved the top Knowledge & Reasoning score (54.07) among reasoning models, generating Chain-of-Thought entirely in Portuguese.
• Efficiency: The continual pretraining strategy yielded massive gains with <0.13% of the total compute cost of the base model's pretraining.
• Limitations: Models lagged behind in Instruction Following (IFEval) and Coding (HumanEval) due to data mixture limitations, and showed reduced performance in long-context (RULER) tasks compared to native Qwen3 models.

5. Significance

• Bridging the Gap: Tucano 2 demonstrates that carefully curated, language-specific data pipelines and efficient tokenization can allow smaller, resource-constrained models to outperform larger, general-purpose multilingual models on specific languages.
• Reproducibility Standard: By releasing the full stack (data, code, recipes), the project sets a new standard for open research in low-resource languages, countering the trend of "black box" model releases.
• Sustainability: The paper highlights the environmental cost of synthetic data generation (73% of total energy) and advocates for more efficient generation strategies, while proving that targeted adaptation is far more energy-efficient than training from scratch.
• Community Impact: The release provides the Portuguese NLP community with the foundational resources needed to build downstream applications, fostering community-driven innovation in a region previously underserved by open AI.