EngGPT2: Sovereign, Efficient and Open Intelligence

EngGPT2-16B-A3B is a sovereign, open-weight, and efficient Mixture-of-Experts large language model trained from scratch with a significant Italian data corpus to deliver high-performance, EU AI Act-compliant reasoning capabilities in both Italian and English while requiring substantially less training data and inference power than comparable dense models.

The Gist

Imagine the world of Artificial Intelligence as a massive, high-stakes cooking competition. For years, the biggest chefs (like the US and China) have been building "Super-Kitchens" with unlimited ingredients and giant ovens to create the most complex dishes (AI models). They use trillions of ingredients (data tokens) and massive amounts of energy.

EngGPT2 is the story of a new, clever Italian team (Engineering Group) that decided to win not by having the biggest kitchen, but by being the most efficient, sovereign, and smart chef in the room.

Here is the story of EngGPT2, broken down into simple concepts:

1. The "Smart Kitchen" Design (The Architecture)

Most AI models are like a kitchen where every single chef tries to cook every single dish. If you have 16 billion chefs, that's a lot of work, even for a simple salad.

EngGPT2 uses a Mixture-of-Experts (MoE) design. Imagine a kitchen with 64 specialized chefs (experts), but for every dish you order, the head chef only wakes up 8 of them to do the work.

• The Analogy: Instead of 16 billion chefs working on a sandwich, you only use 3 billion "active" chefs. The other 13 billion are resting, saving energy and money.
• The Result: You get a delicious, high-quality meal (smart answers) using only a fraction of the electricity and time.

2. The Secret Recipe (The Data)

Big models usually eat everything: the internet, books, code, and random noise. They eat about 15 to 36 trillion "bites" of data.

• EngGPT2's Diet: They ate a smaller, higher-quality meal of 2.5 trillion tokens.
• The Special Ingredient: About 25% of their diet was Italian. While other models are mostly English speakers who know a little Italian, EngGPT2 was raised speaking Italian from day one. This makes it a "local expert" for European culture, laws, and language, ensuring it understands the nuances of the region it serves.

3. The Training Camp (The Process)

Training an AI is like sending a student through school. EngGPT2 went through four distinct grades:

1. Elementary (Pre-training): Learning to read and write in English and Italian by reading books, websites, and code.
2. High School (Long-Context): Learning to read a whole novel without forgetting the first page. They trained to handle very long documents (up to 32,000 words) so they can summarize a whole contract or a long report.
3. University (Mid-Training): Learning how to think logically. They practiced math, puzzles, and step-by-step reasoning to become a "thinker," not just a "talker."
4. Graduate School (Post-Training): Learning how to be polite, follow instructions, and act like a helpful assistant. They learned to say "I don't know" when appropriate and to follow safety rules (compliance with EU laws).

4. The "Turbo" Mode (Reasoning Styles)

One of the coolest features is how EngGPT2 "thinks out loud."

• Standard Mode: It gives you a direct answer.
• Full Reasoning Mode: It shows its homework. It writes down every step of its logic (like showing your work in math class) in English or Italian. This helps you trust why it gave the answer.
• Turbo Mode: This is the "speed dial." It compresses the thinking process into bullet points. It's like a chef saying, "I chopped, sautéed, and seasoned" instead of writing a 20-page recipe. It's much faster and cheaper to run, perfect for real-time apps.

5. Why It Matters (Sovereignty & Efficiency)

• Sovereignty: Europe wants its own AI that follows European rules (the EU AI Act) and values, rather than relying on models built in other countries. EngGPT2 is built from scratch in Europe, with full transparency.
• Efficiency: Because it uses the "8 chefs out of 64" trick, it costs one-fifth to one-half the energy to run compared to other models of similar smarts. It's like driving a hybrid car that gets the same mileage as a gas-guzzler but uses half the fuel.

The Verdict

EngGPT2 is a 16-billion-parameter model that punches way above its weight class.

• It beats other models of its size (8B–16B) in math, logic, and coding.
• It competes with much larger, more expensive models (30B+) but costs a fraction to train and run.
• It is the first open model that is truly "European," speaking Italian fluently and respecting local laws.

In short: EngGPT2 proves you don't need to be the biggest, loudest, or most expensive AI to be the smartest. With the right architecture, a focused diet, and a little bit of Italian flair, you can build a model that is powerful, efficient, and ready for the future.

Technical Summary

1. Problem Statement

Europe faces a strategic gap in Large Language Model (LLM) development compared to the US and China, often relying on foreign models that may not align with local regulatory frameworks (specifically the EU AI Act) or linguistic/cultural nuances. Existing European initiatives are often limited in scale or efficiency. Furthermore, there is a need for models that balance high performance with computational efficiency (lower training and inference costs) and sovereignty (open weights, transparent data provenance, and regulatory compliance).

2. Methodology

EngGPT2-16B-A3B is a Mixture-of-Experts (MoE) language model designed from scratch to address these challenges.

Architecture

• Model Type: Sparse MoE Transformer with 16 billion total parameters and 3 billion active parameters per token.
• Structure: 24 layers, each containing 64 experts, with 8 experts activated per token via dynamic routing.
• Components:
  • Attention: Grouped Query Attention (GQA) for memory efficiency.
  • Activation: SwiGLU.
  • Positional Embeddings: Rotary Positional Embeddings (RoPE).
  • Normalization: RMSNorm with pre-normalization.
  • Tokenizer: Expanded vocabulary of 131,084 tokens (based on Mistral), including 12 specialized tokens for reasoning and tool calling.
• Design Philosophy: The expert sizes are intermediate between Qwen3 and GPT-OSS to balance capacity (3B active params) with efficiency, avoiding the excessive costs of larger dense models.

Training Pipeline

The training process was executed on an HPC infrastructure (up to 128 nodes with NVIDIA A100 GPUs) using the Megatron framework and custom code derived from SmolLM3. It consisted of four distinct phases:

1. Pre-training (2.5 Trillion Tokens):
   • Data: A mix of web text, educational content, code, and PDFs. Approximately 25% of the corpus is Italian, with significant representation of other European languages.
   • Phases: Warmup (600B tokens), Main Scale-up (1.5T tokens), and High-Quality Refinement (400B tokens).
   • Compliance: Rigorous copyright filtering (domain blacklists, pattern matching, risk scoring) to ensure EU AI Act compliance.
2. Long-Context Adaptation:
   • Extended context window to 32,768 tokens (with YARN scaling for evaluation up to 64k+).
   • Used Context Parallelism to maintain efficiency at long sequence lengths.
3. Mid-Training:
   • Focused on consolidating reasoning capabilities using structured datasets (e.g., Llama-Nemotron, OpenThoughts).
   • Introduced explicit reasoning traces (Chain-of-Thought) delimited by special tokens.
4. Post-Training (Alignment):
   • SFT (Supervised Fine-Tuning): Trained on a diverse instruction-following dataset (2.5B tokens) covering reasoning, coding, and tool use.
   • APO (Anchored Preference Optimization): A DPO variant to align the model with user intent and safety policies.
   • Model Souping: Merged multiple checkpoints to stabilize performance and combine complementary strengths.

Inference Modes

The model supports multiple interaction modes controlled via special tokens:

• Non-Reasoning: Direct answers.
• Full Reasoning: Step-by-step Chain-of-Thought in English or Italian.
• Turbo Reasoning: A compressed, bullet-point style reasoning mode designed for low-latency, real-time use cases.

3. Key Contributions

• Sovereign European AI: A fully open-weight, open-data model compliant with the EU AI Act, featuring a dedicated Italian language focus (25% of training data) and rigorous copyright filtering.
• Efficiency via MoE: Achieves performance comparable to dense 8B–16B models while using only 1/5 to 1/2 of the inference power and 1/10 to 1/6 of the training data compared to larger models like Qwen3 or Llama3.
• Multi-Modal Reasoning: Native support for switching between non-reasoning, full reasoning, and "turbo" reasoning in both English and Italian within a single model.
• Transparency: Full disclosure of the training pipeline, data sources, copyright filtering mechanisms, and compute costs.

4. Results

Evaluated against dense models (Llama-3.1-8B, Gemma-2-9B) and larger MoE models (Qwen3-30B-A3B, GPT-OSS-20B):

• Benchmark Performance:
  • Reasoning: Strong performance on MMLU-Pro (57.3), AIME26 (70.0), and GSM8K (88.0), outperforming comparable 8B–9B dense models and approaching the performance of much larger 30B models.
  • Italian Language: Significantly outperforms other open-weight models of similar scale on MMLU-IT and ARC-Challenge-IT.
  • Instruction Following: High scores on IFEval (72.0).
  • Weaknesses: Slightly lower performance on HumanEval (code) and BFCL (tool calling) compared to top-tier models, attributed to a shorter SFT phase focused heavily on reasoning.
• Efficiency Metrics:
  • Cost of Intelligence: When normalized by training tokens and active parameters, EngGPT2 achieves the highest efficiency among all evaluated models, positioning it in the "upper-right" quadrant of the efficiency-performance trade-off graph.
  • Turbo Mode: Reduces reasoning token generation by 83–96% with a manageable performance drop (13–60% depending on task complexity), offering a viable option for latency-sensitive applications.
• Compute Footprint: Total training compute was 5.7 × 10²² FLOPs, well below the EU AI Act's GPAI threshold (10²⁵ FLOPs), avoiding the most stringent regulatory obligations while maintaining high capability.

5. Significance

EngGPT2 represents a pivotal step toward a sovereign, efficient, and sustainable European AI ecosystem. It demonstrates that:

1. Sparse architectures (MoE) can deliver state-of-the-art reasoning and multilingual capabilities with a fraction of the computational resources required by dense models.
2. Regulatory compliance (EU AI Act) and high performance are not mutually exclusive; rigorous data filtering and transparency can be integrated into the training pipeline without sacrificing capability.
3. Local language focus (Italian) is critical for European adoption, and EngGPT2 sets a new baseline for open-weight models in this domain.

The model serves as a foundational building block for European enterprises and public administrations, offering a cost-effective, legally compliant, and high-performance alternative to proprietary foreign models. Future work will focus on improving coding/tool-calling capabilities and scaling the model size.