Technical Summary: Soofi S 30B-A3B

Problem Statement

The paper identifies three critical gaps in the current landscape of open language models that hinder effective deployment:

1. Substantive Openness: Many models are released as "open-weight" but lack the data recipes, construction artifacts, and detailed accounting required to reproduce or audit the training process.
2. Language Representation: General-purpose multilingual models often dilute capacity across dozens of languages, leaving German underrepresented relative to its economic and scientific weight. Conversely, dedicated European efforts have historically prioritized coverage over frontier capability.
3. Deployment Efficiency: In high-concurrency, long-context scenarios, inference costs are dominated by memory bandwidth rather than parameter count. Dense Transformer models suffer from quadratic attention cache growth, causing throughput to degrade significantly as context lengths increase (tens to hundreds of thousands of tokens).

Methodology

Architecture

Soofi S 30B-A3B adopts a hybrid Mamba-Transformer Mixture-of-Experts (MoE) architecture, based on the openly published Nemotron 3 Nano reference design.

• Structure: A 52-layer network interleaving 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers with shared experts, and 6 Grouped-Query Attention (GQA) layers.
• Efficiency: The model contains 31.6 billion total parameters but activates only **3.2 billion per token** (3.6B including embeddings).
• Cache Optimization: Only the 6 GQA layers maintain a KV cache; the 23 Mamba-2 layers utilize a fixed-size recurrent state. This design keeps the per-sequence cache footprint near-constant (~6 KB per token) regardless of context length, unlike full-attention dense models.

Training Recipe

The model was trained on the German Industrial AI Cloud (operated by Deutsche Telekom in Munich) using up to 512 NVIDIA B200 GPUs. The training followed a Warmup–Stable–Decay (WSD) schedule over approximately 27 trillion tokens across three phases:

1. Phase 1 (Diverse Pretraining, ~20T tokens): A quality-tiered mixture of web text, code, mathematics, and reasoning data. German sources were intentionally up-weighted to 7.2% (vs. ~5% in the reference recipe).
2. Phase 2 (High-Quality Annealing, ~6.58T tokens): A decay phase focusing on high-quality web data, skill-oriented code/math/STEM data, and instruction-following (SFT) formats. German share was increased to 15.32%.
3. Phase 3 (Long-Context Extension, ~0.1T tokens): A specialized phase using length-bucketed up-sampling (4K to 1M tokens) to extend the usable context window.

Data Transparency

The authors commit to full data transparency, releasing:

• Complete per-source token accounting for all phases.
• Exact mixing ratios and epoch multipliers.
• Training and evaluation code.
• For commercially licensed data (e.g., Genios), aggregate statistics and exact mixture accounting are provided, though raw text is not redistributed.

Key Contributions

1. German-English Champion: Soofi S achieves state-of-the-art performance among fully open models on aggregate English and German benchmarks, including code, mathematics, and regional knowledge.
2. Long-Context Serving Efficiency: The hybrid architecture delivers 8–9× higher aggregate decode throughput (tokens per second per GPU) compared to dense 14–24B models at 40K context. Unlike dense models, its throughput remains flat from 4K to 256K context lengths.
3. Radical Openness: The release includes weights, intermediate checkpoints, full data accounting, and training code, adhering to the Open Source AI Definition (OSAID) 1.0.
4. Reproducible Recipe: The paper documents the exact learning rate schedule, hyperparameters, and data construction logic, allowing third parties to rebuild the run.

Results

Capability Benchmarks

Evaluated against 16 open base models (including Olmo 3 32B, Apertus 70B, EuroLLM, and Qwen3.5):

• Code: Soofi S ranks first among open-source models on HumanEval, MBPP, HumanEval-DE, and MBPP-DE.
• German Performance: It outperforms every European sovereign baseline (including those with significantly larger active parameters) on all German benchmarks, often by 10–30 points. It matches or exceeds dense 14–27B models on German aggregates.
• English Performance: It matches the performance of dense 14–27B models on English aggregates and achieves the highest English aggregate score among fully open models in the comparison.
• Math & Reasoning: It achieves top-tier scores on GSM8K, Minerva benchmarks, and GPQA-Diamond, though it trails the largest dense models (e.g., Qwen3.5 35B-A3B) on German competition mathematics (Minerva MATH-DE) and open-domain factual recall (NaturalQuestions).

Efficiency Benchmarks

• Throughput: At 40K context and batch size 32, Soofi S sustains 4.82k TPS/GPU, which is 9.2× higher than Ministral 3 14B.
• Scalability: While dense models show significant throughput degradation as context grows, Soofi S maintains near-constant throughput up to 256K context.
• Latency: At 256K context, Soofi S processes prompts (TTFT) significantly faster than dense baselines (372.7s vs. >2000s for Ministral 3 14B).

Significance and Claims

The paper claims that Soofi S represents a new frontier for sovereign, open-source foundation models. Its significance lies in demonstrating that a model can be:

1. Fully Open: Providing the artifacts necessary for scientific audit and reconstruction, not just inference.
2. Bilingual Depth: Specialized for German and English with deliberate data up-weighting, rather than being a thinly spread multilingual model.
3. Efficient: Achieving frontier-level capability with the inference cost of a 3B parameter model due to its hybrid Mamba-MoE architecture.

The authors position Soofi S as filling the gap between broadly multilingual European sovereignty efforts and efficient international open-weight baselines. They assert it is the first European sovereign model to sit on the same capability-per-active-parameter frontier as the strongest international open-weight releases, while being the strongest fully open base model for both English and German aggregates in their evaluation suite.

The paper acknowledges limitations, noting that competition-style German mathematics and open-domain factual recall remain areas where the model trails larger dense baselines, attributing the latter to the capacity constraints of having only 3B active parameters.