Technical Summary: Mellum 2

Problem Statement

The landscape of open-weight Large Language Models (LLMs) for software engineering faces a trade-off between quality and deployment cost. Dense models in the 4–14B parameter range are cost-effective to serve but often plateau on complex coding and reasoning tasks. Conversely, very large Mixture-of-Experts (MoE) models achieve frontier quality but incur deployment costs that strain everyday use. The challenge is to create a model that absorbs the long tail of programming knowledge and reasoning capabilities while maintaining a per-token compute cost comparable to a 2–3B dense model, enabling practical deployment on commodity hardware (specifically single H100 GPUs) within latency-sensitive IDE environments.

Methodology

Architecture Design

Mellum 2 is a 12B-parameter decoder-only Transformer utilizing a Mixture-of-Experts (MoE) architecture with 2.5B active parameters per token. The design is derived from the Qwen3-MoE recipe but modified with three specific deployment-oriented optimizations validated through ablation studies:

• Expert Configuration: The model employs 64 total experts with 8 active per token (top-8 routing). This configuration was selected as the optimal trade-off between inference latency and model quality, allowing the model to scale to ~12B total parameters while matching the latency of a 7B dense baseline.
• Grouped-Query Attention (GQA): To optimize throughput under high concurrency, the model uses only 4 KV heads (down from standard configurations). Ablations showed that 8 heads degraded throughput significantly, while 2 heads compromised quality.
• Sliding Window Attention (SWA): To reduce latency without sacrificing long-range context, SWA is applied to 3 out of every 4 layers with a window size of 1,024 tokens. The remaining layers use full attention.
• Multi-Token Prediction (MTP): A single auxiliary MTP head predicts the next token in addition to the standard next-token objective. This head serves a dual purpose: as an auxiliary pre-training objective and as a built-in draft model for speculative decoding during inference.

Pre-Training

The model was pre-trained on approximately 10.6 trillion tokens using a three-phase curriculum that progressively shifts the data mixture from diverse web content to curated code and mathematical content:

1. Foundation (58% of tokens): 70% web/general knowledge, 23% code, 6% math.
2. Quality Uplift (26.2%): 44% web, 42% code, 14% math. Introduces high-quality curated datasets.
3. Capability Sharpening (15.9%): 23% web, 59% code, 18% math. Focuses on high-quality synthetic code and reasoning data during learning rate decay.

The training utilized the Muon optimizer with FP8 hybrid mixed precision and a Warmup-Hold-Decay (WHD) learning rate schedule. The data pipeline included a "Fill-in-the-Middle" (FIM) objective for code completion, with FIM rates varying by phase (50% in Phase 1, 10% in Phase 2, 50% restricted to code in Phase 3).

Long-Context Extension

The base model was extended from an 8,192-token context to 131,072 tokens (128K) using layer-selective YaRN. This method applies frequency re-mapping only to the global (full-attention) layers, leaving the sliding window layers with their original RoPE parameters. This approach was found to outperform uniform RoPE scaling or leaving parameters unchanged.

Post-Training

Two variants were produced from the long-context base:

1. Instruct: A general-purpose assistant that answers directly.
2. Thinking: A reasoning-augmented assistant that emits an explicit chain of thought before the final answer.

Both variants underwent Supervised Fine-Tuning (SFT) followed by Reinforcement Learning with Verifiable Rewards (RLVR). The RL stage utilized a variant of GRPO (Group Relative Policy Optimization) with asynchronous rollouts and specific stability mechanisms (IcePop truncation) to handle train-inference mismatches in MoE routers. The reward signals were programmatically verifiable (e.g., unit tests for code, symbolic verification for math), eliminating the need for a separate reward model.

Key Contributions

• Efficiency-Aware Architecture: A 12B/2.5B-active MoE configuration that matches or exceeds the throughput of a 7B dense baseline (Qwen2.5-7B) on a single H100 GPU while offering a larger parameter envelope.
• Three-Phase Pre-Training Curriculum: A systematic shift from web data to curated code/math (23% → 42% → 59% code ratio) over 10.6T tokens, optimized with Muon and FP8.
• Production-Scale Training Recipe: Integration of Muon, FP8 hybrid precision, and a Warmup-Hold-Decay schedule with linear decay to zero, demonstrating stability across the full training run.
• Long-Context Extension: Successful extension to 128K context using layer-selective YaRN, validated by RULER benchmarks.
• Dual Variants: Release of both "Instruct" and "Thinking" variants, both refined via RLVR on math and executable coding tasks.
• Open Release: Availability of base, instruct, and thinking checkpoints under the Apache 2.0 license.

Results

Across benchmarks spanning code generation, math, reasoning, tool use, and safety, Mellum 2 demonstrates competitiveness with open-weight baselines in the 4B–14B range:

• Code Generation: The RL-trained Instruct variant achieves 78.4% on EvalPlus (HumanEval+ and MBPP+ average), outperforming Qwen3.5-9B (71.8%) and Seed-Coder-8B (73.8%). The Thinking variant reaches 75.1% on LiveCodeBench v6, the highest score in its comparison panel.
• Reasoning & Math: Mellum 2 achieves 59.3% on MMLU-Pro (surpassing Qwen2.5-7B's 48.6%) and 74.9% on BBH. On GSM-Plus, the Thinking variant reaches 87.0%, comparable to larger models.
• Tool Use: The Thinking variant leads the panel on BFCL v4 (agentic capabilities) with 45.6%, outperforming the Qwen3.5 family.
• Efficiency: In single-request (sync) mode, Mellum 2 matches Qwen2.5-7B at ~193 tokens/s. In high-concurrency (throughput) mode, it achieves 5,179 tokens/s, a 21% improvement over Qwen2.5-7B and 79% over Qwen3-8B.
• Limitations: The model shows a deliberate trade-off in broad world knowledge (MMLU-Redux and GPQA Diamond), where it trails the Qwen3.5 series, reflecting a training focus on code and developer documentation. Safety metrics show a known "alignment tax" where RL training slightly increases harmful rates compared to the SFT stage.

Significance

The paper positions Mellum 2 as a practical, open-weight solution for the software engineering domain, proving that a carefully designed MoE architecture can deliver frontier-level coding and reasoning capabilities at a fraction of the inference cost of dense models of similar performance. By strictly adhering to an inference-budget constraint during architectural design (matching Qwen2.5-7B latency), the authors demonstrate that it is possible to scale total parameters significantly (to 12B) without sacrificing deployment feasibility on commodity hardware. The release of both "Thinking" and "Instruct" variants, along with the detailed technical report on the training recipe and data pipeline, aims to provide the community with a reproducible design point for small-MoE coding models.