Technical Summary: NebulaExp-8B

Problem Statement

Post-training alignment is critical for determining the reasoning and human-preference capabilities of large language models (LLMs). However, existing research often lacks transparency regarding data construction, filtering rules, and training recipes, hindering community reproducibility and the optimization of lightweight models. Furthermore, the practical details governing post-training performance remain insufficiently understood, particularly concerning the trade-offs between instruction adherence, mathematical reasoning, code generation, and general knowledge. This work addresses these gaps by presenting a fully transparent, ablation-driven post-training pipeline for 8B-scale models.

Methodology

The study introduces NebulaExp, a comprehensive post-training pipeline built upon Qwen3-8B-base. The research is organized into two orthogonal model branches: a General Instruct Model and a Complex Reasoning-Specialized Model. The methodology encompasses three primary stages:

1. Data Curation and Processing

The pipeline processes a raw corpus of 3.84 million multi-source SFT samples and a 200K verifiable RL candidate pool. Key processing steps include:

• Distillation: Unifying response styles and dialogue logic using strong teacher models (Qwen3-235B-A22B-Instruct and Qwen3.5-397B-A17B).
• Filtering: A three-phase pipeline (query, response, answer verification) employing rule-based checks, model-based scoring, and task-specific verification (e.g., code execution, math answer checking).
• Difficulty Grading: A fine-grained system classifying samples into low, medium, and high difficulty based on text features and base model pass@4 rates.
• Classification and Sampling: Task classification via a lightweight classifier followed by diversity-aware sampling to balance domain representation.

2. Training Paradigms

The pipeline evaluates three distinct training modules:

• Supervised Fine-Tuning (SFT): A three-stage approach involving within-domain adaptive sampling, cross-domain gain-based mixing, and final model consolidation.
• Reinforcement Learning (RL): Utilizing GRPO (Group Relative Policy Optimization) on verifiable tasks (Math, Code, Instruction Following). The study investigates sample difficulty (filtering out trivially unsolvable or mastered samples) and training techniques (KL loss, rollout size, on-policy constraints).
• On-Policy Distillation (OPD): A verifier-free paradigm where the student model generates sequences, and a teacher model provides per-token distribution supervision. This includes Single-Teacher OPD (focusing on instruction following) and Multi-Teacher OPD (MOPD) (fusing domain-specialist teachers for Math, Code, Science, and Instruction Following).

3. Evaluation Framework

Models are evaluated against a rigorous suite of benchmarks covering general knowledge (MMLU-Redux, GPQA-Diamond), instruction alignment (IFEval), mathematical/logical reasoning (AIME'24/25, MATH-500, ZebraLogic, AutoLogi), and code generation (LiveCodeBench v5).

Key Contributions

1. Transparent Data Pipeline: The construction of a complete, reproducible data processing pipeline covering multi-source collection, response-style distillation, multi-dimensional filtering, difficulty grading, and diversity sampling.
2. Ablation-Driven Training Recipes: Systematic ablation studies quantifying the impact of data quality, sample difficulty, domain mixture ratios, and teacher selection.
   • Instruct Branch: Identified that mathematical reasoning and code generation require opposite data distribution preferences (high entropy/IFD for math vs. low entropy/IFD for code) and that long-form general text transfers gains to formal logical reasoning.
   • Reasoning Branch: Demonstrated that medium-difficulty GRPO RL improves reasoning scores and that OPD can serve as a verifier-free alternative.
3. OPD Efficacy: Empirical evidence that On-Policy Distillation can outperform RL baselines with significantly fewer samples (4K vs. 53K) and higher stability, particularly for instruction following.
4. Knowledge Recombination: Discovery that MOPD enables student models to exceed the performance bounds of individual teachers on specific benchmarks (e.g., mathematical reasoning) through cross-teacher knowledge recombination.

Key Results

Instruct Model Branch

• SFT Stage: The optimized three-stage SFT approach (NebulaExp-Ins-SFT) improved the average benchmark score from 55.01 (Qwen3-8B-nothink) to 60.99.
• RL Stage: Subsequent GRPO reinforcement learning further elevated the average score to 61.85.
• Trade-offs: While reasoning capabilities (AIME'24 +15.00, ZebraLogic +35.90) saw massive gains, instruction following (IFEval) initially suffered an "alignment tax" (−5.54), which was recoverable through targeted data augmentation.

Reasoning Model Branch

• SFT Stage: NebulaExp-reasoning-SFT achieved strong performance on code (LiveCodeBench v5: 68.25) and math benchmarks (AIME'24: 80.83, MATH-500: 98.00), surpassing several external baselines including Qwen3-14B and QWQ-32B in specific metrics.
• RL Stage: Medium-difficulty GRPO RL on an 8K subset improved the average reasoning score from 73.88 to 75.17, with notable gains in scientific reasoning (GPQA-Diamond).
• OPD Stage:
  • Single-Teacher: Using only 4K instruction-following samples, single-teacher OPD surpassed the RL baseline on IFEval by 3.26 points and improved the general capability average by 4.43 points.
  • Multi-Teacher (MOPD): Fusing four domain-specialist teachers with 10K samples lifted the average performance by 4.18 points over the base model. Notably, the student model surpassed the individual teacher upper bounds on AIME'25 and MATH-500.

Significance and Claims

The paper claims that data correctness filtering acts as the first-order optimization factor, while teacher distribution compatibility outweighs model scale for distillation. The core contribution is not merely achieving higher leaderboard scores, but revealing the internal mechanisms and empirical trade-offs of the post-training pipeline.

The authors assert that a carefully designed, transparent post-training pipeline can substantially improve model capability without modifying the base model architecture. They emphasize that:

• Data Quality > Data Scale: Correctness filtering and difficulty-aware sampling are more critical than simply increasing corpus size.
• OPD as a Viable Alternative: For tasks lacking reliable verifiers, OPD offers a stable, data-efficient alternative to RL, capable of fusing multiple specialist capabilities into a unified model.
• Reproducibility: The work provides a fully reproducible empirical recipe for 8B-scale LLMs, dissecting the trade-offs among instruction adherence, reasoning, and coding.

The study concludes that while current results are promising at the 8B scale, future work must investigate the scalability of these recipes to larger dense and MoE architectures, as well as the extension to agentic training scenarios involving tool use and interactive environments.